3

Here is my environment:

  • project A is on the server. My PC connects to the server by 1 Gbit ethernet. Ping latency between PC and server are about 0.3 ms.
  • project A's source code size is about 75 MB. But its ".git" directory is about 800 MB.

My magit is quite slow. Run magit checkout to another branch (with nothing different between these 2 branches) in project A could take as long as 25 seconds.

Here is my profiling info for checkout to branch: Profiling Informantion

I've checked the performance tips in magit doc and tried it, but no obvious improvement.

3

I try to dig deeper into this problem by setting (setq tramp-verbose 10).

Here is one possible step for reproducing the slowness:

  • git clone https://github.com/openjdk-mirror/jdk7u-hotspot.git on server A and create a test branch base on master branch.
  • using tramp to visit a file in above project
  • run magit-checkout test

And find the slowness is caused by that tramp tries to find the end of remote command's output by searching some pattern in tramp-wait-for-output function's line (tramp-wait-for-regexp proc timeout regexp1). In this case:

  • regexp1 is "\\(^\\|\000\\)[^#$\n]*///deab18b5e34aa7487addb2439bbab2a9#\\$\r?$"
  • proc's buffer string is the output of command: git "--no-pager" "--literal-pathspecs" "-c" "core.preloadindex=true" ls-tree -z -r --name-only HEAD in above git repository. Which is one quite huge line. In my test case, it is a line with 226383 character and split by null. Here is part of it: .hgignore^@.hgtags^@.jcheck/conf^@ASSEMBLY_EXCEPTION^@LICENSE^@README^@
  • timeout argument is nil

So I walk around this issue by modifying tramp-check-for-regexp to below:

(defun tramp-check-for-regexp (proc regexp)
  "Check, whether REGEXP is contained in process buffer of PROC.
Erase echoed commands if exists."
  (with-current-buffer (process-buffer proc)
    (if (> (point-max) 1000)
        (progn
          (goto-char (point-max))
          (ignore-errors (re-search-backward regexp (- (point-max) 1000) t)))
      (progn
        (goto-char (point-min))

        ;; Check whether we need to remove echo output.
        (when (and (tramp-get-connection-property proc "check-remote-echo" nil)
                   (re-search-forward tramp-echoed-echo-mark-regexp nil t))
          (let ((begin (match-beginning 0)))
            (when (re-search-forward tramp-echoed-echo-mark-regexp nil t)
              ;; Discard echo from remote output.
              (tramp-set-connection-property proc "check-remote-echo" nil)
              (tramp-message proc 5 "echo-mark found")
              (forward-line 1)
              (delete-region begin (point))
              (goto-char (point-min)))))

        (when (or (not (tramp-get-connection-property proc "check-remote-echo" nil))
                  ;; Sometimes, the echo string is suppressed on the remote side.
                  (not (string-equal
                        (tramp-compat-funcall
                         'substring-no-properties tramp-echo-mark-marker
                         0 (min tramp-echo-mark-marker-length (1- (point-max))))
                        (tramp-compat-funcall
                         'buffer-substring-no-properties
                         (point-min)
                         (min (+ (point-min) tramp-echo-mark-marker-length)
                              (point-max))))))
          ;; No echo to be handled, now we can look for the regexp.
          ;; Sometimes, lines are much to long, and we run into a "Stack
          ;; overflow in regexp matcher".  For example, //DIRED// lines of
          ;; directory listings with some thousand files.  Therefore, we
          ;; look from the end.
          (goto-char (point-max))
          (ignore-errors (re-search-backward regexp nil t)))))))

I add checking whether the buffer is larger than 1000 point, if yes, add "bound" to re-search-backward and ignore other checking and processing. This is really a dirty hack without any considering of other case using tramp-check-for-regexp.

However, it reduces the magit-checkout time from 25s to 3s.

Hope tramp could fix this issue in the future.

| improve this answer | |
  • 2
    Rather than hope, better M-x report-emacs-bug. – Stefan Aug 5 '17 at 12:39
1

Well, most of the time is spent in tramp-check-for-regexp. This is, where the output of the remote processes, magit has fired, is parsed. Don't know how to improve there from user side.

In Tramp 2.3.2, which will be released next two days, there are some changes in this corner. Most important, while reading output from external processes, running timers are discarded. I have no hard figures, but I believe this might increase performance. Maybe you give this Tramp version a try.

| improve this answer | |
  • I have looked into a similar issue a long time ago and it did indeed turn out that some regexp inefficiency in tramp was the culprit. When magit requests a list of files from git, it requests them to be separated with null characters. This can result in very long lines, which is a problem because tramp (if I remember correctly) looks for some information that is usually found at the very beginning of a line by searching from the very end of every line. I hope that's one of the things that was changed! – tarsius Jun 29 '17 at 18:25
  • The last time this issue came up, the solution for that particular user was to setup good git ignore rules to prevent a very long line listing thousands of untracked files. – tarsius Jun 29 '17 at 18:26
  • No, there wasn't done something with respect to regexp parsing in Tramp recently. Just some changes when and how timers are invoked (which could run long time also). Honestly, I'm not a magit user, but maybe I shall try this in order to reproduce the problem. Are there special magit settings or examples which would trigger the problem prominently? – Michael Albinus Jun 29 '17 at 20:25
  • Hi, Michael, here is the step to reproduce this issue: – zwy Jun 30 '17 at 3:16
  • There is no special set up for reproducing this issue. I use magit on hotspot project. It can be downloaded by: git clone "hg::hg.openjdk.java.net/jdk10/hs/hotspot". It is a hg project. I clone it by git. To make git be able to clone hg repo, you should have mercurial installed and put git-remote-hg (raw.githubusercontent.com/felipec/git-remote-hg/master/…) to your $PATH directory. – zwy Jun 30 '17 at 3:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.