108

I see hugely varied performance depending on how many newlines there are in the file I'm visiting.

Here's an example. I have two JSON files:

$ wget https://github.com/Wilfred/ReVo-utilities/blob/a4bdc40dd2656c496defc461fc19c403c8306d9f/revo-export/dictionary.json?raw=true -O one_line.json
$ python -m json.tool <one_line.json >pretty_printed.json

These are two JSON files with the same content. one_line.json is 18MiB of JSON without any newlines. pretty_printed.json has newlines and whitespace added, making it 41MiB.

However, the bigger file split over many lines is much faster to open in Emacs, both in Javascript mode and Fundamental mode.

Why does Emacs have such poor performance with long lines, since it's actually fewer bytes? Is there anything I can do to improve performance without reformatting the data outside of Emacs?

2
  • 3
    Not really an answer but might be of use: View Large Files (vlf) is a minor mode that is aimed to help with editing large files by loading them in batches. Disclaimer: I've never used it and I don't know whether it handles long lines in batches too.
    – elemakil
    Oct 2, 2014 at 10:18
  • 3
    Knowing this kind of behaviour, and especially when trying to guard myself against reading a log which spits out a long line, I often do something like $ tail -f /some/file | fold -s in a shell buffer. This isn't good for editing, obviously, but helps a lot with reading.
    – wvxvw
    Dec 18, 2014 at 21:48

12 Answers 12

72

Emacs' handling of long lines is not very well optimized. For a number of operations, Emacs has to scan the whole line repeatedly. For example, to display a line, Emacs has to figure out the height of the line, which requires scanning the whole line to find the tallest glyph. Additionally, scanning for bidirectional display eats up a lot of time. You can get some additional information in, for example, the docstring of cache-long-line-scans (renamed cache-long-scans in 24.4).

You can try and see if setting bidi-paragraph-direction to left-to-right and setting bidi-inhibit-bpa1 (new in Emacs 27) to t improves the speed for you.2 This removes some significant contributors to line scans, but sadly not the only ones.

The best option is to add newlines. You can pipe a JSON file through e.g. python -c 'import json, sys ; json.dump(json.load(sys.stdin), sys.stdout, indent=2)' to add newlines and improve readability in general.


1 bidi-inhibit-bpa is new in Emacs 27. Setting it to t can have a noticeable (sometimes dramatic) effect in cases where (a) a physical line is very long, and (b) the line includes many parentheses and/or braces, especially if they are nested.

2 There is also bidi-display-reordering, but this is only for internal/debugging purposes. Setting it to nil may improve performance, however that is an unsupported setting which may cause other problems, so it is not recommended. In Emacs 27, the combination of setting bidi-paragraph-direction and bidi-inhibit-bpa is a supported configuration which provides approximately the same performance boost.

9
  • 8
    Out of curiosity, is this something that can't be improved algorithmically?
    – PythonNut
    Feb 24, 2016 at 3:04
  • 14
    When choosing the underlying data structure of an editor, you have to pick between certain pros and cons. Emacs uses a gap buffer, which is a highly space efficient data structure for insertion and deletion, but it makes line-based operations slower as you have to scan sequentially for a newline. Emacs could use a different data structure, but that would make other operations slower. Emacs already uses a line cache, but that does not really help in all situations. So, not easily improved algorithmically, but profiling and optimizing never hurts. :-) Feb 25, 2016 at 8:42
  • 6
    (setq-default bidi-display-reordering nil) -- some users may not realize that this is a buffer-local variable, which may need a default setting to the extent a user wants this to be global. I wish I would have added that to my init.el years ago ... but at least it's there now. Thank you so very much!!!
    – lawlist
    May 24, 2017 at 20:03
  • 2
    What am I giving up by setting bidi-display-reordering ? May 29, 2019 at 14:22
  • 2
    The current Emacs maintainer, Eli, who wrote the BIDI code, writes this about switching off bidi-display-reordering: "One comment I have is that disabling bidi-display-reordering … puts the display engine in a state that is not being tested, and can cause inconsistencies and even bugs (because some portions of the code were written under the assumption that this variable is never nil)."
    – Clément
    Jul 13, 2019 at 21:01
41

so-long.el (Emacs Wiki, GNU ELPA) will help in many situations. It is included by default in Emacs 27+ and available in GNU ELPA for older versions of Emacs (24.4 and later).

This library enables you to configure some simple thresholds to check when visiting a file, beyond which a more performant mode will be used in place of the normal mode, including disabling certain buffer-local minor modes with performance implications, and tuning buffer-local variables to maximise responsiveness. By default this will happen for programming modes only, but the behaviour is all highly configurable.

Use M-x global-so-long-mode to enable/toggle the functionality. To enable the functionality by default, either customize the global-so-long-mode user option, or add the following to your init file:

;; Avoid performance issues in files with very long lines.
(global-so-long-mode 1)

The comprehensive documentation can be read on its GNU ELPA page in plain text, but is nicer to read inside Emacs itself using the command:

M-x so-long-commentary


Using the 18MiB one_line.json from the question as an example, the time until Emacs is responsive (on my system) after visiting that file:

  • Without so-long, Emacs hangs for nearly 3.5 minutes.

    $ time emacs -Q --eval "(setq large-file-warning-threshold nil)" --eval "(run-with-idle-timer 0.1 nil #'kill-emacs)" -- one_line.json
    
    real    3m25.785s
    user    3m25.058s
    sys     0m0.365s
    
  • With so-long, Emacs is responsive in under a second.

    $ time emacs -Q -f global-so-long-mode --eval "(setq large-file-warning-threshold nil)" --eval "(run-with-idle-timer 0.1 nil #'kill-emacs)" -- one_line.json
    
    real    0m0.890s
    user    0m0.538s
    sys     0m0.047s
    

Note that despite that very dramatic improvement, if you actually need to navigate to a position very far into a line of such magnitude, then performance will again become very bad -- still better than it would have been, but the performance issues that so-long can't address will rapidly overshadow the ones it can address, the further into the line you get.

Visiting and moving around near the start of the file should present no problems at all, however -- and for long lines which are not on the scale of this example, the improvements may well be sufficient to make editing practical throughout the buffer.

This library may be noticeably more effective in Emacs 27.1 than in earlier versions, because one of the variables that it sets, bidi-inhibit-bpa, was introduced in 27.1, and that has a significant effect on performance for lines with vast numbers of deeply-nested 'paired bracket' characters (as are abundant in JSON); so the combination of Emacs 27.1 and global-so-long-mode is the best pairing at present.

(If editing files like this one is a regular requirement, however, then Emacs probably isn't the best tool for the job. For JSON specifically, the jq command-line tool may serve you well.)

4
  • 1
    This is a really nice solution (would love to see it on MELPA), but my Emacs instance is still extremely slow when opening one_line.json. I think it would be significantly faster if it didn't first activate original the major mode. Nov 7, 2017 at 14:19
  • 1
    Preventing the original major mode from being called would at minimum be fiddlier and more dependent on implementation details (advising set-auto-mode-0 would be necessary), although that might be ok in practice. Note that we mustn't prevent Emacs from determining the major mode to use, as we must only (re)act upon the selection of certain major modes. I'm not certain this would be a significant improvement, though (but it would be easy to experiment by making auto-mode-alist select so-long-mode directly).
    – phils
    Nov 7, 2017 at 20:31
  • 3
    so-long is now available in GNU ELPA for Emacs versions 24.4 and above. Version 1.0 (current at time of writing) is included in Emacs 27.1 by default.
    – phils
    Aug 17, 2020 at 23:23
  • As noted in Dmitry's answer, Emacs 29 introduces dramatic improvements in this area.
    – phils
    Jul 9, 2023 at 14:56
19

I did some brief experiments with this using a minified copy of jquery. font-lock-mode and flycheck-mode both contributed to slowness, as did js2-mode, and prettify-symbols-mode. line-number-mode and column-number-mode had minor effect. Once I had turned off all the different modes though the performance was relatively snappy. Use C-h m and start disabling different modes that are enabled, or try just switch to fundamental-mode.

Interestingly using hexl-mode I could fly through the file without any issue, though obviously columns were quite short. Unfortunately visual-line-mode really slowed things down.

My guess is that the syntax table is happy to stop processing at line endings, and when it's all on one line it has to reparse everything on every update.

2
  • 2
    Can you open a bug report on Flycheck's tracker? I'm pretty sure we don't want long lines causing issues, and Emacs + Flycheck shouldn't be worse than Emacs (which is still pretty bad).
    – Clément
    Apr 14, 2016 at 12:57
  • To cross-reference this with the so-long.el answer, if you have identified any minor modes and settings which are slowing things down, you can configure so-long to disable those particular modes, and/or set those variables, whenever it triggers. Doing these things for you automatically is its main purpose.
    – phils
    Aug 18, 2020 at 0:53
13

UPDATE http://git.savannah.nongnu.org/cgit/so-long.git/tree/so-long.el is a part of Emacs 27.1!

OLD I usually unroll long lines and indent by tags (like HTML, XML, JSON).

In order to make such operation possible I add:

(setq line-number-display-limit large-file-warning-threshold)
(setq line-number-display-limit-width 200)

(defun my--is-file-large ()
  "If buffer too large and my cause performance issue."
  (< large-file-warning-threshold (buffer-size)))

(define-derived-mode my-large-file-mode fundamental-mode "LargeFile"
  "Fixes performance issues in Emacs for large files."
  ;; (setq buffer-read-only t)
  (setq bidi-display-reordering nil)
  (jit-lock-mode nil)
  (buffer-disable-undo)
  (set (make-variable-buffer-local 'global-hl-line-mode) nil)
  (set (make-variable-buffer-local 'line-number-mode) nil)
  (set (make-variable-buffer-local 'column-number-mode) nil) )

(add-to-list 'magic-mode-alist (cons #'my--is-file-large #'my-large-file-mode))

I split line by regex, for XML it: C-M-% >< RET >NL< RET !.

After Emacs split long lines - it is possible to enable many *-modes and re-indent code.

For the note: How to prevent slow-down when an inferior processes generates long lines?

9

I expect you'll find that the difference is due to font-lock. When fontification is to be performed on the subset of the file which is visible in the window, it proceeds by first extending the fontification region such that it will include full semantic units. See the font-lock-extend-region-functions code for this. It's common for this to include extending the region to include full lines. When the lines are extremely long, this can lead to fontification being performed across a very much larger chunk of content than is actually visible.

Additionally, when newlines themselves have semantic information, their absence may sometimes mean that regexp patterns for font lock have to scan further in order to determine whether they match or not.

7

I created my own solution for this problem here: https://github.com/rakete/too-long-lines-mode

I was not satisfied with phils solution which switches a buffer with very long lines to fundamental-mode, I wanted a solution that lets me keep syntax highlighting and other major-mode features. So I created a minor-mode that uses overlays to hide most characters of overly long lines.

That works around the problem and makes emacs usable even in buffers with very long lines, without having to fall back to fundamental-mode.

1
  • This is the only solution thats worked for me! Searched hours for this, much Thanks!
    – ed9w2in6
    Jul 17, 2020 at 17:02
2

In my Emacs setup I have a mode with custom fontification, i.e. where I set font-lock-defaults. A single page down would use 30 seconds to display part of 30000 character line. This slow down was fixed by reducing regexp backtracking. Instead of:

  (".* ended with an incomplete command*" 0 font-lock-comment-face)

do this

  ("^.\{1,80\} ended with an incomplete command*" 0 font-lock-comment-face)
4
  • This is not an answer to the question, which is not specifically about font-lock-defaults or regexp matching.
    – Drew
    Mar 26, 2016 at 21:21
  • 3
    @Drew Less than ideal regex is making font-lock slow on long lines though...
    – wasamasa
    Mar 27, 2016 at 0:02
  • 1
    @wasamasa: Yes. The question itself is too broad, IMO. There are plenty of things that can slow Emacs down (and for which actions?) when long lines are involved.
    – Drew
    Mar 27, 2016 at 0:06
  • 5
    I don't think the question is to broad ("why do long lines make Emacs slow")? Nor do I think that the answer doesn't address the question ("one possible reason are suboptimal regexps"). Other answers can address other reasons. Opening a file with long lines isn't to broad a topic just because that might be problematic for a variety of reasons, sometimes you have such files and you have to look at them, preferably using Emacs.
    – tarsius
    Mar 27, 2016 at 11:13
2

Although the question asks for how to handle long lines in a file, I often experience this problem when working with a buffer such as a command interpreter, or comint (e.g. shell). It may be possible to modify the strings to be more agreeable to Emacs before they're placed in the buffer.

For example, M-x shell is a comint. The comint.el library provides the comint-preoutput-filter-functions variable. We can pre-filter process output for something like a comma and append a newline after each one:

(add-hook 'comint-preoutput-filter-functions  
          '(lambda (x) (replace-regexp-in-string "," ",\n" x)))
1

In my shell-mode buffers (M-x shell), I find myself piping to cut -c 1-2000 to avoid long lines.

1
  • This answers the second part of the question: how to improve performance. It does not address the first part (which is OK): "Why does Emacs have such poor performance with long lines?"
    – Drew
    Nov 7, 2017 at 2:27
1

Emacs 29 will feature improvements in that area. A lot of that is optimizing the display engine subroutines to work better with longer lines. Better algorithmic complexity, I imagine.

3
  • The most significant part is a new auto-narrowing feature which the redisplay employs by default when extremely long lines are present, as redisplay can then be very efficient on the narrowed text. I suspect it will be worthwhile for users to read the documentation for this feature and have a general awareness of it, because on those (probably-rare) occasions when it's active, there may be scope for conflicts between that and narrowing used by other code. (AFAIU efforts have been made to address such things, but it's hard to make guarantees in general.)
    – phils
    Feb 10, 2023 at 2:59
  • I'm not sure it's the most significant part. See the latest messages in debbugs#56682.
    – Dmitry
    Feb 12, 2023 at 0:33
  • Cross-referencing with the Emacs 29 NEWS file: git.savannah.gnu.org/cgit/emacs.git/tree/etc/…
    – phils
    Jul 9, 2023 at 14:59
0

I use the following function for opening in dired-mode large files with long lines:

(defun dired-find-file-conservatively ()
   (interactive)
   (let ((auto-mode-alist nil))
     (dired-find-file)
     ;; disable costly modes
     (fundamental-mode)
     (setq-local bidi-display-reordering nil)
     (when (boundp 'smartparens-mode)
       (smartparens-mode -1))))

(define-key dired-mode-map (kbd "S-<return>") 'dired-find-file-conservatively)
0

Here is a workaround, taken from emacs-devel:

(add-hook 'find-file-hook
          (defun my-find-file-care-about-long-lines ()
            (save-excursion
              (goto-char (point-min))
              (when (and (not (eq major-mode 'image-mode))
                         (search-forward-regexp ".\\{2000\\}" 50000 t)
                         (y-or-n-p "Very long lines detected - enable 
longlines-mode? "))
                (require 'longlines)
                (longlines-mode +1)))))
2
  • In Emacs as of 24.4 the longlines-mode got marked as obsoleted by visual-line-mode. May 31, 2019 at 21:52
  • However the two features do very different things behind the scenes, and visual-line-mode does not help with the issue in question, whereas longlines-mode does. For this reason, I expect that longlines.el will be restored to a non-deprecated status.
    – phils
    Jul 16, 2019 at 3:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.