How do I prevent extremely long lines making Emacs slow?

Question

I see hugely varied performance depending on how many newlines there are in the file I'm visiting.

Here's an example. I have two JSON files:

$ wget https://github.com/Wilfred/ReVo-utilities/blob/a4bdc40dd2656c496defc461fc19c403c8306d9f/revo-export/dictionary.json?raw=true -O one_line.json
$ python -m json.tool <one_line.json >pretty_printed.json

These are two JSON files with the same content. one_line.json is 18MiB of JSON without any newlines. pretty_printed.json has newlines and whitespace added, making it 41MiB.

However, the bigger file split over many lines is much faster to open in Emacs, both in Javascript mode and Fundamental mode.

Why does Emacs have such poor performance with long lines, since it's actually fewer bytes? Is there anything I can do to improve performance without reformatting the data outside of Emacs?

Answer 1

Emacs' handling of long lines is not very well optimized. For a number of operations, Emacs has to scan the whole line repeatedly. For example, to display a line, Emacs has to figure out the height of the line, which requires scanning the whole line to find the tallest glyph. Additionally, scanning for bidirectional display eats up a lot of time. You can get some additional information in, for example, the docstring of cache-long-line-scans (renamed cache-long-scans in 24.4).

You can try and see if setting bidi-paragraph-direction to left-to-right and setting bidi-inhibit-bpa¹ (new in Emacs 27) to t improves the speed for you.² This removes some significant contributors to line scans, but sadly not the only ones.

The best option is to add newlines. You can pipe a JSON file through e.g. python -c 'import json, sys ; json.dump(json.load(sys.stdin), sys.stdout, indent=2)' to add newlines and improve readability in general.

---

¹ bidi-inhibit-bpa is new in Emacs 27. Setting it to t can have a noticeable (sometimes dramatic) effect in cases where (a) a physical line is very long, and (b) the line includes many parentheses and/or braces, especially if they are nested.

² There is also bidi-display-reordering, but this is only for internal/debugging purposes. Setting it to nil may improve performance, however that is an unsupported setting which may cause other problems, so it is not recommended. In Emacs 27, the combination of setting bidi-paragraph-direction and bidi-inhibit-bpa is a supported configuration which provides approximately the same performance boost.

Answer 2

so-long.el (Emacs Wiki, GNU ELPA) will help in many situations. It is included by default in Emacs 27+ and available in GNU ELPA for older versions of Emacs (24.4 and later).

This library enables you to configure some simple thresholds to check when visiting a file, beyond which a more performant mode will be used in place of the normal mode, including disabling certain buffer-local minor modes with performance implications, and tuning buffer-local variables to maximise responsiveness. By default this will happen for programming modes only, but the behaviour is all highly configurable.

Use M-x global-so-long-mode to enable/toggle the functionality. To enable the functionality by default, either customize the global-so-long-mode user option, or add the following to your init file:

;; Avoid performance issues in files with very long lines.
(global-so-long-mode 1)

The comprehensive documentation can be read on its GNU ELPA page in plain text, but is nicer to read inside Emacs itself using the command:

M-x so-long-commentary

---

Using the 18MiB one_line.json from the question as an example, the time until Emacs is responsive (on my system) after visiting that file:

• Without so-long, Emacs hangs for nearly 3.5 minutes.

  $ time emacs -Q --eval "(setq large-file-warning-threshold nil)" --eval "(run-with-idle-timer 0.1 nil #'kill-emacs)" -- one_line.json
  
  real    3m25.785s
  user    3m25.058s
  sys     0m0.365s
  

• With so-long, Emacs is responsive in under a second.

  $ time emacs -Q -f global-so-long-mode --eval "(setq large-file-warning-threshold nil)" --eval "(run-with-idle-timer 0.1 nil #'kill-emacs)" -- one_line.json
  
  real    0m0.890s
  user    0m0.538s
  sys     0m0.047s
  

Note that despite that very dramatic improvement, if you actually need to navigate to a position very far into a line of such magnitude, then performance will again become very bad -- still better than it would have been, but the performance issues that so-long can't address will rapidly overshadow the ones it can address, the further into the line you get.

Visiting and moving around near the start of the file should present no problems at all, however -- and for long lines which are not on the scale of this example, the improvements may well be sufficient to make editing practical throughout the buffer.

This library may be noticeably more effective in Emacs 27.1 than in earlier versions, because one of the variables that it sets, bidi-inhibit-bpa, was introduced in 27.1, and that has a significant effect on performance for lines with vast numbers of deeply-nested 'paired bracket' characters (as are abundant in JSON); so the combination of Emacs 27.1 and global-so-long-mode is the best pairing at present.

(If editing files like this one is a regular requirement, however, then Emacs probably isn't the best tool for the job. For JSON specifically, the jq command-line tool may serve you well.)

Answer 3

I did some brief experiments with this using a minified copy of jquery. font-lock-mode and flycheck-mode both contributed to slowness, as did js2-mode, and prettify-symbols-mode. line-number-mode and column-number-mode had minor effect. Once I had turned off all the different modes though the performance was relatively snappy. Use C-h m and start disabling different modes that are enabled, or try just switch to fundamental-mode.

Interestingly using hexl-mode I could fly through the file without any issue, though obviously columns were quite short. Unfortunately visual-line-mode really slowed things down.

My guess is that the syntax table is happy to stop processing at line endings, and when it's all on one line it has to reparse everything on every update.

Answer 4

UPDATE http://git.savannah.nongnu.org/cgit/so-long.git/tree/so-long.el is a part of Emacs 27.1!

OLD I usually unroll long lines and indent by tags (like HTML, XML, JSON).

In order to make such operation possible I add:

(setq line-number-display-limit large-file-warning-threshold)
(setq line-number-display-limit-width 200)

(defun my--is-file-large ()
  "If buffer too large and my cause performance issue."
  (< large-file-warning-threshold (buffer-size)))

(define-derived-mode my-large-file-mode fundamental-mode "LargeFile"
  "Fixes performance issues in Emacs for large files."
  ;; (setq buffer-read-only t)
  (setq bidi-display-reordering nil)
  (jit-lock-mode nil)
  (buffer-disable-undo)
  (set (make-variable-buffer-local 'global-hl-line-mode) nil)
  (set (make-variable-buffer-local 'line-number-mode) nil)
  (set (make-variable-buffer-local 'column-number-mode) nil) )

(add-to-list 'magic-mode-alist (cons #'my--is-file-large #'my-large-file-mode))

I split line by regex, for XML it: C-M-% >< RET >NL< RET !.

After Emacs split long lines - it is possible to enable many *-modes and re-indent code.

For the note: How to prevent slow-down when an inferior processes generates long lines?

Answer 5

I expect you'll find that the difference is due to font-lock. When fontification is to be performed on the subset of the file which is visible in the window, it proceeds by first extending the fontification region such that it will include full semantic units. See the font-lock-extend-region-functions code for this. It's common for this to include extending the region to include full lines. When the lines are extremely long, this can lead to fontification being performed across a very much larger chunk of content than is actually visible.

Additionally, when newlines themselves have semantic information, their absence may sometimes mean that regexp patterns for font lock have to scan further in order to determine whether they match or not.

Answer 6

I created my own solution for this problem here: https://github.com/rakete/too-long-lines-mode

I was not satisfied with phils solution which switches a buffer with very long lines to fundamental-mode, I wanted a solution that lets me keep syntax highlighting and other major-mode features. So I created a minor-mode that uses overlays to hide most characters of overly long lines.

That works around the problem and makes emacs usable even in buffers with very long lines, without having to fall back to fundamental-mode.

Answer 7

In my Emacs setup I have a mode with custom fontification, i.e. where I set font-lock-defaults. A single page down would use 30 seconds to display part of 30000 character line. This slow down was fixed by reducing regexp backtracking. Instead of:

  (".* ended with an incomplete command*" 0 font-lock-comment-face)

do this

  ("^.\{1,80\} ended with an incomplete command*" 0 font-lock-comment-face)

Answer 8

Although the question asks for how to handle long lines in a file, I often experience this problem when working with a buffer such as a command interpreter, or comint (e.g. shell). It may be possible to modify the strings to be more agreeable to Emacs before they're placed in the buffer.

For example, M-x shell is a comint. The comint.el library provides the comint-preoutput-filter-functions variable. We can pre-filter process output for something like a comma and append a newline after each one:

(add-hook 'comint-preoutput-filter-functions  
          '(lambda (x) (replace-regexp-in-string "," ",\n" x)))

Answer 9

Emacs 29 will feature improvements in that area. A lot of that is optimizing the display engine subroutines to work better with longer lines. Better algorithmic complexity, I imagine.

Answer 10

In my shell-mode buffers (M-x shell), I find myself piping to cut -c 1-2000 to avoid long lines.

Answer 11

I use the following function for opening in dired-mode large files with long lines:

(defun dired-find-file-conservatively ()
   (interactive)
   (let ((auto-mode-alist nil))
     (dired-find-file)
     ;; disable costly modes
     (fundamental-mode)
     (setq-local bidi-display-reordering nil)
     (when (boundp 'smartparens-mode)
       (smartparens-mode -1))))

(define-key dired-mode-map (kbd "S-<return>") 'dired-find-file-conservatively)

Answer 12

Here is a workaround, taken from emacs-devel:

(add-hook 'find-file-hook
          (defun my-find-file-care-about-long-lines ()
            (save-excursion
              (goto-char (point-min))
              (when (and (not (eq major-mode 'image-mode))
                         (search-forward-regexp ".\\{2000\\}" 50000 t)
                         (y-or-n-p "Very long lines detected - enable 
longlines-mode? "))
                (require 'longlines)
                (longlines-mode +1)))))