73

I see hugely varied performance depending on how many newlines there are in the file I'm visiting.

Here's an example. I have two JSON files:

$ wget https://github.com/Wilfred/ReVo-utilities/blob/a4bdc40dd2656c496defc461fc19c403c8306d9f/revo-export/dictionary.json?raw=true -O one_line.json
$ python -m json.tool <one_line.json >pretty_printed.json

These are two JSON files with the same content. one_line.json is 18MiB of JSON without any newlines. pretty_printed.json has newlines and whitespace added, making it 41MiB.

However, the bigger file split over many lines is much faster to open in Emacs, both in Javascript mode and Fundamental mode.

Why does Emacs have such poor performance with long lines, since it's actually fewer bytes? Is there anything I can do to improve performance without reformatting the data outside of Emacs?

  • 2
    Not really an answer but might be of use: View Large Files (vlf) is a minor mode that is aimed to help with editing large files by loading them in batches. Disclaimer: I've never used it and I don't know whether it handles long lines in batches too. – elemakil Oct 2 '14 at 10:18
  • 3
    Knowing this kind of behaviour, and especially when trying to guard myself against reading a log which spits out a long line, I often do something like $ tail -f /some/file | fold -s in a shell buffer. This isn't good for editing, obviously, but helps a lot with reading. – wvxvw Dec 18 '14 at 21:48

10 Answers 10

51

Emacs' handling of long lines is not very well optimized. For a number of operations, Emacs has to scan the whole line repeatedly. For example, to display a line, Emacs has to figure out the height of the line, which requires scanning the whole line to find the tallest glyph. Additionally, scanning for bidirectional display eats up a lot of time. You can get some additional information in, for example, the docstring of cache-long-line-scans (renamed cache-long-scans in 24.4).

You can try and see if setting bidi-paragraph-direction to left-to-right improves the speed for you [setting bidi-display-reordering to nil, does more or less the same but is only meant for internal/debugging purposes]. This removes one significant contributor to line scans, but sadly not the only one.

The best option is to add newlines. You can pipe a JSON file through e.g. python -c 'import json, sys ; json.dump(json.load(sys.stdin), sys.stdout, indent=2)' to add newlines and improve readability in general.

  • 5
    Out of curiosity, is this something that can't be improved algorithmically? – PythonNut Feb 24 '16 at 3:04
  • 9
    When choosing the underlying data structure of an editor, you have to pick between certain pros and cons. Emacs uses a gap buffer, which is a highly space efficient data structure for insertion and deletion, but it makes line-based operations slower as you have to scan sequentially for a newline. Emacs could use a different data structure, but that would make other operations slower. Emacs already uses a line cache, but that does not really help in all situations. So, not easily improved algorithmically, but profiling and optimizing never hurts. :-) – Jorgen Schäfer Feb 25 '16 at 8:42
  • 4
    (setq-default bidi-display-reordering nil) -- some users may not realize that this is a buffer-local variable, which may need a default setting to the extent a user wants this to be global. I wish I would have added that to my init.el years ago ... but at least it's there now. Thank you so very much!!! – lawlist May 24 '17 at 20:03
  • 1
    What am I giving up by setting bidi-display-reordering ? – Joseph Garvin May 29 at 14:22
  • 1
    The current Emacs maintainer, Eli, who wrote the BIDI code, writes this about switching off bidi-display-reordering: "One comment I have is that disabling bidi-display-reordering … puts the display engine in a state that is not being tested, and can cause inconsistencies and even bugs (because some portions of the code were written under the assumption that this variable is never nil)." – Clément Jul 13 at 21:01
18

I did some brief experiments with this using a minified copy of jquery. font-lock-mode and flycheck-mode both contributed to slowness, as did js2-mode, and prettify-symbols-mode. line-number-mode and column-number-mode had minor effect. Once I had turned off all the different modes though the performance was relatively snappy. Use C-h m and start disabling different modes that are enabled, or try just switch to fundamental-mode.

Interestingly using hexl-mode I could fly through the file without any issue, though obviously columns were quite short. Unfortunately visual-line-mode really slowed things down.

My guess is that the syntax table is happy to stop processing at line endings, and when it's all on one line it has to reparse everything on every update.

  • 2
    Can you open a bug report on Flycheck's tracker? I'm pretty sure we don't want long lines causing issues, and Emacs + Flycheck shouldn't be worse than Emacs (which is still pretty bad). – Clément Apr 14 '16 at 12:57
16

I've uploaded http://www.emacswiki.org/emacs/OverLongLineMode

This library enables you to set simple line-length thresholds beyond which a variant of fundamental-mode will be used for a file instead of its normal mode (for programming modes only).

Potentially something along these lines could be added to Emacs by default, but this can be an interim workaround for the primary problem of Emacs slowing to a crawl upon encountering such a file.

n.b. This is an improvement upon the code I initially posted in this answer, but still a work-in-progress. Testing has been minimal. Comments are welcomed.

Suggestions for other (besides css-mode) non-prog-mode-derived major modes to support by default are also welcomed.

  • 1
    Now further improved, and shamefully renamed to so-long.el :) (the above link will redirect). There's more that could be done with this, but it's 100% functional and useful as-is. – phils Jan 22 '16 at 8:49
  • This is a really nice solution (would love to see it on MELPA), but my Emacs instance is still extremely slow when opening one_line.json. I think it would be significantly faster if it didn't first activate original the major mode. – Wilfred Hughes Nov 7 '17 at 14:19
  • 3
    Re-reading this and using your one_line.json file from the question, I gave up waiting for default-config Emacs 25.3 and 26.0.91 to respond after asking them to open that file (after waiting more than a minute), whereas my own config with so-long.el active opened the file in under 2 seconds. Actually editing the file is still hugely problematic (e.g. trying to move to the 'next line' will take an extremely long time), but nevertheless this does restore my faith in the usefulness of the library I wrote, so I should resume my plans to add it to GNU ELPA... – phils Jan 24 '18 at 14:15
  • 1
    Is it in (M)ELPA yet? – binki Jun 5 '18 at 14:23
  • 3
    Status report: version 1.0 of so-long.el (with numerous enhancements) is included in the current development versions of Emacs 27, and will be available (for earlier versions of Emacs) via GNU ELPA sometime in the near future. – phils Jul 13 at 10:35
7

I expect you'll find that the difference is due to font-lock. When fontification is to be performed on the subset of the file which is visible in the window, it proceeds by first extending the fontification region such that it will include full semantic units. See the font-lock-extend-region-functions code for this. It's common for this to include extending the region to include full lines. When the lines are extremely long, this can lead to fontification being performed across a very much larger chunk of content than is actually visible.

Additionally, when newlines themselves have semantic information, their absence may sometimes mean that regexp patterns for font lock have to scan further in order to determine whether they match or not.

7

I usually unroll long lines and indent by tags (like HTML, XML, JSON).

In order to make such operation possible I add:

(setq line-number-display-limit large-file-warning-threshold)
(setq line-number-display-limit-width 200)

(defun my--is-file-large ()
  "If buffer too large and my cause performance issue."
  (< large-file-warning-threshold (buffer-size)))

(define-derived-mode my-large-file-mode fundamental-mode "LargeFile"
  "Fixes performance issues in Emacs for large files."
  ;; (setq buffer-read-only t)
  (setq bidi-display-reordering nil)
  (jit-lock-mode nil)
  (buffer-disable-undo)
  (set (make-variable-buffer-local 'global-hl-line-mode) nil)
  (set (make-variable-buffer-local 'line-number-mode) nil)
  (set (make-variable-buffer-local 'column-number-mode) nil) )

(add-to-list 'magic-mode-alist (cons #'my--is-file-large #'my-large-file-mode))

I split line by regex, for XML it: C-M-% >< RET >NL< RET !.

After Emacs split long lines - it is possible to enable many *-modes and re-indent code.

For the note: How to prevent slow-down when an inferior processes generates long lines?

4

I created my own solution for this problem here: https://github.com/rakete/too-long-lines-mode

I was not satisfied with phils solution which switches a buffer with very long lines to fundamental-mode, I wanted a solution that lets me keep syntax highlighting and other major-mode features. So I created a minor-mode that uses overlays to hide most characters of overly long lines.

That works around the problem and makes emacs usable even in buffers with very long lines, without having to fall back to fundamental-mode.

2

In my Emacs setup I have a mode with custom fontification, i.e. where I set font-lock-defaults. A single page down would use 30 seconds to display part of 30000 character line. This slow down was fixed by reducing regexp backtracking. Instead of:

  (".* ended with an incomplete command*" 0 font-lock-comment-face)

do this

  ("^.\{1,80\} ended with an incomplete command*" 0 font-lock-comment-face)
  • This is not an answer to the question, which is not specifically about font-lock-defaults or regexp matching. – Drew Mar 26 '16 at 21:21
  • 2
    @Drew Less than ideal regex is making font-lock slow on long lines though... – wasamasa Mar 27 '16 at 0:02
  • 1
    @wasamasa: Yes. The question itself is too broad, IMO. There are plenty of things that can slow Emacs down (and for which actions?) when long lines are involved. – Drew Mar 27 '16 at 0:06
  • 4
    I don't think the question is to broad ("why do long lines make Emacs slow")? Nor do I think that the answer doesn't address the question ("one possible reason are suboptimal regexps"). Other answers can address other reasons. Opening a file with long lines isn't to broad a topic just because that might be problematic for a variety of reasons, sometimes you have such files and you have to look at them, preferably using Emacs. – tarsius Mar 27 '16 at 11:13
1

In my shell-mode buffers (M-x shell), I find myself piping to sed -r 's/(.{2000}).*/\1/' -u to avoid long lines.

  • This answers the second part of the question: how to improve performance. It does not address the first part (which is OK): "Why does Emacs have such poor performance with long lines?" – Drew Nov 7 '17 at 2:27
0

I use the following function for opening in dired-mode large files with long lines:

(defun dired-find-file-conservatively ()
   (interactive)
   (let ((auto-mode-alist nil))
     (dired-find-file)
     ;; disable costly modes
     (fundamental-mode)
     (setq-local bidi-display-reordering nil)
     (when (boundp 'smartparens-mode)
       (smartparens-mode -1))))

(define-key dired-mode-map (kbd "S-<return>") 'dired-find-file-conservatively)
0

Here is a workaround, taken from emacs-devel:

(add-hook 'find-file-hook
          (defun my-find-file-care-about-long-lines ()
            (save-excursion
              (goto-char (point-min))
              (when (and (not (eq major-mode 'image-mode))
                         (search-forward-regexp ".\\{2000\\}" 50000 t)
                         (y-or-n-p "Very long lines detected - enable 
longlines-mode? "))
                (require 'longlines)
                (longlines-mode +1)))))
  • In Emacs as of 24.4 the longlines-mode got marked as obsoleted by visual-line-mode. – Alexander I.Grafov May 31 at 21:52
  • However the two features do very different things behind the scenes, and visual-line-mode does not help with the issue in question, whereas longlines-mode does. For this reason, I expect that longlines.el will be restored to a non-deprecated status. – phils Jul 16 at 3:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.