up vote 50 down vote favorite
14

I see hugely varied performance depending on how many newlines there are in the file I'm visiting.

Here's an example. I have two JSON files:

$ wget https://github.com/Wilfred/ReVo-utilities/blob/a4bdc40dd2656c496defc461fc19c403c8306d9f/revo-export/dictionary.json?raw=true -O one_line.json
$ python -m json.tool <one_line.json >pretty_printed.json

These are two JSON files with the same content. one_line.json is 18MiB of JSON without any newlines. pretty_printed.json has newlines and whitespace added, making it 41MiB.

However, the bigger file split over many lines is much faster to open in Emacs, both in Javascript mode and Fundamental mode.

Why does Emacs have such poor performance with long lines, since it's actually fewer bytes? Is there anything I can do to improve performance without reformatting the data outside of Emacs?

share|improve this question
  • 2
    Not really an answer but might be of use: View Large Files (vlf) is a minor mode that is aimed to help with editing large files by loading them in batches. Disclaimer: I've never used it and I don't know whether it handles long lines in batches too. – elemakil Oct 2 '14 at 10:18
  • 3
    Knowing this kind of behaviour, and especially when trying to guard myself against reading a log which spits out a long line, I often do something like $ tail -f /some/file | fold -s in a shell buffer. This isn't good for editing, obviously, but helps a lot with reading. – wvxvw Dec 18 '14 at 21:48
up vote 36 down vote accepted

Emacs' handling of long lines is not very well optimized. For a number of operations, Emacs has to scan the whole line repeatedly. For example, to display a line, Emacs has to figure out the height of the line, which requires scanning the whole line to find the tallest glyph. Additionally, scanning for bidirectional display eats up a lot of time. You can get some additional information in, for example, the docstring of cache-long-line-scans (renamed cache-long-scans in 24.4).

You can try and see if setting bidi-display-reordering to nil improves the speed for you. This removes one significant contributor to line scans, but sadly not the only one.

The best option is to add newlines. You can pipe a JSON file through e.g. python -c 'import json, sys ; json.dump(json.load(sys.stdin), sys.stdout, indent=2)' to add newlines and improve readability in general.

share|improve this answer
  • 4
    Out of curiosity, is this something that can't be improved algorithmically? – PythonNut Feb 24 '16 at 3:04
  • 7
    When choosing the underlying data structure of an editor, you have to pick between certain pros and cons. Emacs uses a gap buffer, which is a highly space efficient data structure for insertion and deletion, but it makes line-based operations slower as you have to scan sequentially for a newline. Emacs could use a different data structure, but that would make other operations slower. Emacs already uses a line cache, but that does not really help in all situations. So, not easily improved algorithmically, but profiling and optimizing never hurts. :-) – Jorgen Schäfer Feb 25 '16 at 8:42
  • 1
    (setq-default bidi-display-reordering nil) -- some users may not realize that this is a buffer-local variable, which may need a default setting to the extent a user wants this to be global. I wish I would have added that to my init.el years ago ... but at least it's there now. Thank you so very much!!! – lawlist May 24 '17 at 20:03
  • In my case it was not a big improvent (really long json lines with base64 documents body) but helps a lot on beign freezing – anquegi Apr 12 at 16:05
up vote 13 down vote

I did some brief experiments with this using a minified copy of jquery. font-lock-mode and flycheck-mode both contributed to slowness, as did js2-mode, and prettify-symbols-mode. line-number-mode and column-number-mode had minor effect. Once I had turned off all the different modes though the performance was relatively snappy. Use C-h m and start disabling different modes that are enabled, or try just switch to fundamental-mode.

Interestingly using hexl-mode I could fly through the file without any issue, though obviously columns were quite short. Unfortunately visual-line-mode really slowed things down.

My guess is that the syntax table is happy to stop processing at line endings, and when it's all on one line it has to reparse everything on every update.

share|improve this answer
  • 2
    Can you open a bug report on Flycheck's tracker? I'm pretty sure we don't want long lines causing issues, and Emacs + Flycheck shouldn't be worse than Emacs (which is still pretty bad). – Clément Apr 14 '16 at 12:57
up vote 11 down vote

I've uploaded http://www.emacswiki.org/emacs/OverLongLineMode

This library enables you to set simple line-length thresholds beyond which a variant of fundamental-mode will be used for a file instead of its normal mode (for programming modes only).

Potentially something along these lines could be added to Emacs by default, but this can be an interim workaround for the primary problem of Emacs slowing to a crawl upon encountering such a file.

n.b. This is an improvement upon the code I initially posted in this answer, but still a work-in-progress. Testing has been minimal. Comments are welcomed.

Suggestions for other (besides css-mode) non-prog-mode-derived major modes to support by default are also welcomed.

share|improve this answer
  • Now further improved, and shamefully renamed to so-long.el :) (the above link will redirect). There's more that could be done with this, but it's 100% functional and useful as-is. – phils Jan 22 '16 at 8:49
  • This is a really nice solution (would love to see it on MELPA), but my Emacs instance is still extremely slow when opening one_line.json. I think it would be significantly faster if it didn't first activate original the major mode. – Wilfred Hughes Nov 7 '17 at 14:19
  • @WilfredHughes It was destined for GNU ELPA, and then I started to doubt whether it was as beneficial as I'd first thought (none of my original examples were particularly problematic with just the standard Emacs libraries), so I'd be keen to have examples of it proving its worth. I'd also be interested to play with that one_line.json (or equivalent?). – phils Nov 7 '17 at 20:22
  • 3
    Re-reading this and using your one_line.json file from the question, I gave up waiting for default-config Emacs 25.3 and 26.0.91 to respond after asking them to open that file (after waiting more than a minute), whereas my own config with so-long.el active opened the file in under 2 seconds. Actually editing the file is still hugely problematic (e.g. trying to move to the 'next line' will take an extremely long time), but nevertheless this does restore my faith in the usefulness of the library I wrote, so I should resume my plans to add it to GNU ELPA... – phils Jan 24 at 14:15
  • 1
    Is it in (M)ELPA yet? – binki Jun 5 at 14:23
up vote 4 down vote

I expect you'll find that the difference is due to font-lock. When fontification is to be performed on the subset of the file which is visible in the window, it proceeds by first extending the fontification region such that it will include full semantic units. See the font-lock-extend-region-functions code for this. It's common for this to include extending the region to include full lines. When the lines are extremely long, this can lead to fontification being performed across a very much larger chunk of content than is actually visible.

Additionally, when newlines themselves have semantic information, their absence may sometimes mean that regexp patterns for font lock have to scan further in order to determine whether they match or not.

share|improve this answer
up vote 3 down vote

I usually unroll long lines and indent by tags (like HTML, XML, JSON).

In order to make such operation possible I add:

(setq line-number-display-limit large-file-warning-threshold)
(setq line-number-display-limit-width 200)

(defun my--is-file-large ()
  "If buffer too large and my cause performance issue."
  (< large-file-warning-threshold (buffer-size)))

(define-derived-mode my-large-file-mode fundamental-mode "LargeFile"
  "Fixes performance issues in Emacs for large files."
  ;; (setq buffer-read-only t)
  (setq bidi-display-reordering nil)
  (jit-lock-mode nil)
  (buffer-disable-undo)
  (set (make-variable-buffer-local 'global-hl-line-mode) nil)
  (set (make-variable-buffer-local 'line-number-mode) nil)
  (set (make-variable-buffer-local 'column-number-mode) nil) )

(add-to-list 'magic-mode-alist (cons #'my--is-file-large #'my-large-file-mode))

I split line by regex, for XML it: C-M-% >< RET >NL< RET !.

After Emacs split long lines - it is possible to enable many *-modes and re-indent code.

For the note: How to prevent slow-down when an inferior processes generates long lines?

share|improve this answer
up vote 1 down vote

In my Emacs setup I have a mode with custom fontification, i.e. where I set font-lock-defaults. A single page down would use 30 seconds to display part of 30000 character line. This slow down was fixed by reducing regexp backtracking. Instead of:

  (".* ended with an incomplete command*" 0 font-lock-comment-face)

do this

  ("^.\{1,80\} ended with an incomplete command*" 0 font-lock-comment-face)
share|improve this answer
  • This is not an answer to the question, which is not specifically about font-lock-defaults or regexp matching. – Drew Mar 26 '16 at 21:21
  • 1
    @Drew Less than ideal regex is making font-lock slow on long lines though... – wasamasa Mar 27 '16 at 0:02
  • 1
    @wasamasa: Yes. The question itself is too broad, IMO. There are plenty of things that can slow Emacs down (and for which actions?) when long lines are involved. – Drew Mar 27 '16 at 0:06
  • 2
    I don't think the question is to broad ("why do long lines make Emacs slow")? Nor do I think that the answer doesn't address the question ("one possible reason are suboptimal regexps"). Other answers can address other reasons. Opening a file with long lines isn't to broad a topic just because that might be problematic for a variety of reasons, sometimes you have such files and you have to look at them, preferably using Emacs. – tarsius Mar 27 '16 at 11:13
up vote 1 down vote

In my shell-mode buffers (M-x shell), I find myself piping to sed -r 's/(.{2000}).*/\1/' -u to avoid long lines.

share|improve this answer
  • This answers the second part of the question: how to improve performance. It does not address the first part (which is OK): "Why does Emacs have such poor performance with long lines?" – Drew Nov 7 '17 at 2:27
up vote 1 down vote

I created my own solution for this problem here: https://github.com/rakete/too-long-lines-mode

I was not satisfied with phils solution which switches a buffer with very long lines to fundamental-mode, I wanted a solution that lets me keep syntax highlighting and other major-mode features. So I created a minor-mode that uses overlays to hide most characters of overly long lines.

That works around the problem and makes emacs usable even in buffers with very long lines, without having to fall back to fundamental-mode.

share|improve this answer
up vote 0 down vote

I use the following function for opening in dired-mode large files with long lines:

(defun dired-find-file-conservatively ()
   (interactive)
   (let ((auto-mode-alist nil))
     (dired-find-file)
     ;; disable costly modes
     (fundamental-mode)
     (setq-local bidi-display-reordering nil)
     (when (boundp 'smartparens-mode)
       (smartparens-mode -1))))

(define-key dired-mode-map (kbd "S-<return>") 'dired-find-file-conservatively)
share|improve this answer

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.