Optimizing font lock performance

Question

I want to perform a variant of anchored font lock matching. I have function definitions that start with a list of names, and I want those names to be highlighted inside the function body.

I have created a function that does this and registered it as a jit-lock function with jit-lock-register, however, performance is pretty poor and scrolling lags in larger files.

• How can I measure performance? If I just time calling my function on a large file (with float-time before and after or with elp) I get wildly varying performance, it takes anything from 0.65 to 12 seconds. Is there a recommended way to benchmark font lock performance?
• Is there any difference in performance between an anchored matcher defined in font-lock-keywords and adding a function via jit-lock-register?

Edit: It seems that the variability in performance is related to garbage collection, invocations of my jit-lock function get successively slower with each invocation until garbage collection is run, at which point they get fast again.

Answer 1

It turns out that the wildly varying performance was related to garbage collection. Each call to the function would get slower until a garbage collection was run. With stock emacs, gc was run every couple of seconds, but I had a line in my init.el to improve startup time that set gc-cons-threshold to 20 MB, and that meant gc was run much more infrequently, causing benchmarks to report slower and slower timing until a gc was run after a couple of minutes, then times would plummet and be fast again.

After reverting to the default gc-cons-threshhold, benchmarking became easier.

I then profiled for memory with the built in profiler (M-x profiler-start), and discovered that calls to syntax-ppss caused the most allocations, so after some optimization to call syntax-ppss less often I achieved acceptable performance.

Using jit-lock-mode (adding a function via jit-lock-register) seems to be the easiest way to get multi line font locking to work reliably, so that was the method I chose.

Edit: After discovering that performance was still not good enough in very large buffers I spent a lot of time optimizing cpu use and allocation, measuring the performance improvements with the built in Emacs profiler (M-x profiler-start). However, Emacs would still stutter and hang when scrolling quickly through very large buffers. Removing the jit-lock function I registered with jit-lock-register would remove the stuttering and hangs, but profiling showed the jit-lock function to complete in around 8 ms which should be fast enough for smooth scrolling. Removing the call to jit-lock-registerand instead using a regular font-lock-keywords matcher solved the issue.

TLDR: Doing this was slow and would stutter:

(defun my-font-lock-function (start end)
"Set faces for font-lock between START and END.")

(jit-lock-register 'my-font-lock-function)

Doing this was fast and would not stutter:

(defun my-font-lock-function (start end)
"Set faces for font-lock between START and END.")

(defun my-font-lock-matcher (limit)
    (my-font-lock-function (point) limit)
   nil)

(setq font-lock-defaults
  (list 
     ...
    ;; Note that the face specified here doesn't matter since
    ;; my-font-lock-matcher always returns nil and sets the face on
    ;; its own.
    `(my-font-lock-matcher (1 font-lock-keyword-face nil))))