Best way to implement dynamic highlighting

Question

I am working on a minor mode that will use Slime to highlight defined symbols when editing Lisp code. The code is here. It is based on the code for Highlight-Defined which registers a matcher function for font lock. The function attempts to parse the next symbol (hackily) and then uses swank to test if it is defined. Currently this process is extremely slow. I'm wondering if there are best practices for font locking that will increase performance. For instance, is there some standard way to parse symbols (right now the same symbol gets parsed multiple times sometimes)? Is this just a bad approach and instead should I try to get a list of defined symbols and then add regexs explicitly for them?

Specifically I have registered the following font-lock matcher and I am wondering if there is some way to make this run at a reasonable speed or should I give up and try a different approach.

(defun shigh-highlight (_limit)
  "Highlight Lisp functions and/or variables that are currently defined.
Use as a font-lock MATCHER function for `slime-highlight-mode'."
  ;; If your code uses any of these variable names too then bad - they will be highlighted:
  ;; `shigh-found', `shigh-highlight-type', `shigh-obj', and `shigh-opoint'.
  (let ((shigh-opoint  (point))
        (shigh-epoint nil)
        (shigh-found   nil)
        (move-error nil))
    (with-syntax-table lisp-mode-syntax-table 
      (skip-chars-forward "()#` []")
      (while (and (not shigh-found) (not (eobp)) (< (point) _limit))
        (condition-case () 
            (save-excursion
              (setq shigh-opoint  (point))
              (forward-sexp 1)
              (setq shigh-epoint (point)))
          (error (progn (setf move-error t) (setq shigh-epoint (point)))))
        (let ((shigh-str (trim-string
                          (upcase (buffer-substring-no-properties shigh-opoint 
                                                                  shigh-epoint)))))
          (if (and (not move-error) 
                   (> (length shigh-str) 0)
                   (symbolp (read shigh-str)))
            (let ((bndp (slime-boundp shigh-str))
                  (fbndp (slime-fboundp shigh-str))
                  (lclp (slime-local-symbolp shigh-str)))
             (if (and 
                  (not (member shigh-str '("NIL" "T")))
                                        ;(not (keywordp hdefd-obj))
                  (or (and (memq shigh-highlight-type '(fns-and-vars functions))
                           fbndp
                           (or (and (not shigh-highlight-cl) lclp)
                               shigh-highlight-cl))
                      (and (memq shigh-highlight-type '(fns-and-vars variables))
                           bndp
                           (or (and (not shigh-highlight-cl) lclp)
                               shigh-highlight-cl))
                      (and (eq shigh-highlight-type 'undefined)
                           (not fbndp)
                           (not bndp))))
                 (progn
                   (setq shigh-face
                         (if (and (memq shigh-highlight-type 
                                        '(fns-and-vars functions))
                                  fbndp)
                             (if lclp
                                 'shigh-functions
                               'shigh-clfunctions)
                           (if (eq shigh-highlight-type 'undefined)
                               'shigh-undefined
                             (if lclp
                                 'shigh-variables
                               'shigh-clvariables))))
                   (forward-sexp 1)
                   (setq shigh-found t))
               (progn
                 (if (looking-at "\\(\\sw\\|\\s_\\)")
                     (forward-sexp 1)
                   (forward-char 1))
                 (skip-chars-forward "()#`")
                 (setf move-error nil))))
            (progn
                 (if (looking-at "\\(\\sw\\|\\s_\\)")
                     (forward-sexp 1)
                   (forward-char 1))
                 (skip-chars-forward "()#`")
                 (setf move-error nil)))))
      (when shigh-found
       (set-match-data (list shigh-opoint shigh-epoint)))
      shigh-found)))

Answer 1

As I understand your question the main problem is that the function

swank:eval-and-grab-output works slowly, and you call it around 3 times for each symbol in region you need to "font-lock".

It may be better to first parse all the region needed to font-lock and extract substrings witch may be symbols (maybe with information about their place), and then, pass that list of symbols to Common Lisp to find the true symbols that need to be fontified.

I don't no how exactly font-lock mode work (it seems to me it depends on a lot of variables) but, For example, if the variable font-lock-support-mode is eq to jit-lock-mode, you can use (jit-lock-register #'fun t) where fun is a function "will be called with two arguments START and END indicating the region that needs to be (re)fontified" (c) to extract data you need from region going to be font-lock.

For example, it seems to me, it can be written like this (just to explane what I meen).

https://github.com/Rad-Rat/scratches-to-post/blob/master/live-font-lock-for-CL-in-emacs-scratch.el