The way highlight-symbol handles itself, IMO, is not ideal. In order to be able to navigate at full speed and still have highlighting applied quickly after a pause I had to make some changes to the core functions of the package.
First, highlight symbol actually checks for a 0 delay and acts differently, this isn't needed, so you can redefine
highlight-symbol-post-command to this version which removes the 0 delay check and just always does highlighting via a timer, not directly.
(defun highlight-symbol-mode-post-command ()
"After a command, change the temporary highlighting.
Remove the temporary symbol highlighting and, unless a timeout is specified,
create the new one."
(if (eq this-command 'highlight-symbol-jump)
Second, we need to change how the timers are started for highlighting. There is no need to have the timer repeat and no need to use an idle timer over a standard timer. You can redefined
highlight-symbol-update-timer like as this:
(defun highlight-symbol-update-timer (value)
(run-with-timer value nil 'highlight-symbol-temp-highlight)))
Finally, set the delay time very small, I used .1 seconds.
(setq highlight-symbol-idle-delay .1)
With these changes I am able to move around the buffer quickly with without unnecessary highlighting occurring and slowing things down along the way. Instead highlighting occurs promptly after I stop moving.