why is a let binding is ignored in compiled function?

Question

I have this function which works perfectly in an uncompiled function. It should let-bind the variable bibtex-completion-bibliography to the result of (org-ref-find-bibliography).

    (defun org-ref-valid-keys ()
      "Return a list of valid bibtex keys for this buffer.
    This is used a lot in `org-ref-cite-activate' so it needs to be
    fast, but also up to date."
      ;; this seems to be needed, but we don't want to do this every time
      (unless bibtex-completion-display-formats-internal
        (bibtex-completion-init))
    
      ;; I don't fully understand what is happening here.
      (let ((bibtex-completion-bibliography (org-ref-find-bibliography)))
        (cl-loop for entry in (bibtex-completion-candidates)
             collect
             (cdr (assoc "=key=" (cdr entry))))))

What should happen is (org-ref-find-bibliography) gets a list of bibliography files if they are defined in a buffer (it defaults to a variable setting if they are not), and then let-binds that list to bibtex-completion-bibliography, and the keys are extracted from (bibtex-completion-candidates) which uses bibtex-completion-bibliography to get keys from those files. This works fine when the function is not compiled.

When the function is compiled, the keys end up coming from the default list of bibtex files, and not from the list defined by (org-ref-find-bibliography) (which still does the right thing). It is as if the let binding does not work.

If I debug the function, or re-evaluate it, it works fine, it only does not work when called from the compiled function. It also works if I delete all the .elc files in the package.

All the bibtex-completion* functions and variables are defined in the external library bibtex-completion.

I have -*- lexical-binding: t; -*- in the first line of that file.

What could be causing this?

extra details:

GNU Emacs 28.0.50 (build 1, x86_64-apple-darwin20.6.0, NS appkit-2022.60 Version 11.5 (Build 20G71)) of 2021-07-29

Answer 1

You're compiling the file using lexical binding, so I suspect the problem is that your file does neither of the following:

• (defvar bibtex-completion-bibliography)
• (require 'bibtex-completion)

either of which would tell the code in your file that bibtex-completion-bibliography is a dynamic variable.

Without that information, the following creates a lexical binding for bibtex-completion-bibliography (which is distinct from the symbol by that name):

  (let ((bibtex-completion-bibliography (org-ref-find-bibliography)))
    (cl-loop for entry in (bibtex-completion-candidates)

You then call (bibtex-completion-candidates) which is undoubtedly looking at the dynamic variable bibtex-completion-bibliography which was not set by your code.

When you're testing all of this interactively, I imagine that bibtex-completion-bibliography is getting declared as dynamic along the way, thus causing your code to do what you had intended.

In summary, when you're using lexical binding, it's crucial that all dynamic variables are declared as such, either by requireing the library which defines them, or else by explicitly declaring them with the (defvar FOO) syntax (with only a single parameter, defvar does not define a variable FOO, but rather says that references to FOO in the same file are to the dynamic variable by that name).

---

When ‘require’ is used at top level in a file, it takes effect when you byte-compile that file as well as when you load it. This is in case the required package contains macros that the byte compiler must know about. It also avoids byte compiler warnings for functions and variables defined in the file loaded with ‘require’.

-- C-hig (elisp)Named Features

Effectively ‘require’ is automatically ‘eval-and-compile’, the package is loaded both when compiling and executing.

-- C-hig (elisp)Eval During Compile