How to use xref-find-references on Elisp files without "No references found for" errors

Question

I am using GNU Emacs 26.3 on Ubuntu 19.10.

I have a directory full of Emacs Lisp files with the .el file extension. All of them are loaded into the currently running Emacs. I can open up one of those files, put point on a function that is being called, and then use M-. to go to that definition. That works. But when I use M-? (which is bound to xref-find-references) I get an error of the form:

user-error: No references found for: xxx

where xxx was the symbol at point.

I thought this was due to not using a TAGS file, so I generated one using:

cd ~/my_emacs_lisp_dir1
etags -l lisp ../my_emacs_lisp_dir1/*.el ../my_emacs_lisp_dir2/*.el

And that did work as it produced a TAGS file where I expected it to be which is in ~/my_emacs_lisp_dir1/TAGS.

In Emacs, I visited that TAGS file using visit-tags-table. And then tried M-? on the symbol again, and it still failed with the above error.

I then executed xref-etags-mode from within that same .el buffer and tried again. Still I got the same error.

Opening up the TAGS file and searching for the symbol only turns up what I believe to be an indication of which file a symbol is defined in, but I don't see how it could record references to where that symbol is referenced from. So that may be the problem, and if so, how do I make it tell me which functions and files refer to the symbol at point?

Update 1: It works for standalone files, but not in my Git-controlled HOME directory

I noticed that it does work if I do something silly like this:

rm -rf /tmp/xref-test-dir
mkdir -p /tmp/xref-test-dir
cd /tmp/xref-test-dir


cat > func-1.el <<'EOF'
(defun xxx-func-1 ()
  (xxx-func-2))
EOF

cat > func-2.el <<'EOF'
(defun xxx-func-2 ()
  (message "xxx-func-2 called"))
EOF

etags -l lisp *.el

Then go to each of the two .el files, and run eval-buffer on them and then point point on the call to xxx-func-2 and it at least did something that I think is correct: It showed a buffer with this info:

/tmp/xref-test-dir/func-1.el
2:   (xxx-func-2))
/tmp/xref-test-dir/func-2.el
1: (defun xxx-func-2 ()
2:   (message "xxx-func-2 called"))

I noticed that somewhere along the way I was prompted for a project directory and I supplied it /tmp/xref-test-dir.

But I got no such prompting from when I did it inside my ~/my_emacs_lisp_dir1 directory.

I dug even deeper into the xref-backend-references function, and found some code dealing with projects. So I went back to the .el file inside ~/my_emacs_lisp_dir1 directory, and ran this via eval-expression:

(project--find-in-directory ".")

and it returned this:

(vc . "~/")

that sort of makes sense because both ~/my_emacs_lisp_dir1 and ~/my_emacs_lisp_dir2 are within a Git repo.

But I'm still stuck because I don't know how to tell "it" to just look into those two directories, or even look into the files in that same directory.

Answer 1

xref provides UI for code navigation. The actual functionalities are provided by the backend. In you case, the backend is etags.

If you open TAGS file, its format is very simple. It has only the information of tag definition. So you can not find reference of tag.

So you got two solutions,

Solution 1, Install lsp-mode and some language server, hope they will have implementation for finding references.

Solution 2, grep project with other packages. You can try counsel-git-grep or counsel-rg.

You can read Emacs Lisp code by yourself to confirm my point.

By default, xref--find-xrefs is called. Its definition,

(funcall (intern (format "xref-backend-%s" kind))
                        (xref-find-backend)
                        arg)

kind is references, so xref-backend-references is called. Here's its definition,

(cl-defgeneric xref-backend-references (_backend identifier)
  "Find references of IDENTIFIER.
The result must be a list of xref objects.  If no references can
be found, return nil.

The default implementation uses `semantic-symref-tool-alist' to
find a search tool; by default, this uses \"find | grep\" in the
`project-current' roots."
  (cl-mapcan
   (lambda (dir)
     (xref-collect-references identifier dir))
   (let ((pr (project-current t)))
     (append
      (project-roots pr)
      (project-external-roots pr)))))

Notice the parameter _backend is ignored. I'm not sure what's the value of semantic-symref-tool-alist, maybe it's set up by you or the 3rd party packages you installed.

Definition of semantic-symref-tool-alist,

(defvar semantic-symref-tool-alist
  '( ( (lambda (rootdir) (file-exists-p (expand-file-name "GPATH" rootdir))) .
       global)
     ( (lambda (rootdir) (file-exists-p (expand-file-name "ID" rootdir))) .
       idutils)
     ( (lambda (rootdir) (file-exists-p (expand-file-name "cscope.out" rootdir))) .
       cscope )
    )
  "Alist of tools usable by `semantic-symref'.
Each entry is of the form:
   ( PREDICATE . KEY )
Where PREDICATE is a function that takes a directory name for the
root of a project, and returns non-nil if the tool represented by KEY
is supported.

If no tools are supported, then 'grep is assumed.")

So you could find real references if and only if you install GNU Global or GNU ID Utils or Cscope.

The documentation does mention the grep will be used as fallback. But I tested with a real project. The grep didn't happen.

Answer 2

Note: This is a very complicated solution. See How to use xref-find-references on Elisp files without "No references found for" errors for when I came to my senses and simplified things.

Problems and fixes

I found multiple things that I did that tripped me up:

1. I had an ~/ID file that is generated using ID Utils. xref, or something beneath it, was trying to use ID as its backend, because it is much, much faster than grep, of course. However, the way I have to generate these ID files is by disabling all language specific parsing in the lang-map file I use with gid (see the mkid lang-map below). But mkid for text treats dashes as punctuation, and thus, not a part of a symbol name. And, most if not all of the Emacs Lisp symbols I define have dashes. Thus, when I was searching using M-?, it used lid to find them and got nothing. Even when I called mkid with a lang-map that marked *.el files as lisp, something still wanted to search all of ~/ which is not what I need (see next item).

2. Removing the ID files caused it to work, but it took quite a bit longer time than I know it should if it was only to search my small bit of Elisp code. It turns out that the slowness was caused by the elisp definitions that include everything in the load-path which is a ton of directories. So that had to be changed to limit the search to only one directory (containing subdirectories). It is probably correct to do this by default (if you intend to search for everything that Emacs knows about, which is useful in some contexts), however, not in this special case where I have only two directories.

My ~/elisp/.dir-locals.el file

;; -*- fill-column: 100 -*-
;;
;; This is needed so that M-? (runs the command xref-find-references) can not trip over ID files
;; (created by my use of mkid from the idutils package).  When I create these ID files, I cannot
;; specify things for Elisp that uses "-"s in the symbol names, because I use gid to find things
;; everywhere, even text files. When searching for an Elisp symbol that contains dashes, it cannot
;; find it inside the .el files referenced by that ID file. So the ID backend has to be deactiviated
;; for searching purposes for M-?.
;;
((emacs-lisp-mode . ((eval . (let ((my-elisp-dir "~/elisp"))
                   ;;
                   ;; By default, the "vc" part of the xref code finds ~/.git and thus
                   ;; assumes the entire Git-controlled directory, ~/, is to be
                   ;; searched. But that results is a HUGE amount of search time
                   ;; (because grep is used, and not something like ID or Global etc.)
                   ;; because ~/ also has ~/emacs.d/ which houses all of the packages I
                   ;; use. Instead, we have to force that "off" by forcing it to only
                   ;; look inside my-elisp-dir.
                   ;;
                   (eval `(defun my-elisp-project-find (dir)
                    (cons 'vc
                          ;; Use file-name-as-directory to avoid the assert inside
                          ;; xref-collect-references:
                          ,(file-name-as-directory my-elisp-dir))))
                   ;;
                   ;; Make project-find-functions buffer-local as this should not apply
                   ;; anywhere else but here:
                   ;;
                   (make-variable-buffer-local 'project-find-functions)
                   (add-hook 'project-find-functions #'my-elisp-project-find)
                               ;;
                               ;; Do not let the xref code search all of the other directories in
                               ;; the elisp "load-path", which tends to be huge, as it includes both
                               ;; the site-lisp area in the installed Emacs *and* all of the stuff
                               ;; inside ~/emacs.d from Elpa etc. I intend to be searching for my
                               ;; personally-defined Elisp functions when I'm doing my own personal
                               ;; development.
                               ;;
                   (setq-local project-vc-external-roots-function
                       `(lambda ()
                          (mapcar
                           ;; Something is buggy in some lower level emacs lisp
                           ;; code that does not accept "~" in the paths. So
                           ;; hack around that bug with expand-file-name:
                           #'expand-file-name
                           (list ,my-elisp-dir)))))))))

My mkid lang-map

# Welcome to the mkid language mapper.
#
# The format of each line is:
#
#       <pattern> <language> [options]
#
# Filenames are matched top-to-bottom against the patterns, and the
# first match is chosen.  The special language `IGNORE' means that
# this file should be ignored by mkid.  The options are
# language-specific command-line options to mkid.
#
# If a file name doesn't match any pattern, it is assigned the default
# language.  The default language may be specified here with the
# special pattern `**', or overridden from the mkid command-line with
# the `--default-lang=LANG' option.
#
# The special pattern `***' means to include the named file that
# immediately follows.  If no file is named, then the default system
# language mapper file (i.e., this file) is included.

# Default language
**          text

# Ignore PDF files as they are binary and don't provide much value in
# the output, and cause mkid to chew up CPU:
*.pdf           IGNORE