Where can I find some thorough documentation / a specification for the Emacs ctags format? The ultimate goal here is to make it easier to generate a tags file in parallel (preferably using a tool like xargs or GNU Parallel).

Somewhat confusingly, there are two very different formats commonly called "ctags". One of which is used by vi and its workalikes.

example (from Perl 5 source code using ctags -f- op.c | head -n 3):

AAS_DANGEROUS   op.c    /^    AAS_DANGEROUS       = 0x080, \/* an op (other than the above)$/;" e   enum:__anon1file:
AAS_DEFAV   op.c    /^    AAS_DEFAV           = 0x200  \/* contains just a single '@_' on RHS *\/$/;"   e   enum:__anon1    file:
AAS_LEX_AGG op.c    /^    AAS_LEX_AGG         = 0x008, \/* @lexical or %lexical aggregate *\/$/;"   e   enum:__anon1    file:

This format has the advantage of being line-oriented and storing the symbol first, so work can be efficiently parallelized

find . -name '*.c' -o -name '*.h' \
   | xargs -I% -n100 -P8 ctags -f- % \
   | env LC_ALL=C sort > tags

By contrast, the Emacs variant of the ctags format is a little bit harder to decipher since, among other things, it uses non-printable characters internally.

File Edit Options Buffers Tools Hexl Help
87654321  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789abcdef
00000000: 0c0a 6f70 2e63 2c31 3935 3131 0a23 6465  ..op.c,19511.#de
00000010: 6669 6e65 2050 4552 4c5f 494e 5f4f 505f  fine PERL_IN_OP_
00000020: 437f 5045 524c 5f49 4e5f 4f50 5f43 0131  C.PERL_IN_OP_C.1

It also uses byte offsets instead of a search string. It also mentions the file name once at the start of a "stanza" corresponding to each file rather than having each line stand on its own.

The closest I was able to find to documentation of the internal format (rather than the kinds of pattern in source codes that emacs recognizes) was https://stackoverflow.com/q/1990579/931154 .

  • I'd just look at the source. – wasamasa Sep 2 '17 at 12:27

So, as it turns out, this is actually a pretty quick way to parallelize the work for generating a tags file. Here are two strategies that I've found effective. You can rewrite either shell fragment to use xargs instead of parallel.

  1. Use GNU Parallel with the --files option to generate separate temporary files for each job.
  2. swap ^L and \n when processing the files and then swap them back when you're done aggregating the output streams of separate ctags jobs.

(1) This relies on the --files feature of GNU Parallel, which redirects the stdout of each job into a separate temporary file and then, after all the jobs are finished, writes each temporary file to stdout.

cat $(find . |
    awk '/[.](h|cc|hh|cpp|hpp|c)$/ {print}' |
    parallel --no-notice -n100 --files 'ctags -e -f - {}'
) > TAGS

(2) Different "stanzas" corresponding to a source file in the etags format are separated using the ^L character (\f in another notation). You can swap ^L and \n, use line-oriented tools, and then swap them back (using tr '\n\f' '\f\n'). The result will look something like this if you use GNU Parallel. (Sorting the resulting lines is not strictly necessary)

find . |
    awk '/[.](h|cc|hh|cpp|hpp|c)/ {print}' |
    parallel --no-notice -n100 'ctags -e -f - {}' '|' "tr '\\n\\f' '\\f\\n'" | 
    env LC_ALL=C sort |
    tr '\n\f' '\f\n' >TAGS

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.