Syntax table to match arbitrary forms with regexp

Question

I'm trying to write a small major-mode for editing pico-8 files in emacs and I had a quick question regarding best practices and matching a somewhat arbitrary form in the pico-8 cartridge files for the syntax table of the mode.

A pico-8 cartridge, in plain-text looks like this:

pico-8 cartridge // http://www.pico-8.com
version 4
__lua__
-- pico-8 api test  // by zep
-- demonstrates all pico-8
-- functions and some lua

cls() -- clear screen
rect(0,0,127,127,1)

music(0)
sfx(0, 3)

-- draw palette
x=3
rectfill(1,1,7,7,5)
for i=0,15 do
  print(i,x,2,i)
  x = x + 6 + flr(i/10)*4
end

__gfx__
0000000000ff0ff044444444bbbbbbbb000000004444444400000000000000000000000000000000000000000000000000000000000000000000000000000000
0700007000ff0ff04444444433333333000000004444444400000000000000000000000000000000000000000000000000000000000000000000000000000000
0070070000fffff042444444b3b3b3b3000000004444444400000000000000000000000000000000000000000000000000000000000000000000000000000000
0007700000f1ff104444444433333333000000004444424400000e00000000000000000000000000000000000000000000000000000000000000000000000000
...

Basically, it's some metadata, a bunch of lua code, and then some more raw data of the cart's graphics/music/sound effects.

I wrote the following regex to match lines up to and including __lua__:

"\\(.\\|\n\\)*?__lua__$"

This works to present the first three lines in a comment face.

The following regex was an attempt to match every line after __gfx__:

"__gfx__\\(.\\|\n\\)*"

I am pretty sure there's a better way to write this regex, but even so, it actually only puts __gfx__ and the following few lines into a comment face, rather than the rest of the buffer like I had hoped.

Is using regexes the best way to match arbitrary syntaxes like this though? I tried looking at some other major-modes to see how they handled it (particularly things like code blocks in markdown-mode), and it seems they also use regex for these kinds of expressions.

My full major-mode code is here: https://gist.github.com/taylskid/a81ca9343711ada4ba85c7451803185a

I also attached an image showing the comment face not correctly applying here:

Sorry if this is trivial or can easily be found somewhere else. My searches didn't get very far. Thanks in advance.

Answer 1

By default, font-lock only highlights a handful of lines at a time. In your case, it highlighted the line containing __gfx__ and the four following lines (which were correctly highlighted). It then proceeded to highlight the next couple of lines that, of course, don't match your regexp.

You can see this by using the package https://github.com/Lindydancer/highlight-refontification

The reason it appears to be working in font-lock-studio is that the entire buffer is highlighted at once.

This can be corrected using a number of techniques, including:

• Extend the highlighting region by adding a function to font-lock-extend-region-functions. Note that if the gfx section is large, this might become really slow.

• Match lines looking like hex data, regardless of context. This is not a good idea if such lines might appear in other contexts, but I doubt it. This can be done using a simple regexp.

• Match lines looking like hex data, but also check if they are preceded by __gfx__. This can be done by using a matcher function that 1) use re-search-forward to look for hex lines and 2) search backward for the nearest section label.

When it comes to the __lua__ section, it's possible to highlight it by copying it to a temporary buffer, apply lua-mode, and copy the highlighting back. This is done in code blocks in Org mode. Fortunately, there is a support package that simplifies this, see https://github.com/cpitclaudel/indirect-font-lock for more information.

Answer 2

If you need to match all the way to the end of the buffer, you ought to end your regexp with \' (i.e. "\\'") which matches the empty string at the end of the text/buffer.

You could alternatively use the alternative C-hig (elisp)POSIX Regexps support, which is guaranteed to always return the longest match (although as your first regexp includes a non-greedy operator, you wouldn't use the POSIX functions with that one).

I'm not sure whether using \' enables any optimizations in the regexp engine, such that matching "the rest of the buffer" is efficient. I would be inclined to look for a way to search only for the __gfx__ marker and then specify the start of the match and (point-max) as the region of interest.