2

I'm trying to write a small major-mode for editing pico-8 files in emacs and I had a quick question regarding best practices and matching a somewhat arbitrary form in the pico-8 cartridge files for the syntax table of the mode.

A pico-8 cartridge, in plain-text looks like this:

pico-8 cartridge // http://www.pico-8.com
version 4
__lua__
-- pico-8 api test  // by zep
-- demonstrates all pico-8
-- functions and some lua

cls() -- clear screen
rect(0,0,127,127,1)

music(0)
sfx(0, 3)

-- draw palette
x=3
rectfill(1,1,7,7,5)
for i=0,15 do
  print(i,x,2,i)
  x = x + 6 + flr(i/10)*4
end

__gfx__
0000000000ff0ff044444444bbbbbbbb000000004444444400000000000000000000000000000000000000000000000000000000000000000000000000000000
0700007000ff0ff04444444433333333000000004444444400000000000000000000000000000000000000000000000000000000000000000000000000000000
0070070000fffff042444444b3b3b3b3000000004444444400000000000000000000000000000000000000000000000000000000000000000000000000000000
0007700000f1ff104444444433333333000000004444424400000e00000000000000000000000000000000000000000000000000000000000000000000000000
...

Basically, it's some metadata, a bunch of lua code, and then some more raw data of the cart's graphics/music/sound effects.

I wrote the following regex to match lines up to and including __lua__:

"\\(.\\|\n\\)*?__lua__$"

This works to present the first three lines in a comment face.

The following regex was an attempt to match every line after __gfx__:

"__gfx__\\(.\\|\n\\)*"

I am pretty sure there's a better way to write this regex, but even so, it actually only puts __gfx__ and the following few lines into a comment face, rather than the rest of the buffer like I had hoped.

Is using regexes the best way to match arbitrary syntaxes like this though? I tried looking at some other major-modes to see how they handled it (particularly things like code blocks in markdown-mode), and it seems they also use regex for these kinds of expressions.

My full major-mode code is here: https://gist.github.com/taylskid/a81ca9343711ada4ba85c7451803185a

I also attached an image showing the comment face not correctly applying here: pico-8 major mode screenshot

Sorry if this is trivial or can easily be found somewhere else. My searches didn't get very far. Thanks in advance.

2
  • Is it possible the fourth line after __gfx__ had an \r in it? Otherwise Emacs' behavior seems kind of random. Though, in general regular expressions in Emacs / POSIX don't work well with very long strings that don't have a definitive end marker. So, maybe there's a hardcoded restriction in Emacs implementation of regex preventing it from matching very long strings (this is very inefficient). In general, regular expressions can only approximate the grammar of the language you work with, so you need to take this into account up-front.
    – wvxvw
    Feb 20, 2018 at 16:13
  • I think your latter idea is more reasonable. It's highlighted only two lines sometimes as well. I know this isn't the greatest application of regex in the world, so I suppose a better rephrasing of my question is: "How do I create a syntax table to match those patterns idiomatically?" Looking at other modes, they mostly seem to use similar regex patterns, although you're right, not usually as openended as something like __gfx__ and down to the end of buffer. Feb 20, 2018 at 16:26

2 Answers 2

0

By default, font-lock only highlights a handful of lines at a time. In your case, it highlighted the line containing __gfx__ and the four following lines (which were correctly highlighted). It then proceeded to highlight the next couple of lines that, of course, don't match your regexp.

You can see this by using the package https://github.com/Lindydancer/highlight-refontification

The reason it appears to be working in font-lock-studio is that the entire buffer is highlighted at once.

This can be corrected using a number of techniques, including:

  • Extend the highlighting region by adding a function to font-lock-extend-region-functions. Note that if the gfx section is large, this might become really slow.

  • Match lines looking like hex data, regardless of context. This is not a good idea if such lines might appear in other contexts, but I doubt it. This can be done using a simple regexp.

  • Match lines looking like hex data, but also check if they are preceded by __gfx__. This can be done by using a matcher function that 1) use re-search-forward to look for hex lines and 2) search backward for the nearest section label.

When it comes to the __lua__ section, it's possible to highlight it by copying it to a temporary buffer, apply lua-mode, and copy the highlighting back. This is done in code blocks in Org mode. Fortunately, there is a support package that simplifies this, see https://github.com/cpitclaudel/indirect-font-lock for more information.

0

If you need to match all the way to the end of the buffer, you ought to end your regexp with \' (i.e. "\\'") which matches the empty string at the end of the text/buffer.

You could alternatively use the alternative C-hig (elisp)POSIX Regexps support, which is guaranteed to always return the longest match (although as your first regexp includes a non-greedy operator, you wouldn't use the POSIX functions with that one).

I'm not sure whether using \' enables any optimizations in the regexp engine, such that matching "the rest of the buffer" is efficient. I would be inclined to look for a way to search only for the __gfx__ marker and then specify the start of the match and (point-max) as the region of interest.

6
  • Do you have an example of using elisp like point-max in the syntax of a major-mode to use with highlighting? I haven't seen people use anything besides regexs for their syntax highlighting in a major mode, but it's entirely possible I've been looking in the wrong places Feb 20, 2018 at 22:00
  • Not an example, but see C-h v font-lock-keywords: "MATCHER can be either the regexp to search for, or the function name to call to make the search (called with one argument, the limit of the search; it should return non-nil, move point, and set match-data appropriately if it succeeds; like re-search-forward would)". Refer also to C-h i g (elisp)Search-based Fontification.
    – phils
    Feb 20, 2018 at 22:16
  • I found an example of almost what I was looking for here after reading the documentation on "fontification" here and wrote the function here: gist.github.com/taylskid/… which properly sets match-data after some manual testing. However it still only highlights the first couple lines after __gfx__ and as soon as something new gets typed into the buffer it un-highlights. Feb 21, 2018 at 16:15
  • As far as I can see it's working, but something else messes with it afterwards? I'm no expert in debugging font-locking, so I don't know offhand what's happening. I can point you at the font-lock-studio package, which confirmed that your code is matching what it is supposed to match, and you might like to look at the other Tools for font-lock developers written by Lindydancer as well.
    – phils
    Feb 21, 2018 at 18:39
  • yeah I am having the same experience. Everything highlights nicely in font-lock-studio but when I try to activate the mode normally it still leaves the __gfx__ bits left unhighlighted. I'm really stumped on this one, but maybe I should open a new question specifically about this? Feb 21, 2018 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.