How does Emacs compile the regular expressions?

Question

I'm writing code that will have to do a lot of regular expression matching. I understand that a function like string-match or re-search-forward needs to process its regexp argument given as a string and turn it into a data structure that will be used for the actual matching/searching. This processing is costly and I would like to know if it is done every time I call re-search-forward or if it is done only once for each new regexp and is not redone when one uses the same regexp several times.

Let me consider a concrete situation. Assume that I want to loop over all lower-cap letters in a large text buffer. I can loop with a (while (re-search-forward rx nil t) ...) using string "[a-z]" as a regexp. Or I can write a loop over buffer positions and test the (char-after pos) each time. If the regexp is recompiled with each new call of re-search-forward the first option will be much slower than the second one. If the regexp is compiled just once then the first option will be more or less as fast as the second option but it leads to much cleaner code, easier to read and modify, so should be preferred.

Answer 1

Emacs' regexp implementation caches previously used patterns in compiled form (20 of them, last time I looked). This is not a detail you should rely on too much.

(while (re-search-forward rx nil t) ...) is the idiomatic way to search through a buffer for a regexp, so unless you measure it and find it to be slower than an alternative, that's what you should use.