4

I'm trying to implement a tokenizer for Haskell, it has a bit involved token structure that can be seen in code here:

https://github.com/ghc/ghc/blob/master/compiler/basicTypes/Lexeme.hs

In regexp-like terms it would be:

(LowercaseLetter|OtherLetter|_)(UppercaseLetter|LowercaseLetter|OtherLetter|TitlecaseLetter|DecimalNumber|OtherNumber|'|_)*ModifierLetter*#*

In human terms: a non-uppercase letter or underscore, followed by any numbers of letters, numbers, underscores, apostrophes followed by any number of modifier letters followed by any number of hashes.

How do I write a regexp in emacs lisp that uses Unicode General Categories as indicated above?

  • 1
    Seems like limited support for some Unicode general categories in regexps was added in Emacs 25 recently: debbugs.gnu.org/cgi/bugreport.cgi?bug=19878 Though it seems it's not possible to write an Emacs Lisp regexp matching all of the above categories. – legoscia Jul 7 '15 at 17:32
1

While a regexp based solution seems to be out of the question, you could write your own little matcher, using (get-char-code-property char 'general-category) and comparing it's result to the various Lx and Nx categories.

  • This is what I ended up doing. – Gracjan Polak Jul 15 '15 at 11:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.