1

I have a regexp, org-parse-regexp which matches org items which I define to be a heading and a source block (actually, I include more stuff but I want to keep it simple). I've tested it using re-search-forward in the buffer config.org.

(with-current-buffer "config.org"
    (goto-char (point-min))
    (re-search-forward org-parse-regexp)
    (match-string-no-properties 0))

And it works! Here's a first level heading with a source block.

"* Variables

#+begin_src emacs-lisp
(message \"some code\")
#+end_src"

I thought it should also work, if I try using the regexp to match org items for the buffer contents as a string instead of traversing the buffer itself.

(let ((my-org-file-contents (with-current-buffer "config.org" (buffer-string)))
  (string-match org-parse-regexp my-org-file-contents)
  (match-string 0 my-org-file-contents))

Surprisingly, using the same regexp I got a different result. This is the same heading as before, but the source block wasn't captured!

"* Variables

"

Hmm... I admit I am a little stumped as to why this is the case. But I decided to try just capturing a source block and see what happens.

I use a small excerpt from my source block capturing expression.

;;                                       language
(rx "#+begin_src" (1+ blank) (group-n 7 letter (1+ (or letter "-")))
   ;; switches
   (opt (1+ blank) (group-n 8 (1+ ":" (1+ (or letter blank)))))
   ;; one or more whitespace characters
   (1+ white)
   ;; body
   (group-n 9 (regexp "\\([^\000]*?\n\\)??[ \t]*")) "#+end_src")) 

And swap it in for org-parse-regexp for both the buffer and the buffer substring example.

As I expected I get the first code block of my org file when I traverse the buffer itself with re-search-forward.

"#+begin_src emacs-lisp
(message \"some code\")
#+end_src"

I didn't expect to get anything for the substring example, but I was amazed when it gave me a source block! Albeit, not the first one (it was in fact the 92nd code block in my org file). It's also the first one with a switch.

"#+begin_src emacs-lisp :tangle yes
(message "Hi, I'm number 92!")
#+end_src"

After deleting the :tangle yes switch, I tried again and I got the next source block with a switch. Clearly, this has to do with the switch or the whitespace around it.

I realized that one of the problems was the (1+ white) in my regexp which is the same as [[:space:]]+. When I replaced this with "\n" both methods caught the first source block in the file.

I got the source block part working but I still need to make changes like this to the rest of my regexp, but I don't understand.

Why does this change made a difference? And in general why does a regexp that matches targets using re-search-forward in a buffer not necessary match the same things when used with string-match?

UPDATE: The way I reproduced the problem:

Use (1+ white) instead of "\n" in the regular expression.

(let (my-org-string (with-current-buffer "config.org"
                              (buffer-substring-no-properties (point-min)
                                                              (point-max))))
  (string-match org-parse-regexp my-org-string)
  (match-string-no-properties 0 my-org-string))

;; ouput
"*** Variables

"

(with-currernt-buffer "config.org"
  (goto-char (point-min))
  (re-search-forward org-parse-regexp)
  (match-string-no-properties 0))

;; output
"*** Variables

#+begin_src emacs-lisp
(message "hi")
#+end_src"

The org file looks like:

*** Variables

#+begin_src emacs-lisp
(message "hi")
#+end_src

For reference, the full regexp from my code is:

  (rx (sequence
       ;; needs to be at beginning of line - so asteriks
       ;; that are in paragraphs are matched.
       bol
       ;; heading level
       (group-n 1 (1+ "*"))

       ;; todo name
       (opt blank (group-n 2 upper upper-case (+ (or upper "-")) upper))

       ;; header name
       blank (group-n 3 (+ (any letter punct digit)) (opt (+ blank (+ (any letter punct digit)))))

       (* blank)

       ;; tags
       (opt (group-n 4 (1+ ":" (1+ letter)) ":"))

       "\n"

       (opt "\n")

       ;; description
       ;; for now not "*" or "#" as it interferes with org headings and org blocks respectively
       (opt (+ (group-n 5
                        (+ (or digit letter "[" "=" "~") (+ nonl) "\n")
                        (opt (+ "\n" (+ (or digit letter "[" "=" "~") (+ nonl) "\n"))))))

       ;; source block
       (opt
        (opt "\n")
        (opt "#+name:" (1+ blank) (group-n 6 (1+ (or alnum "-"))))
        ;; language
        (opt "\n")
        "#+begin_src" (1+ blank) (group-n 7 letter (1+ (or letter "-")))
        ;; optional source arguments
        (opt (1+ blank) (group-n 8 (1+ ":" (1+ letter) blank (1+ (or letter punct)))))
        ;; go to new line
        ;; was -> (1+ white)
        "\n"
        ;; arguments
        ;; body
        (group-n 9 (regexp "\\([^\000]*?\n\\)??[ \t]*")) "#+end_src")))
7
  • A more minimal example would be useful. Your rx form produces an invalid regexp on account of (regexp "\\[^000]*?\n\\)??[ \t]*") introducing that closing "\\)" without any matching opening paren (and there look to be other syntax issues with that bit). If it's awkward to pare down your original regexp, just post the full version.
    – phils
    Apr 8, 2018 at 23:36
  • Oh I'm sorry that was a typo I will fix it. I see your point. I made findings that will can hopefully be useful to checkout despite the not-so-minimal example. Apr 8, 2018 at 23:39
  • Sure I can post the full version, it's a bit long though. Apr 8, 2018 at 23:39
  • Using the full regexp, and substituting in the "before" part in place of the "after" part, I cannot replicate your problem. Can you provide an example .org file which demonstrates the issue?
    – phils
    Apr 9, 2018 at 1:05
  • 1
    The presence of with-currernt-buffer (along with the earlier "that was a typo") has me worried that you're adding code to your question by manually typing it, rather than copying and pasting from the original source in Emacs? Please ensure that you are always pasting the exact code you were using -- the potential for typos resulting in wasted time on all sides is rather high.
    – phils
    Apr 9, 2018 at 5:22

1 Answer 1

1

I believe your question is this:

Why would [[:space:]] match a newline here:

(with-current-buffer "config.org"
  (goto-char (point-min))
  (re-search-forward org-parse-regexp)
  (match-string-no-properties 0))

But not here?

(let ((my-org-file-contents (with-current-buffer "config.org" (buffer-string))))
  (string-match org-parse-regexp my-org-file-contents)
  (match-string 0 my-org-file-contents))

The reason why it is possible to get two different results is that in the first block of code the regular expression search happens while config.org is the current buffer, while in the second block of code the regular expression search happens in the context of whatever the current buffer was when that code was evaluated.

C-hig (elisp)Char Classes tells us:

‘[:space:]’
     This matches any character that has whitespace syntax (*note Syntax
     Class Table::).

That is to say, the result is dependent on the syntax table -- and syntax tables are buffer-local!

So, as unintuitive as it may seem, it's possible for a newline to have whitespace syntax in some buffers, but not in others. I suspect an emacs lisp buffer was current when you were testing this code?

C-hig (elisp)Syntax Class Table tells us:

Comment starters: ‘<’
Comment enders: ‘>’
     Characters used in various languages to delimit comments.  Human
     text has no comment characters.  In Lisp, the semicolon (‘;’)
     starts a comment and a newline or formfeed ends one.

We can verify this with the char-syntax function:

(char-syntax ?\n)

In an org-mode buffer: 32 (#o40, #x20, ? ) -- whitespace syntax

In an elisp buffer: 62 (#o76, #x3e, ?>) -- endcomment syntax

n.b. C-uC-x= with point at a newline will also give you this (and other) information interactively.


The key take-away from all this is that evaluating code within the context of the correct buffer can be very important.

3
  • 1
    My take-away is that [[:space:]] is best avoided and you're better off being explicit about what space characters you accept.
    – wasamasa
    Apr 9, 2018 at 5:54
  • @wasamasa: Unless your purpose is to match whitespace, however it might be defined in the current context. That's the point of [[:space:]]: it matches a whitespace character, which is abstract and depends on the context.
    – Drew
    Apr 9, 2018 at 14:21
  • It's also the case that there are many other syntax-based constructs for regular expressions, which have the potential to trip you up in similar fashion. Not using such constructs unnecessarily may be sane, but I do think the most important thing is to be aware that buffer-local values can affect behaviour.
    – phils
    Apr 9, 2018 at 23:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.