Why do regexp that matches text in buffer does not (necessarily) match same text in string?

Question

I have a regexp, org-parse-regexp which matches org items which I define to be a heading and a source block (actually, I include more stuff but I want to keep it simple). I've tested it using re-search-forward in the buffer config.org.

(with-current-buffer "config.org"
    (goto-char (point-min))
    (re-search-forward org-parse-regexp)
    (match-string-no-properties 0))

And it works! Here's a first level heading with a source block.

"* Variables

#+begin_src emacs-lisp
(message \"some code\")
#+end_src"

I thought it should also work, if I try using the regexp to match org items for the buffer contents as a string instead of traversing the buffer itself.

(let ((my-org-file-contents (with-current-buffer "config.org" (buffer-string)))
  (string-match org-parse-regexp my-org-file-contents)
  (match-string 0 my-org-file-contents))

Surprisingly, using the same regexp I got a different result. This is the same heading as before, but the source block wasn't captured!

"* Variables

"

Hmm... I admit I am a little stumped as to why this is the case. But I decided to try just capturing a source block and see what happens.

I use a small excerpt from my source block capturing expression.

;;                                       language
(rx "#+begin_src" (1+ blank) (group-n 7 letter (1+ (or letter "-")))
   ;; switches
   (opt (1+ blank) (group-n 8 (1+ ":" (1+ (or letter blank)))))
   ;; one or more whitespace characters
   (1+ white)
   ;; body
   (group-n 9 (regexp "\\([^\000]*?\n\\)??[ \t]*")) "#+end_src")) 

And swap it in for org-parse-regexp for both the buffer and the buffer substring example.

As I expected I get the first code block of my org file when I traverse the buffer itself with re-search-forward.

"#+begin_src emacs-lisp
(message \"some code\")
#+end_src"

I didn't expect to get anything for the substring example, but I was amazed when it gave me a source block! Albeit, not the first one (it was in fact the 92nd code block in my org file). It's also the first one with a switch.

"#+begin_src emacs-lisp :tangle yes
(message "Hi, I'm number 92!")
#+end_src"

After deleting the :tangle yes switch, I tried again and I got the next source block with a switch. Clearly, this has to do with the switch or the whitespace around it.

I realized that one of the problems was the (1+ white) in my regexp which is the same as [[:space:]]+. When I replaced this with "\n" both methods caught the first source block in the file.

I got the source block part working but I still need to make changes like this to the rest of my regexp, but I don't understand.

Why does this change made a difference? And in general why does a regexp that matches targets using re-search-forward in a buffer not necessary match the same things when used with string-match?

UPDATE: The way I reproduced the problem:

Use (1+ white) instead of "\n" in the regular expression.

(let (my-org-string (with-current-buffer "config.org"
                              (buffer-substring-no-properties (point-min)
                                                              (point-max))))
  (string-match org-parse-regexp my-org-string)
  (match-string-no-properties 0 my-org-string))

;; ouput
"*** Variables

"

(with-currernt-buffer "config.org"
  (goto-char (point-min))
  (re-search-forward org-parse-regexp)
  (match-string-no-properties 0))

;; output
"*** Variables

#+begin_src emacs-lisp
(message "hi")
#+end_src"

The org file looks like:

*** Variables

#+begin_src emacs-lisp
(message "hi")
#+end_src

For reference, the full regexp from my code is:

  (rx (sequence
       ;; needs to be at beginning of line - so asteriks
       ;; that are in paragraphs are matched.
       bol
       ;; heading level
       (group-n 1 (1+ "*"))

       ;; todo name
       (opt blank (group-n 2 upper upper-case (+ (or upper "-")) upper))

       ;; header name
       blank (group-n 3 (+ (any letter punct digit)) (opt (+ blank (+ (any letter punct digit)))))

       (* blank)

       ;; tags
       (opt (group-n 4 (1+ ":" (1+ letter)) ":"))

       "\n"

       (opt "\n")

       ;; description
       ;; for now not "*" or "#" as it interferes with org headings and org blocks respectively
       (opt (+ (group-n 5
                        (+ (or digit letter "[" "=" "~") (+ nonl) "\n")
                        (opt (+ "\n" (+ (or digit letter "[" "=" "~") (+ nonl) "\n"))))))

       ;; source block
       (opt
        (opt "\n")
        (opt "#+name:" (1+ blank) (group-n 6 (1+ (or alnum "-"))))
        ;; language
        (opt "\n")
        "#+begin_src" (1+ blank) (group-n 7 letter (1+ (or letter "-")))
        ;; optional source arguments
        (opt (1+ blank) (group-n 8 (1+ ":" (1+ letter) blank (1+ (or letter punct)))))
        ;; go to new line
        ;; was -> (1+ white)
        "\n"
        ;; arguments
        ;; body
        (group-n 9 (regexp "\\([^\000]*?\n\\)??[ \t]*")) "#+end_src")))

Answer 1

I believe your question is this:

Why would [[:space:]] match a newline here:

(with-current-buffer "config.org"
  (goto-char (point-min))
  (re-search-forward org-parse-regexp)
  (match-string-no-properties 0))

But not here?

(let ((my-org-file-contents (with-current-buffer "config.org" (buffer-string))))
  (string-match org-parse-regexp my-org-file-contents)
  (match-string 0 my-org-file-contents))

---

The reason why it is possible to get two different results is that in the first block of code the regular expression search happens while config.org is the current buffer, while in the second block of code the regular expression search happens in the context of whatever the current buffer was when that code was evaluated.

C-hig (elisp)Char Classes tells us:

‘[:space:]’
     This matches any character that has whitespace syntax (*note Syntax
     Class Table::).

That is to say, the result is dependent on the syntax table -- and syntax tables are buffer-local!

So, as unintuitive as it may seem, it's possible for a newline to have whitespace syntax in some buffers, but not in others. I suspect an emacs lisp buffer was current when you were testing this code?

C-hig (elisp)Syntax Class Table tells us:

Comment starters: ‘<’
Comment enders: ‘>’
     Characters used in various languages to delimit comments.  Human
     text has no comment characters.  In Lisp, the semicolon (‘;’)
     starts a comment and a newline or formfeed ends one.

We can verify this with the char-syntax function:

(char-syntax ?\n)

In an org-mode buffer: 32 (#o40, #x20, ? ) -- whitespace syntax

In an elisp buffer: 62 (#o76, #x3e, ?>) -- endcomment syntax

n.b. C-uC-x= with point at a newline will also give you this (and other) information interactively.

---

The key take-away from all this is that evaluating code within the context of the correct buffer can be very important.