calling search-forward function with ^ and $ in the regexp pattern

Question

I want to create a table in org-mode and use it in Ditaa The following function is created to replace “^+-” to “|-” and similarly, “^-+\s-*” to “-|”.

Surprisingly, this does not work.

(defun ditaa-to-table (begin end)
  "Replace “^\+-” to “|-” in region; Replace “-\+\s-*$” to “-|”"
  "I want to create a table in org-mode and use it in Ditaa"
  "org-mode uses |- and -| for the row/cell delimiter at the extremes of the     line, while Ditaa uses +- and -+"

  (interactive "r")
  (save-restriction
    (narrow-to-region begin end)

    (goto-char (point-min))
    (while (search-forward-regexp "^\s-*\+-" nil t) (replace-match "|-" nil t))

    (goto-char (point-min))
    (while (search-forward-regexp "-\+\s-*$" nil t) (replace-match "-|" nil t))

))

If I use,

(while (search-forward-regexp "\+-" nil t) (replace-match "|-" nil t)) ;; works

works, But once I add the "^\s-*" char to mark the start-of-line, it stops working.

Similarly, basic one works, but once the "\s-*$" is added, it stops working

(while (search-forward-regexp "-\+" nil t) (replace-match "-|" nil t)) ;; works
(while (search-forward-regexp "-\+\s-*$" nil t) (replace-match "-|" nil t)) ;; does not work

I tried posix-search-forward also, and it does not work.

Answer 1

You're getting tripped up by the escaping requirements for Emacs regexps when expressed in the double-quoted read syntax for strings, in conjunction with some historical-compatibility behaviour in Emacs' regexp engine.

Regarding the syntax requirements for special characters, please refer to these other Q&As:

• https://emacs.stackexchange.com/a/13391
• https://stackoverflow.com/a/10864091
• Why do regular expressions created with the regex builder use syntax different from the interactive regular expressions?

(The last one covers the issue in the context of the re-builder feature of Emacs, which is extremely useful.)

The reason your simpler initial regexp was working at all is explained in C-hig (elisp) Regexp Special as follows:

*Please note:* For historical compatibility, special characters are
treated as ordinary ones if they are in contexts where their special
meanings make no sense.  For example, ‘*foo’ treats ‘*’ as ordinary
since there is no preceding expression on which the ‘*’ can act.  It is
poor practice to depend on this behavior; quote the special character
anyway, regardless of where it appears.

"\+-" evaluates to +- which you are using as a regexp. + is normally special, but in that pattern is being treated the same as the regexp \+-, which in string read syntax would be written as "\\+-".

"^\s-*\+-" on the other hand evaluates to ^ -*+- which is certainly not the regexp you were intending.

See C-hig (elisp) Basic Char Syntax for why "\s" is a space.