How to get the position after matching and replacing a regular expression (replace-match)?

Question

SHORT
How to substitute (nicely) a regular expression by a string in pure elisp, like the unix command sed.

LONG
I'm using the function replace-match after string-match, but I can't get the right position/index (i.e the offset to start the next search) for string-match after the replacement. So I'm in an infinite loop.
I'd like to know how to get directly the position after the replacement.

Of course I could trick by setting the offset to start next search by adding the lengh of the new text, but I don't think it is the correct way.

Here is my code which does an infinite loop

(let ((pos1 0) ;; position of last match
      (str1 "bar") ;; string input
      (regexp1 "a") ;; the regular expression
      (newtext1 "aa")) ;; the new text
  (while (string-match regexp1 str1 pos1)
    (setq str1 (replace-match newtext1 t t str1 nil))
    (setq pos1 (match-end 0)))
  (message "%s" str1))

For example, if I add code to set the limit to 3 loops I will get the output "baaaaaar" while it should be "baar".

EDIT : replacing (setq pos1 (match-end 0))) with (setq pos1 (+ (match-end 0) (length "newtext1"))) does the job, but it is a trick I find inelegant.

Answer 1

The short answer is: Use (replace-regexp-in-string REGEXP REP STRING).

If you want to know how it is done, look at its source code.

Your modified version

EDIT: replacing (setq pos1 (match-end 0))) with (setq pos1 (+ (match-end 0) (length "newtext1"))) does the job, but it is a trick I find inelegant.

is wrong. It worked in your example by accident. You can use match-beginning in place of match-end to get a working version for simple replacement strings. Phils already gave you another fix in his comment.

But if you want to allow for more complicated replacement strings such as group references (as e.g., \1) you need to do what you really want.

You really want to further process the remaining tail of the string. So determine the tail length and calculate the next position on the basis this length.

You must also care about empty matches. Empty matches may only be treated a finite number of times otherwise you easily get infinite loops. It is most natural to consider empty matches exactly one time.

Essentially they also use these principles in replace-regexp-in-string. But, they decompose the original string by the matches and compose it back with the replacement strings in place of the matches. That is probably the most efficient method but it makes things a bit nastier.

I give here an alternative implementation demonstrating the principle.

(defun my-replace-regexp-in-string (regexp rep string)
  "Alternative version of `replace-regexp-in-string'.
REGEXP, REP, and STRING are the same args as for `replace-regexp-in-string'.
This is just for demonstration.  Therefore we do not care about
the optional arguments of `replace-regexp-in-string' here."
  (let ((pos 0)
    (old-tail-length (length string)))
    (catch :empty-match-at-end
      (while (string-match regexp string pos)
    (let ((tail-length (- (length string) (match-end 0))))
      (setq string (replace-match rep nil nil string))
      (when (eq tail-length old-tail-length) ;; empty match
        (if (zerop tail-length) ;; empty match at the end
        (throw :empty-match-at-end nil)
          (cl-decf tail-length))) ;; avoid infinite loop
      (setq pos (- (length string) tail-length)
        old-tail-length tail-length)
      ))))
  string)

;; tests:
(my-replace-regexp-in-string "a" "aa" "bar") ;; Returns "baar"
(my-replace-regexp-in-string "a" "aa" "baar") ;; Returns "baaaar"
(my-replace-regexp-in-string "a+" "\\&b" "baaaar") ;; Returns "baaaabr"
(my-replace-regexp-in-string "a*" "\\&c" "bar") ;; Returns "cbaccrc"

Answer 2

It seems like you're trying to emulate processing text in a buffer without using a buffer, and consequently you're complicating things for yourself considerably. Just use a buffer -- they are cheap and easy, and Emacs is packed full of functionality for manipulating them.

You can adapt your code like so:

(let ((str1 "bar") ;; string input
      (regexp1 "a") ;; the regular expression
      (newtext1 "aa")) ;; the new text
  (with-temp-buffer
    (insert str1)
    (goto-char (point-min))
    (while (re-search-forward regexp1 nil t)
      (replace-match newtext1))
    (buffer-string)))