How to substitute (nicely) a regular expression by a string in pure elisp, like the unix command sed.

I'm using the function replace-match after string-match, but I can't get the right position/index (i.e the offset to start the next search) for string-match after the replacement. So I'm in an infinite loop.
I'd like to know how to get directly the position after the replacement.

Of course I could trick by setting the offset to start next search by adding the lengh of the new text, but I don't think it is the correct way.

Here is my code which does an infinite loop

(let ((pos1 0) ;; position of last match
      (str1 "bar") ;; string input
      (regexp1 "a") ;; the regular expression
      (newtext1 "aa")) ;; the new text
  (while (string-match regexp1 str1 pos1)
    (setq str1 (replace-match newtext1 t t str1 nil))
    (setq pos1 (match-end 0)))
  (message "%s" str1))

For example, if I add code to set the limit to 3 loops I will get the output "baaaaaar" while it should be "baar".

EDIT : replacing (setq pos1 (match-end 0))) with (setq pos1 (+ (match-end 0) (length "newtext1"))) does the job, but it is a trick I find inelegant.

  • 3
    Just use (replace-regexp-in-string regexp1 newtext1 str1). If you want to know how it is done look at its source code. It is by far not what you would expect... – Tobias Apr 10 at 22:17
  • 3
    Surely it's the size of the difference between the old text and the new text that you want to increase the position by? I don't see why you think this is a "dirty" approach -- you don't have a point for a string, so you'd need to track it yourself. Or use a buffer, and then you do have a point. Buffers are cheap, so if you find this dirty, just don't use string manipulation. – phils Apr 10 at 22:19

The short answer is: Use (replace-regexp-in-string REGEXP REP STRING).

If you want to know how it is done, look at its source code.

Your modified version

EDIT: replacing (setq pos1 (match-end 0))) with (setq pos1 (+ (match-end 0) (length "newtext1"))) does the job, but it is a trick I find inelegant.

is wrong. It worked in your example by accident. You can use match-beginning in place of match-end to get a working version for simple replacement strings. Phils already gave you another fix in his comment.

But if you want to allow for more complicated replacement strings such as group references (as e.g., \1) you need to do what you really want.

You really want to further process the remaining tail of the string. So determine the tail length and calculate the next position on the basis this length.

You must also care about empty matches. Empty matches may only be treated a finite number of times otherwise you easily get infinite loops. It is most natural to consider empty matches exactly one time.

Essentially they also use these principles in replace-regexp-in-string. But, they decompose the original string by the matches and compose it back with the replacement strings in place of the matches. That is probably the most efficient method but it makes things a bit nastier.

I give here an alternative implementation demonstrating the principle.

(defun my-replace-regexp-in-string (regexp rep string)
  "Alternative version of `replace-regexp-in-string'.
REGEXP, REP, and STRING are the same args as for `replace-regexp-in-string'.
This is just for demonstration.  Therefore we do not care about
the optional arguments of `replace-regexp-in-string' here."
  (let ((pos 0)
    (old-tail-length (length string)))
    (catch :empty-match-at-end
      (while (string-match regexp string pos)
    (let ((tail-length (- (length string) (match-end 0))))
      (setq string (replace-match rep nil nil string))
      (when (eq tail-length old-tail-length) ;; empty match
        (if (zerop tail-length) ;; empty match at the end
        (throw :empty-match-at-end nil)
          (cl-decf tail-length))) ;; avoid infinite loop
      (setq pos (- (length string) tail-length)
        old-tail-length tail-length)

;; tests:
(my-replace-regexp-in-string "a" "aa" "bar") ;; Returns "baar"
(my-replace-regexp-in-string "a" "aa" "baar") ;; Returns "baaaar"
(my-replace-regexp-in-string "a+" "\\&b" "baaaar") ;; Returns "baaaabr"
(my-replace-regexp-in-string "a*" "\\&c" "bar") ;; Returns "cbaccrc"
| improve this answer | |
  • Thank you very much Tobias for your complete answer. I completely forgot the empty match case. And thanks for the other correction (i.e match-beginning in place of match-end). I've looked at replace-regexp-in-string source code too, but for the moment it is a bit beyond my elisp knowledge. I didn't really understand everything yet. But I understood everything in your example and I learnt some new things because of you. I deeply thank you for your advices and the time you took for the newbie I'm :-) – Albert Colvert Apr 19 at 21:04

It seems like you're trying to emulate processing text in a buffer without using a buffer, and consequently you're complicating things for yourself considerably. Just use a buffer -- they are cheap and easy, and Emacs is packed full of functionality for manipulating them.

You can adapt your code like so:

(let ((str1 "bar") ;; string input
      (regexp1 "a") ;; the regular expression
      (newtext1 "aa")) ;; the new text
    (insert str1)
    (goto-char (point-min))
    (while (re-search-forward regexp1 nil t)
      (replace-match newtext1))
| improve this answer | |
  • 1
    I wonder why they implemented replace-regexp-in-string differently. It would be interesting to profile and compare the approaches. – Tobias Apr 11 at 13:24
  • 1
    I imagine replace-regexp-in-string is more efficient -- for starters it's working on an existing string object, rather making a copy via insert. That's an important concern for a core API function like that one, but I suspect it's a non-issue for the purposes of this question. – phils Apr 12 at 1:35
  • I've just validated the reply of Tobias, but your answer it excellent too. I thank you very much too. I didn't know buffers were so cheap. So, from now on, I won't hesitate to use them. Many thanks for your advice. :-) – Albert Colvert Apr 19 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.