How can a regex replace that computes the replacement in a function using back references be converted to a CLI batch version processing line by line?

Question

An example of this is a script to output the contents of .zsh_history file in the manner of the history command, so an emacs script would be invoked by a command line like display.sh.el < ~/.zsh_history_file and would process it by reading the commands line by line rather than loading the whole file into a buffer, as the files could be very large.

Entries in .zsh_history files are in the format

`: unixtime:duration;command`

: 1671297801:0;vim settings.php
: 1671298007:0;z private
: 1671298013:0;popd
: 1671298034:0;ll
: 1671298044:0;cd files/private/backup_migrate

and the history command displays the excerpt above in the manner shown. It takes out the leading : and the duration eg :0; and outputs the right-aligned line number of the entry, the formatted unixtime and the command separated by two spaces.

1491  2022-12-17 17:23  vim settings.php
1492  2022-12-17 17:26  z private
1493  2022-12-17 17:26  popd
1494  2022-12-17 17:27  ll
1495  2022-12-17 17:27  cd files/private/backup_migrate

The interactive version which is noted at https://emacs.stackexchange.com/a/79254 works thus:

The search regexp is: \(: \)\([0-9]\{10\}\)\(:0;\) which matches the : , the unixtime and the duration noted above.

The replacement string is: \,(concat (format "%6d " (line-number-at-pos)) (rgx-get-time-string (match-string 2)) " "))

The reason why back references for \1 and \3 are not used in the replacement expression is because they are discarded, but in principle the function should take (match-string 0) and (match-string 1) and replace them with the empty string.

I tried replace-regexp-in-string and failed, which prompted my earlier question, but it seems it may not be the most suitable function for this, and without examples the documentation has not proved helpful.

This question is about any other regex search and replace functions or combinations thereof which enable the use of the back references in the replacement function(s).

It may be that replace-regex-in-string is the most appropriate one and I will just have to master it.

PS. I have managed a not quite correct version using replace-regexp-in-string but will hold back from showing it to avoid muddying the waters further.

Answer 1

Answer moved from Am I using "replace-regexp-in-string" the right way?

---

It seems that my earlier answer has caused confusion, so here's a separate answer with what I think you're actually trying to achieve:

#!/bin/sh
":"; exec emacs -Q --script "$0" -- "$@" # -*-emacs-lisp-*-
(setq debug-on-error t)
(condition-case e
    (let ((counter 0)
          line)
      (while (setq line (read-from-minibuffer ""))
        (when (string-match "\\`: \\([0-9]\\{10\\}\\):0;" line)
          (setq line (replace-match
                      (concat (format "%6d " (setq counter (1+ counter)))
                              (format-time-string "%Y-%m-%d %H:%M  "
                                                  (string-to-number
                                                   (match-string 1 line))))
                      nil nil line)))
        (princ (concat line "\n"))))
  (end-of-file nil)) ;; read until EOF
(kill-emacs 0)

Refer to https://stackoverflow.com/a/10211087/324105 regarding the first two lines and the explicit call to kill-emacs. That is good boiler-plate for executable elisp scripts.

Answer 2

Answer moved from Am I using "replace-regexp-in-string" the right way?

---

The \, syntax for evaluating lisp within a replacement is for interactive replacements only. Search/replace code which is written in lisp to begin with has no need of that.

Typical elisp pseudocode for search/replace is:

(when (search PATTERN)
  (replace-match NEWTEXT &optional FIXEDCASE LITERAL STRING SUBEXP))

Where NEWTEXT can be established using any lisp code you want, and where search is whichever search function you're actually using -- it would be string-match in your case, but notice that the paradigm is (almost) exactly the same with that as it is when doing a search/replace against buffer contents with, say, re-search-forward.

See C-hig (elisp)Match Data for documentation on accessing the matched data.

If you only want to replace a portion of the matched string, you can make use of the SUBEXP argument.

And in your case, take note of:

If optional fourth argument STRING is non-nil, it should be a string
to act on; this should be the string on which the previous match was
done via `string-match'.  In this case, `replace-match' creates and
returns a new string, made by copying STRING and replacing the part of
STRING that was matched (the original STRING itself is not altered).

So be sure to use the return value.

---

In response to comments made in the original question asking about replacing multiple groups...

Replacing multiple sub-groups, one at a time, when working on a string:

(let ((foo "foo"))
  (when (string-match "\\(.\\)\\(.\\)\\(.\\)" foo)
    (setq foo (replace-match "three" t t foo 3))
    (setq foo (replace-match "two" t t foo 2))
    (setq foo (replace-match "one" t t foo 1)))
  foo)

=> "onetwothree"

Or chained together:

(let ((foo "foo"))
  (when (string-match "\\(.\\)\\(.\\)\\(.\\)" foo)
    (setq foo (replace-match
               "one" t t (replace-match
                          "two" t t (replace-match
                                     "three" t t foo 3)
                          2)
               1)))
  foo)

=> "onetwothree"

Either way, note (carefully) that we're working from right-to-left -- the right-most group 3 of the match is processed before group 2 which is processed before group 1. This is because the match data knows the positions which are valid for the original string, so we don't want to modify the length of any substring which would affect another group before we've dealt with that other group.

If we had gone left-to-right we would end up with "otthreeoeoo":

foo
├─┐
oneoo
 ├─┐
otwoeoo
  ├───┐
otthreeoeoo

(This is a concern only when working on strings. When working on buffer text, the match data can use markers to adapt to changes in the buffer when replacements are made.)

---

You could also insert your original string into a temporary buffer in order to simply work on it as buffer text instead of dealing with strings. (Note that we're safely working left-to-right here.)

(with-temp-buffer
  (save-excursion
    (insert "foo"))
  (when (re-search-forward "\\(.\\)\\(.\\)\\(.\\)" nil t)
    (replace-match "one" t t nil 1)
    (replace-match "two" t t nil 2)
    (replace-match "three" t t nil 3))
  (buffer-string))

=> "onetwothree"