how to properly move forward to the next sexp

Question

I was expecting a similar result with the two snippets below, but this is what I've got:

(with-temp-buffer
  (insert "(a)(b)(c)")
  (goto-char (point-min))
  (list
   (prog1 (cons (point) (cons (sexp-at-point) nil)) (forward-sexp))
   (prog1 (cons (point) (cons (sexp-at-point) nil)) (forward-sexp))
   (prog1 (cons (point) (cons (sexp-at-point) nil)) (forward-sexp))))

;; result: ((1 (a)) (4 (b)) (7 (c)))

and

(with-temp-buffer
  (insert "(a) (b) (c)")
  (goto-char (point-min))
  (list
   (prog1 (cons (point) (cons (sexp-at-point) nil)) (forward-sexp))
   (prog1 (cons (point) (cons (sexp-at-point) nil)) (forward-sexp))
   (prog1 (cons (point) (cons (sexp-at-point) nil)) (forward-sexp))))

;; result: ((1 (a)) (4 (a)) (8 (b)))

I'm using GNU Emacs 26.3.

How can I get something similar to ((1 (a)) (? (b)) (? (c))) on the second snippet? I think there is a proper way to parse things like this.

There is a caveat, if it matters, the buffer is and must remain read-only.

Answer 1

Here's what's happening, for the second case. You have this in the buffer, with point at position 1 (beginning of buffer): (a) (b) (c)

After the first forward-sexp point is just after the first right paren ()).

The next (sexp-at-point) returns the first sexp, not the second. Why? Because point is not on the second list ((b)), and it is just after the first list ((a)).

Vanilla Emacs sexp-at-point returns the sexp that is just before point, if there is not really any sexp at point.

(This is a bug, IMHO, but Emacs Dev does not agree with me. The correct behavior is to return nil, since there is no sexp at point.)

---

You can get around this by adding this after each (forward-sexp): (forward-whitespace 1). Or add this instead: (forward-char), if you are sure that a single space separates the sexps.

Another thing you can do, if you are sure there is a next sexp, is to go forward 2 sexps and then go back one sexp, to put you at the beginning of the sexp you want. That's in fact more typical. In other words, do this: (forward-sexp 2) (backward-sexp) instead of just (forward-sexp).

Still another thing you can do, which is more robust, is to use (next-visible-thing 'sexp) instead of (forward-sexp 2) (backward-sexp). This function ignores (skips over) comments if option ignore-comments-flag is non-nil. For this you need library thing-cmds.el (it in turn requires library hide-comnt.el). See Thing At Point Commands.

---

But keep in mind that this kind of thing is in general fragile. If you can be sure of the syntax you're checking, and if it is simple, then this can be good enough. But if the syntax is complex or flexible then this is not really the best approach - this is not really the way to parse.