I want to transform a text file with many lines :

| refXXXXX | here useful text, first sentence, last word. |
| refYYYYY | First word, second sentence. |

into :

here useful text, first sentence, last word. First word, second sentence.

but appearing as :

[refXXXXX] here useful text, first sentence, last word. [refYYYYY] First word, second sentence.

Where [refs within brackets] would be Overlays or Text Properties.

(The aim would be to be able to make use of C-s or M-x occur to retrieve directly "last word. First word", at the end of first sentence + beginning of second sentence in the example. Btw, I know how to do the corresponding search in the source file with a regexp, but it's not the solution I want.)

My question is :

is this possible with Overlays or Text properties, with a defun retrieving all the "refs", cutting them from the text and replacing them by Overlays/TextProperties?

Something like :

(defun source-to-final ()
    "cut refs from the txt, but letting them appear as overlays (or text properties)"
        (search-forward-regexp "| ref.*?|")
      (replace-match "")   ;; cutting the ref text
      (set-some-overlay-or-textproperty-here (match-string 0)) 
      (search-forward-regexp " +|\n") ;; assembling current sentence with next one
      (replace-match "")

where I just don't know how the defun set-some-overlay-or-textproperty-here should look like.

2 Answers 2


Yes, it’s possible

A couple of points on your function:

  1. Don’t cut the text before you need it.
  2. Use the third argument for search-forward-regexp to avoid getting errors.

And so:

(defun source-to-final ()
  "Cut refs from the txt, but letting them appear as text properties."
    (goto-char (point-min))
    (while (search-forward-regexp "| \\(ref.*?\\) |" nil 'noerror)
      (let ((ref (match-string 1)))
        (replace-match "") ;; cutting the ref text
        ;; This kind of assumes the character after point is a blank space.
        (set-some-overlay-or-textproperty-here (concat ref " ")) 
        (when (search-forward-regexp " +|\n" nil 'noerror)
          (replace-match ""))))))

And here’s how you set text properties.

(defun set-some-overlay-or-textproperty-here (text)
  (add-text-properties (point) (1+ (point))
                       `(display ,(propertize text 'face 'font-lock-comment-face))))


If you run the command above, the buffer should be the way you want. Then you can isearch for last word. First word and you'll be taken to the right place.

  • Well, almost OK. But after use source-to-final, I cannot retrieve last word. First word as is but only separated by 2 spaces : last word. First word. I have tried to correct this, but no success.
    – JeanMichel
    Oct 9, 2014 at 14:52
  • @Jean-Michel Sorry, there was a bug in there. Try again.
    – Malabarba
    Oct 9, 2014 at 15:07
  • Yes, in the mean time I got it too. Tks !
    – JeanMichel
    Oct 9, 2014 at 15:12

Library isearch-prop.el can help with at least some of what you want to do, I think.

See command isearchp-add-regexp-as-property or function isearchp-regexp-scan, for adding text properties based on regexps, to give you propertied zones amenable for searching.

Other commands and functions in the library let you search such propertied zones or their complements. For example, you can use C-t within Isearch (or C-M-t within regexp Isearch) to search within such a zone. (C-M-~ toggles searching the complement.)

This adds property foo (with value ("ref.+")) to text matching the regexp ref.+:

(isearchp-add-regexp-as-property (point-min) (point-max) 'foo)

(You can also apply other properties, such as face or mouse-face or invisible, to the same text, depending on whether you want some particular display behavior.)

But note that if your aim is just to propertize certain positions or zones of text so that you can quickly get to them later then, then unless you need to insert your refXXXX text for some other reason, you can skip that text insertion and just propertize the locations/zones directly. You can easily do that using isearch-prop.el.

IOW, why are you bothering to insert refXXXX text that you later want to hide (or highlight?), if that text is just meant to identify buffer positions or zones?

If you just want to mark positions, you can of course use markers, but they do not have identities (i.e., additional info) beyond their positions. If you want to mark the text itself, you can use properties to do that.

Here are the doc strings for the functions that add properties that let you search regexp-defined zones (or their complements):

(isearchp-regexp-scan BEG END PROPERTY REGEXP &optional PREDICATE ACTION)

Scan for REGEXP, adding text property PROPERTY to matches.
If the region is active then scan it.  Otherwise scan the buffer.
Return the value of PROPERTY, if added somewhere.  Else return nil.
See `isearchp-add-regexp-as-property' for the parameter descriptions.

(isearchp-add-regexp-as-property PROPERTY REGEXP &optional BEG END PREDICATE ACTION MSGP)

Add PROPERTY with value (REGEXP . PREDICATE) to REGEXP matches.
If region is active, limit action to region.  Else, use whole buffer.
If REGEXP has subgroups, then use what the Nth subgroup matches as the
 search context (hit), where N = `isearchp-context-level'.
 If N=0, then use the overall match of REGEXP as the search context.
PROPERTY is a text property to add to the search-hit text.
 the value of the property is REGEXP (a string).
 Interactively, PREDICATE is nil, so the PROPERTY value is (REGEXP).


BEG, END are the region limits.  If nil, the buffer limits are used.
PREDICATE is nil or a boolean function that takes these arguments:
  - the search-context string
  - a marker at the end of the search-context
If PREDICATE is non-nil then act on only the hits for which it holds.
If ACTION is non-nil then it is a function that accepts no arguments.
 It is called after matching buffer text with REGEXP.  After ACTION,
 the search hit end position is extended or restricted to point.
Non-interactively, non-nil BEG and END are used as the region limits.

Library isearch-prop.el is a companion to library isearch+.el (neither is needed for the other, but they enhance each other). See also Isearch+.

  • I have somme difficulties to find my way in the ovewhelming load of functions offered by the library. How would I use it to : 1. kill the "ref" from the text and put it in the "property" 2. retrieve the ref in the property ? What I want to achieve is having 2 ways to search the text : 1 way with the normal C-s (or M-x occur) for text only, without the refs beeing in my way and 1 way to search just by looking for the refs, which are in the properties
    – JeanMichel
    Oct 30, 2014 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.