What are overlays for, and how do they differ from text properties?

Question

The manual states:

You can use overlays to alter the appearance of a buffer's text on the screen, for the sake of presentation features.

From its name, it suggests it can be used to create “layers” over existing text, but the description above seems to indicate it alters the appearance of text, which feels very similar to what text properties do.

What is a concrete use of overlays? What do they offer that text properties don't? Can they be used to write text over text in a buffer, like a popup, info box, etc.?

Answer 1

What are Overlays?

First of all, they are elisp objects. This will be relevant later.

As you said yourself, they represent layers that apply on top of regions of the buffer. These layers have text properties, just like the actual text in the buffer. Any regular property that an overlay has, applies to the text below it. However, there are some properties that are special to overlays (they don’t do anything if applied on text).

Why are they Useful?

For the two reasons I hinted at above:

They are objects

This means you can store them in lists and handle them easily. You can change their properties without having to keep track of where they are right now. Even better, you can use them to keep track of where things are in the buffer.

This answer by Stefan is a good example of overlays being used to keep track of buffer regions. Here’s a short snippet of it.

(let ((ol (make-overlay beg end)))
  (overlay-put ol 'evaporate t)
  (overlay-put ol 'my--auto-align-regexp regexp)
  (push ol my--auto-align-overlays))

It uses overlays to record which regions are to be aligned, and which regexp to use on each of them. my--auto-align-overlays is

a list where the overlays are stored, and they can be conveniently accessed by looking through this list.
In contrast, if we were using text properties for this (which is possible) we would have no easy way of accessing them. You need to parse the buffer to find text properties.

They are slightly more powerful

There are a few text-properties which only have an effect on overlays. You can find the entire list on the manual. It’s too large to include here, but here’s a short highlight.

before-string
This property's value is a string to add to the display at the beginning of the overlay. The string does not appear in the buffer in any sense—only on the screen.
line-prefix
This property specifies a display spec to prepend to each non-continuation line at display-time. See Truncation.
wrap-prefix
This property specifies a display spec to prepend to each continuation line at display-time. See Truncation.

In particular, the before-string property allows you to affect the display of the buffer even with a 0 width overlay. Which is something you can’t do with text property. Everything you do with text properties needs to either go over an existing text (which might hide this text) or go over a new string of text you insert (which alters the real contents of the buffer).

Here’s an example snippet about that. Evaluate it on a temporary buffer.

(overlay-put (make-overlay (point) (point)) 'before-string "Hi there!")

When are they Bad?

Overlays are much more demanding (processing wise) than text properties. Some relevant operations (such as text insertion, if I’m not mistaken) take time proportional to the number of overlays in a buffer. For that reason, they are unsuitable when you need a large number of them. In this case you should resign to text properties.

Answer 2

Overlays and text properties have exactly the same capabilities. They add change margins, buttons, faces, fringe icons, tooltips, images, highlights, etc. to a specific part of of text. However, there is a huge conceptual difference:

Overlays are independent, text properties are not. Emacs has a distinct overlay type, but there is no “text property type”. Text properties are inherently tied to the string they were set on. Overlays on the other hand are attached to a buffer.

There are a few practical implications of this difference:

• You can set text properties on strings that are not part of any buffer, whereas you always need a buffer for overlays.
• You can't keep a list of text properties you added somewhere, whereas it's easy to keep a list of overlays added to a buffer, and for instance, delete all your overlays in one go.
• You can have a single overlay that sets multiple overlays on a text (e.g. a face, a tooltip, and a fringe icon), so you can use an overlay to manage a “group” of properties as a single unit, whereas there is no such thing for text properties. You set them individually, and you manage them individually.

The essence of all this is that you can manage overlays separately from the text content. Conceptually you'd use them for stuff that you'd show in the buffer, even though it's independent from the text.

Practically, you'd use overlays if the stuff that you'd like to add to text changes frequently, whereas you'd use text properties if the stuff is constant.

For instance, Emacs uses text properties to create the buttons and input widgets in a M-x customize buffer. These widgets are created once, and stay there as long as the buffer is alive, so there's no need to bring in overlays.

However, Flycheck uses overlays to highlight errors in the buffer, because it frequently needs to remove and add highlights.