Ways to unobtrusively vary text rendering?

Question

I'm writing an emacs extension for use with speech recognition, and I'm looking for help with a particular feature. Some words the speech recognizer (Dragon) recognizes consistently poorly -- it doesn't matter how many times you train it, it will just suck at recognizing certain words. At the same time usually when you're writing on a topic or when coding you'll be using a lot of the same words over and over again.

So I've written a mode that uses overlays to change how words are rendered in the buffer. It takes a random letter in the word, underlines it in a random color, and puts a random diacritical mark (accent, umlaut, etc.) over the top of it. Here's a screenshot (you will probably need to zoom to see marks/underlines):

Then you can say, "purple p hair" and it will look for the word with a purple underline under its 'a' with a diacritical mark that looks like hair and type that word for you. So in the above screenshot saying that would cause emacs to type "regexp-quote" for you.

The idea is this lets you refer to any word you've already used that's onscreen using a finite set of words the recognizer is consistently good at recognizing.

It works pretty well, except occasionally there is a collision. To make it so I can learn to consistently refer to words the same way I'm using bytes from the md5 hash of the word instead of (random) or having an algorithm assign the changes such that collisions are avoided. I've only found 6 easily distinguishable colors (it's hard when the underline is only one character wide and a single pixel thick) and 3 easily distinguishable diacritical marks (easy to tell apart from one another and also not confusable with an underline on the above line or overlapping with the underline), seen at the top of the source above.

I need more ways to alter rendering in order to reduce the collision frequency. Ideally a rendering modification would:

• Not be jarring from the rest of the text. This has lead me to dismiss for example the inverse-video property.
• Not be easily confusable with other changes. Overlines are easily mistaken for underlines on the previous line. Lots of diacritical marks look similar unless your font size is impractically huge.
• Be spatially near where the other changes are. Right now once my eye finds the targeting character all the information is there, the marker, the underline, and the letter.
• Work nicely with a fixed width font (needed for coding) that correctly renders diacritical marks (I had to switch to DejaVu Sans Mono from Consolas in order to have the marks render correctly)
• Work on latin alphabet letters. There are arabic combining marks for example, but they don't combine on latin alphabet characters.
• Not change letter color, since that's already being used for syntax highlighting.
• Actually be doable in emacs with emacs lisp ;)

Maybe there are special unicode characters controlling rendering that could be abused to open up new possibilities? Or a way to thicken underlines so I might be able to easily distinguish more colors? Or some other obscure emacs feature letting you render marks on top of characters besides unicode?

Answer 1

Another possibility would be to display line numbers and say the line number before the word, or, since looking over to get the exact line number would be bothersome, you could have the algorithm search within + or - 5 or 10 lines of the number you say.

Or perhaps declare a region or function that you are working in and have all searches only look there. I would guess that would limit collisions.

You could also render unicode symbols after or before a word in a given color to help them stand out. And also box or underline the word in another color. This way you could have 6 word colors * 6 symbol colors * N symbol possibilities. You could probably find 10 good symbols and have 360 combinations. For example you could say "blue yellow star" to refer to the word cat here.

If the star is too jarring, you could couple :box, and two different :underlines.

So you could refer to the word tree here using "blue yellow red" which would give you 216 combinations to use.

Answer 2

Have you heard of ace-jump-mode?

It doesn't meet any of the requirements you specify, but it looks like it fits perfectly what you're trying to achieve. It would allow the user to specify any word by saying only 2 or 3 words.

You can define the set of characters it offers you, so you can avoid consonants that are hard to distinguish. Then the use could just say "fix A nine", and correct the 9th word that starts with a.

Answer 3

Interesting question. I'm betting you'll get some interesting suggestions.

One minor suggestion that occurs to me is to use different colors and styles for underlining. See the Elisp manual, node Face Attributes about attribute :underline and its :color and :style components.

You can also experiment with attribute :box and different line widths and styles for it, but that's perhaps too jarring.

Answer 4

I'll answer by proposing an alternative way to select the target word. Highlight half of the words (randomly chosen). The user says "yes" if the target word is highlighted and "no" otherwise. If the user said "yes", take all words that were highlighted and randomly highlight half of them. If the user said "no", randomly highlight the half of the words that were not highlighted. Again the user indicates whether the target word is highlighted by saying "yes" or "no". Repeat that until only the target word is highlighted.

Some benefits of this approach:

• This works, no matter how many words you have on the screen.
• You don't need fancy colors, fonts, or symbols. A monochrome display is enough.
• Very low cognitive load because it is easy to tell whether a word is highlighted or not.

Drawback: You have to say "yes" and "no" too often. However, this is fixed by the following variation of the idea: do not highlight the words but use colors for them. You say you have 6 easily distinguishable colors. This means that if you have 100 words on the screen, selecting the target word requires naming 2.6 colors on average. If there are 1000 words, you have to name 3.9 colors on average.

Answer 5

The following is an example using an overlay with an xpm image for graphical Emacs versions that support xpm image format. It is 11 pixels wide; 20 pixels high; and has a user specified number of 4 possible colors. I am on a Mac running Snow Leopard 10.6.8 and the font I prefer when using Emacs is -*-Courier-normal-normal-normal-*-18-*-*-*-m-0-iso10646-1 -- the frame-char-width is 11 and the frame-char-height is 20. I have added a thin vertical yellow line to the left of the capital letter "A" as an example of how to draw custom images. Substitution of the character at point can be made programmatically using (char-after (point)) and taking that number -- which in this case is 65 for the capital letter "A" -- and substituting the appropriate variable -- e.g., (cond ((eq (char-after (point)) 65) cap-ltr-a-xpm) . . . -- and using that variable in the the overlay placement -- e.g., (overlay-put (make-overlay (point) (1+ (point))) 'display cap-ltr-a-xpm). This works very nicely for both truncated buffers and also with word-wrap because the display overlay property on a character in the middle of a word does not cause word-wrap to think that the first part of the word belongs at the end of the previous line. Of course, it will take time to create a custom library of favorite xpm images.

ImageMagick is capable of producing a semi-accurate xpm of a particular character based on a specific font family and size, but it was not as precise as I had hoped -- here is a link to instructions for using that external utility: https://stackoverflow.com/a/14168154/2112489 In a nutshell, the user should be prepared to spend time customizing the xpm images to his / her liking.

(defun xpm-example ()
(interactive)
"Doc-string"
  (let* (
      (cap-ltr-a-xpm `(image :type xpm :mask nil :ascent center :data
        "/* XPM */
        static char * letters_xpm[] = {
        /* columns rows colors chars-per-pixel */
        /* columns = 1 pixel in width -- see also (frame-char-width) */
        /* rows = 1 pixel in height -- see also (frame-char-height) */
        \"11 20 4 1\",
        \". c #000000\",
        \"+ c #FF0000\",
        \"@ c #7F0000\",
        \"% c yellow\",
        \"%..........\",
        \"%....++....\",
        \"%....++....\",
        \"%..++..++..\",
        \"%..++..++..\",
        \"%++......++\",
        \"%++......++\",
        \"%++......++\",
        \"%++......++\",
        \"%++......++\",
        \"%++......++\",
        \"%++++++++++\",
        \"%++++++++++\",
        \"%++......++\",
        \"%++......++\",
        \"%++......++\",
        \"%++......++\",
        \"%++......++\",
        \"%++......++\",
        \"%..........\"};"))  )
    (overlay-put (make-overlay (point) (1+ (point))) 'display cap-ltr-a-xpm)))