I was investigating a problem with how an Emacs process (monroe.el) decodes messages formatted in bencode, where the process would not skip over the correct number of characters in the message, causing the connection to hang. Bencode encodes strings like <length>:<bytes>, where <bytes> is a series of bytes with no specified encoding. In the server's case, it sends UTF-8 where possible, because that's how the data being gathered on the other side happens to be encoded. The process is set to encode and decode in utf-8-unix.

Some responses from the server to monroe.el contain bytes that fall outside the valid range of ASCII and are also not valid UTF-8. It turns out that when the decoding filter uses byte-to-position to determine where to move the point to next, those invalid bytes were being counted with width 2 instead of 1 for purposes of moving over the string. To test this theory, I ran this code:

  (set-buffer-file-coding-system 'binary)
  (insert "testあ\335")
  (let ((i 1) pos res)
    (while (cddr (setq pos (cons (char-after (byte-to-position i)) (cons i (byte-to-position i)))))
      (push pos res)
      (setq i (1+ i)))
    (nreverse res)))

Note that \335 is supposed to be an eight-bit sequence displayed in Emacs representing a single byte, not a string as in the code above. This code outputs the following:

((116 1 . 1) (101 2 . 2) (115 3 . 3) (116 4 . 4) (12354 5 . 5) (12354 6 . 5) (12354 7 . 5) (4194269 8 . 6) (4194269 9 . 6) (nil 10 . 7))

This was not what I was expecting to see: the 4194269 at the end is apparently the "codepoint" of the eight-bit sequence \335, and it's repeated twice in the output, meaning more than one byte position in the buffer maps to it. This does not make sense to me, because it's supposed to represent the single byte 0xDD, or 221 in decimal, so there should only be one buffer position for it. Indeed describe-char shows as much:

             position: 132 of 366 (36%), column: 17
            character:   (displayed as  ) (codepoint 4194269, #o17777735, #x3fffdd)
              charset: eight-bit (Raw bytes 128-255)
code point in charset: 0xDD
               syntax: w    which means: word
             category: L:Left-to-right (strong)
             to input: type "C-x 8 RET 3fffdd"
          buffer code: #xDD
            file code: #xDD (encoded by coding system raw-text-dos)
              display: no font available

It shows the "buffer code" as 0xDD, correctly.

If I call (set-buffer-multibyte nil) inside the temporary buffer, I get this result instead.

((116 1 . 1) (101 2 . 2) (115 3 . 3) (116 4 . 4) (66 5 . 5) (221 6 . 6) (nil 7 . 7))

This time the 0xDD byte is mapped correctly, but the hiragana (which is proper UTF-8) is not, so the overall size of the string as binary still isn't correct.

The end goal is to be able to parse the following bencode:


where test is ASCII, is UTF-8 and \335 is the byte 0xDD.

Is there something more that needs to be setup for this to work correctly? I'm using Emacs 28.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.