How to strip invalid UTF-8 characters from a string?

Question

TL;DR:

I have a string with characters that are unencodable in UTF-8, and I need to remove these characters before writing the string to a file.

For instance, the following snippet creates a file that contains an invalid UTF-8 byte sequence. I would like this sequence to just not be present in the output file.

(let ((coding-system-for-write 'utf-8))
  (write-region "This is invalid: \x19008e" nil "~/outputfile"))

By checking this character with describe-char, I can see Emacs knows it is not encodable through UTF-8. So there must be a way to detect these characters and remove them.

---

The Background

I have an emacs-lisp script which converts docstrings to html files, which must be entirely in UTF-8. The following snippet shows essentially what I've been doing so far.

(with-temp-file output-file
  (insert symbol-name "\n" symbol-documentation)
  (set-buffer-file-coding-system 'no-conversion))

Where symbol-documentation is the string returned by describe-variable (the content of the Help buffer).

This seemed to work fine at first, but now I've realised this was causing some files to contain byte-sequences that are not valid UTF-8. Some perpetrators are tibetan-precomposition-rule-alist, language-info-alist, composition-function-table. By manually checking their documentation, I can see my Emacs indeed fails to comprehend part of the content.

Q: How can I ensure only valid UTF-8 byte sequences are written to the output file?
I'm perfectly fine with skipping these invalid sequences entirely (instead of trying to understand them). I just want to make sure the output file is valid.

I tried changing no-conversion to utf-8, but then my script gets interrupted by a Select coding system: prompt, and the problem still happens.

Answer 1

Why not use the same mechanism that describe-char uses to identify whether a character is encodable using utf-8 i.e. use encode-coding-char. From the documentation

Encode CHAR by CODING-SYSTEM and return the resulting string. If CODING-SYSTEM can't safely encode CHAR, return nil. The 3rd optional argument CHARSET, if non-nil, is a charset preferred on encoding.

(defun my-remove-unencodeable-chars (string)
  (string-join
   (delq nil (mapcar (lambda (ch) (encode-coding-char ch 'utf-8 'unicode))
               string))))

Answer 2

You want to let-bind coding-system-for-write to utf-8 around the call to write-region. This won't prompt you, even if there are characters that can't be encoded.