Quail-based Japanese conversion error

Question

The following error that I am reporting appears both in emacs-27.2 and emacs-28.2.

This question is written under the assumption that the reader is familiar with Quail and with Japanese syllabary.

When using (set-language-environment "Japanese") and (set-input-method "japanese") with Quail, when we type romaji, the displayed text is correctly converted to hiragana. If we want to convert the hiragana to katakana, we just follow the hiragana with "K", and the text display is correctly toggled to katakana. Typing "K" again will correctly toggle back to hiragana.

In most cases, this indeed works correctly. For example, typing "hoteru" correctly yields the hiragana "ほてる", and then following that with "K" switches it correctly to the katakana "ホテル". And then following that again with "K" correctly toggles it back to "ほてる".

However, when the word contains an embedded hyphen, the "K" toggling does not work properly.

For example, typing "ko-to" correctly yields the hiragana "こーと", and then typing "K" correctly converts it to the hiragana "コート". But then, if I type "K" again, the word does not get converted back to hiragana, and the katakana form remains.

This erroneous behavior occurs with all katakana words which contain an embedded hyphen. The existence of the hyphen in any word prevents that word from being toggled from katakana to hiragana when "K" is typed.

Can anyone suggest a way that this incorrect behavior can be fixed?

Thank you in advance.

Answer 1

I can replicate that in Emacs 29.1.

Typing K calls this command:

;; Convert Hiragana <-> Katakana in the current translation region.
(defun quail-japanese-toggle-kana ()
  (interactive)
  (setq quail-translating nil)
  (let ((start (overlay-start quail-conv-overlay))
        (end (overlay-end quail-conv-overlay)))
    (save-excursion
      (goto-char start)
      (if (re-search-forward "\\cH" end t)
          (japanese-katakana-region start end)
        (japanese-hiragana-region start end)))
    (setq quail-conversion-str
          (buffer-substring (overlay-start quail-conv-overlay)
                            (overlay-end quail-conv-overlay)))))

The problem is that the hyphen is a valid character for both of those character sets, and consequently the test (re-search-forward "\\cH" end t)¹ succeeds in finding Hiragana text within the region of Katakana characters, which in turn makes the command use japanese-katakana-region instead of japanese-hiragana-region (which effectively does nothing, as the text was already Katakana characters).

It's not entirely clear to me whether the following change is safe/robust, but you might try testing whether all characters in the region are Hiragana characters:

;; Convert Hiragana <-> Katakana in the current translation region.
(defun quail-japanese-toggle-kana ()
  (interactive)
  (setq quail-translating nil)
  (let ((start (overlay-start quail-conv-overlay))
        (end (overlay-end quail-conv-overlay)))
    (save-excursion
      (goto-char start)
      (save-restriction
        (narrow-to-region start end)
        (if (re-search-forward "\\`\\cH*\\'" end t)
            (japanese-katakana-region start end)
          (japanese-hiragana-region start end))))
    (setq quail-conversion-str
          (buffer-substring (overlay-start quail-conv-overlay)
                            (overlay-end quail-conv-overlay)))))

I recommend you report this (including all of these details), using M-x report-emacs-bug. I think you'll be in a better position than myself to establish whether my suggested change would be a sensible fix. Potentially it's necessary to instead determine whether there are Hiragana characters which are not also Katakana characters, if it's intended for it to be valid for there to additionally be characters in the region which are neither of those categories.

---

¹ "\\cH" matches characters in category H which is "Japanese Hiragana characters of 2-byte character sets."

Moreover, the character ー belongs to all of the following categories:

(category-set-mnemonics (char-category-set ?ー))
=> ".HKLj|"

Which are:

(seq-map (lambda (c) (format "%c: %s" c (category-docstring c)))
         (category-set-mnemonics (char-category-set ?ー)))

• .: Base: Base characters (Unicode General Category L,N,P,S,Zs).
• H: 2-byte Hiragana. Japanese Hiragana characters of 2-byte character sets.
• K: 2-byte Katakana. Japanese Katakana characters of 2-byte character sets.
• L: Strong L2R. Characters with "strong" left-to-right directionality, i.e. with L, LRE, or LRO Unicode bidi character type.
• j: Japanese.
• |: line breakable. While filling, we can break a line at this character.

See also M-x describe-categories