11

In Arabic as with some other languages there is what is called diacritics to enhance pronunciation. There is no convention on how many diacritics should be written for a single word. Some use the minimum (which I prefer) just enough to disambiguate pronunciation, whereas some use them superfluously or just for aesthetic calligraphic purposes. Thus, there is a wide variation on what and how many diacritics are associated with one word. When I do isearch-forward/backward by pressing C-s/r, problem arises when I type it in the search mini-buffer without diacritics it will not match the same word in the text if it had diacritics, making the task of looking for this word with its potential diacritics ever unsatisfactory.

Is there a way to make search/regexp search unaware of diacritics? I hope there would be an answer that can be extended to include regexp C-M-s/r and grep search that I use quite often in helm-projectile to look for a word in a multi-file latex projects.

Update
It would be nice to see that Emacs in all of its search functions doing the stripping off step on the text (from accents/diacritics/you name it) before matching step as a default behavior that might be turned off by a prefix on demand no matter what language is at hand. Typically, when I search for something I don't expect from the best editor (Emacs) to fail in this errand just because of some diacritics or accents that are rarely if ever needed to accomplish mundane text chores.

  • 1
    Look at the ucs-normalize-* functions in lisp/international/ucs-normalize.el. There is no pre-defined search folding for those, like there is with case folding, but you can at least normalize a region before searching it. A good implementation is probably a fairly complex task. – Ted Zlatanov Feb 5 '15 at 14:12
  • Can superuser.com/a/675172/233868 help? – Name Feb 5 '15 at 15:42
  • @Name, Arabic has much more possibilities of combinations of letters (26) with accents/diacritics, so it is not for Arabic. It seems there is no substitute for language-specific libraries. I can't believe that this has already been implemented in Microsoft Word and not in Emacs all those years back. – doctorate Feb 5 '15 at 16:02
  • 1
    Arabic has about 80 diacritics and 26 letters, making all combinations is a daunting task. There must be some way to strip the text of its diacritics, like what in php implemented: stackoverflow.com/a/25563250/1288722 - also implemented in Javascript: stackoverflow.com/a/7193622/1288722 – doctorate Feb 5 '15 at 16:31
  • Thought: is it not possible to run the string through that php cleansing function and then pass the result to something similar to helm-swoop? – Sean Allred Feb 9 '15 at 0:38
5

Here's a rough start, based on the list of combining characters in this answer (and then extended). (Marking this as community wiki — please edit and improve this!)

(defconst arabic-diacritics '(#x064b #x064c #x064d #x064e #x064f #x0650 #x0651 #x0652 #x0653 #x0654 #x0655 #x0670)
  "Unicode codepoints for Arabic combining characters.")
(defconst arabic-diacritics-regexp (regexp-opt (mapcar #'string arabic-diacritics)))

(defconst arabic-equivalents
  '(
    ;; "alef" is equivalent to "alef with hamza above" etc
    (#x0627 #x0623 #x0625 #x0622)))

;; (require 'cl-lib)    
;; (defun arabic-strip-diacritics (string)
;;   (cl-reduce (lambda (s c) (remove c s)) arabic-diacritics :initial-value string))

(defun arabic-search-without-diacritics (string)
  (interactive (list (read-string "Search for: " nil nil nil t)))
  (let ((regexp
         (apply #'concat
                (mapcar (lambda (c)
                          (let ((equivalents (assq c arabic-equivalents)))
                            (concat
                             (if equivalents
                                 (regexp-opt (mapcar #'string equivalents))
                               (regexp-quote (string c)))
                             arabic-diacritics-regexp "*")))
                        string))))
    (search-forward-regexp regexp)))

So if a buffer contains "الْحَمْدُ لِلَّهِ رَبِّ الْعَالَمِينَ", and I evaluate (arabic-search-without-diacritics "الحمد لله رب العالمين"), it finds the text. It also works interactively, as M-x arabic-search-without-diacritics.

Alternative approach:

Here's a full code example that demonstrates how diacritical and other nonspacing marks (Mn property) can be removed from normalized strings in regexp matches. It works with the examples given and IMO is the right approach.

(defun kill-marks (string)
  (concat (loop for c across string
                when (not (eq 'Mn (get-char-code-property c 'general-category)))
                collect c)))

(let* ((original1 "your Arabic string here")
      (normalized1 (ucs-normalize-NFKD-string original1))
      (original2 "your other Arabic string here")
      (normalized2 (ucs-normalize-NFKD-string original2)))
  (equal
   (replace-regexp-in-string "." 'kill-marks normalized1)
   (replace-regexp-in-string "." 'kill-marks normalized2)))
  • I added two more diacritics commonly used in Arabic to your nice list. This is the complete sorted list 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1648 -- fee free to update. – doctorate Feb 5 '15 at 19:37
  • The first function arabic-search-without-diacritics works well but breaks with some words, I don't know why like this one الأَ. Other caveat, I have always to set-input-method to arabic when I enter my string in mini-buffer, while in isearch-forward/backward function it remains there. – doctorate Feb 5 '15 at 19:43
  • kill-marks is the better approach to provide hassle-free text ready for all kinds of search. What is unclear to me is how to implement that on a whole buffer and then on multifiles? – doctorate Feb 5 '15 at 19:51
  • 1
    Thanks! is it possible to make it like isearch-forward/backward highlight all occurrences and the current one differently and by invoking s will move forward and r move backward? – doctorate Feb 6 '15 at 15:32
  • 2
    Discussion on emacs-devel: thread.gmane.org/gmane.emacs.devel/182483 – Ted Zlatanov Feb 9 '15 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.