How to search an Arabic word in text without its diacritics/accents?

Question

In Arabic as with some other languages there is what is called diacritics to enhance pronunciation. There is no convention on how many diacritics should be written for a single word. Some use the minimum (which I prefer) just enough to disambiguate pronunciation, whereas some use them superfluously or just for aesthetic calligraphic purposes. Thus, there is a wide variation on what and how many diacritics are associated with one word. When I do isearch-forward/backward by pressing C-s/r, problem arises when I type it in the search mini-buffer without diacritics it will not match the same word in the text if it had diacritics, making the task of looking for this word with its potential diacritics ever unsatisfactory.

Is there a way to make search/regexp search unaware of diacritics? I hope there would be an answer that can be extended to include regexp C-M-s/r and grep search that I use quite often in helm-projectile to look for a word in a multi-file latex projects.

Update
It would be nice to see that Emacs in all of its search functions doing the stripping off step on the text (from accents/diacritics/you name it) before matching step as a default behavior that might be turned off by a prefix on demand no matter what language is at hand. Typically, when I search for something I don't expect from the best editor (Emacs) to fail in this errand just because of some diacritics or accents that are rarely if ever needed to accomplish mundane text chores.

Answer 1

Here's a rough start, based on the list of combining characters in this answer (and then extended). (Marking this as community wiki — please edit and improve this!)

(defconst arabic-diacritics '(#x064b #x064c #x064d #x064e #x064f #x0650 #x0651 #x0652 #x0653 #x0654 #x0655 #x0670)
  "Unicode codepoints for Arabic combining characters.")
(defconst arabic-diacritics-regexp (regexp-opt (mapcar #'string arabic-diacritics)))

(defconst arabic-equivalents
  '(
    ;; "alef" is equivalent to "alef with hamza above" etc
    (#x0627 #x0623 #x0625 #x0622)))

;; (require 'cl-lib)    
;; (defun arabic-strip-diacritics (string)
;;   (cl-reduce (lambda (s c) (remove c s)) arabic-diacritics :initial-value string))

(defun arabic-search-without-diacritics (string)
  (interactive (list (read-string "Search for: " nil nil nil t)))
  (let ((regexp
         (apply #'concat
                (mapcar (lambda (c)
                          (let ((equivalents (assq c arabic-equivalents)))
                            (concat
                             (if equivalents
                                 (regexp-opt (mapcar #'string equivalents))
                               (regexp-quote (string c)))
                             arabic-diacritics-regexp "*")))
                        string))))
    (search-forward-regexp regexp)))

So if a buffer contains "الْحَمْدُ لِلَّهِ رَبِّ الْعَالَمِينَ", and I evaluate (arabic-search-without-diacritics "الحمد لله رب العالمين"), it finds the text. It also works interactively, as M-x arabic-search-without-diacritics.

Alternative approach:

Here's a full code example that demonstrates how diacritical and other nonspacing marks (Mn property) can be removed from normalized strings in regexp matches. It works with the examples given and IMO is the right approach.

(defun kill-marks (string)
  (concat (loop for c across string
                when (not (eq 'Mn (get-char-code-property c 'general-category)))
                collect c)))

(let* ((original1 "your Arabic string here")
      (normalized1 (ucs-normalize-NFKD-string original1))
      (original2 "your other Arabic string here")
      (normalized2 (ucs-normalize-NFKD-string original2)))
  (equal
   (replace-regexp-in-string "." 'kill-marks normalized1)
   (replace-regexp-in-string "." 'kill-marks normalized2)))