As suggested by @Drew, this can be achieved by setting
isearch-filter-predicate appropriately. Indeed, the docstring for
C-h f query-replace
Ignore read-only matches if ‘query-replace-skip-read-only’ is non-nil,
ignore hidden matches if ‘search-invisible’ is nil, and ignore more
matches using ‘isearch-filter-predicate’.
Here's a proposal allowing to toggle between "normal" and "math only" query replace:
(defun isearch-filter-visible-and-texmath (beg end)
"Possible value for isearch-filter-predicate."
(and (isearch-filter-visible beg end)
(save-excursion (goto-char beg) (texmathp))))
(defun toggle-query-replace-in-maths ()
"Toggle restricting query-replace to TeX math content."
(if (eq isearch-filter-predicate #'isearch-filter-visible)
(message "Query replace now only in math")
(message "Query replace back to normal")
setq-local is used so that the change is buffer local (it probably does not make sense for non tex buffers, anyway).
I don't find it totally satisfactory, since it will break other changes that could have been done to
isearch-filter-predicate. Maybe someone will propose something better!