You can make the face used for math and the so-called sedate face used for macro names inherit from fixed-pitch, and activate variable-pitch-mode in the LaTeX mode hook:
(set-face-attribute 'font-latex-sedate-face nil :inherit 'fixed-pitch)
(set-face-attribute 'font-latex-math-face nil :inherit 'fixed-pitch)
(add-hook 'LaTeX-mode-hook #'variable-pitch-mode)
You already discovered yourself that MathJax does not offer all the finesses of LaTeX.
That is the main reason why you can use another LaTeX-backend for HTML export with the help of the tex option of the html exporter. Example:
Choosing dvipng as LaTeX backend, your document is exported fine to HTML.
That is already built-in if you use texfrag-mode in Orgmode instead of the built-in LaTeX fragment support of Orgmode.
Just regenerate the LaTeX fragments with C-c C-p C-d after changing the size with C-x C-+ C-+ .... The size of the LaTeX fragments will fit the size of the surrounding text.
Normal text size:
After pressing C-x C-+ C-+ C-x and C-c C-p C-d:...
This is a problem with the CM fonts. The link shows some workarounds, but for your purposes (going from an org mode file to a PDF) I think the easiest/best thing to do is to use a different set of fonts. You can try Palatino e.g. by adding this to your org mode file:
You'll have to install whatever fonts you decide to ...