The variable is org-preview-latex-process-alist.
Note that there are some xelatex org-preview-process-alist settings floating around on the web that are not correct. Xelatex outputs an enhanced dvi file that ends in .xdv, not dvi. dvipng cannot process these. This is why I experimented with luatex (didn't get it to work), and finally used xelatex and svg (which should give better rendering anyway).
'(org-preview-latex-process-alist
(quote
((dvipng :programs
("lualatex" "dvipng")
:description "dvi > png" :message "you need to install the programs: latex and dvipng." :image-input-type "dvi" :image-output-type "png" :image-size-adjust
(1.0 . 1.0)
:latex-compiler
("lualatex -output-format dvi -interaction nonstopmode -output-directory %o %f")
:image-converter
("dvipng -fg %F -bg %B -D %D -T tight -o %O %f"))
(dvisvgm :programs
("latex" "dvisvgm")
:description "dvi > svg" :message "you need to install the programs: latex and dvisvgm." :use-xcolor t :image-input-type "xdv" :image-output-type "svg" :image-size-adjust
(1.7 . 1.5)
:latex-compiler
("xelatex -no-pdf -interaction nonstopmode -output-directory %o %f")
:image-converter
("dvisvgm %f -n -b min -c %S -o %O"))
(imagemagick :programs
("latex" "convert")
:description "pdf > png" :message "you need to install the programs: latex and imagemagick." :use-xcolor t :image-input-type "pdf" :image-output-type "png" :image-size-adjust
(1.0 . 1.0)
:latex-compiler
("xelatex -no-pdf -interaction nonstopmode -output-directory %o %f")
:image-converter
("convert -density %D -trim -antialias %f -quality 100 %O")))))
To set the converter, use (setq org-preview-latex-default-process 'dvisvgm) ;; same as org-latex-create-formula-image-program, which is obsolete
To get unicode (I use it for Greek math characters) to work, you'll have to set (setq org-latex-inputenc-alist '(("utf8" . "utf8x")))
and
(add-to-list 'org-latex-packages-alist '("" "unicode-math"))
If you don't change dvi to xdv in the snippet above, the process hangs with a message "Creating image".
Here is a minimal org example that works for me
#+Title: Test of Unicode Operation
#+STARTUP: latexpreview
* unicode in latex fragments
Both unicode and latex-style Greek symbols work
\[ \alpha β \gamma δ\]
I also installed a fallback font for unicode symbols
(set-fontset-font "fontset-default" nil
(font-spec :size 40 :name "Symbola"))