Org-mode to Latex matrix automatically escapes ampersand

Question

I am trying to write LaTex matrizes in org-mode. It works fine with exporting tables via #+attr_latex :mode math :environment matrix.

However, when I try

\begin{pmatrix}
1 & 2\\
3 & 4\\
\end{pmatrix}

this exports to

\begin{pmatrix}
1 \& 2\\
3 \& 4
\end{pmatrix}

in the latex document. And that gives me a literal & in the final latex document.

Is this default behaviour? How can I fix this?