org-mode How to use latex nested environments?

Question

I want to use a matrix environment that is included in an outer environment, so:

Environment:
\begin{equation*}
\left|
\begin{matrix}
a_1 & b_1\\
a_2 & b_2
\end{matrix}
\right|
\end{equation*}

Display--math:
\[
\left|
\begin{matrix}
a_1 & b_1\\
a_2 & b_2
\end{matrix}
\right|
\]

Inline--math:
\(
\left|
\begin{matrix}
a_1 & b_1\\
a_2 & b_2
\end{matrix}
\right|
\)

But for the second and third cases (display-math and inline-math) the preview and export to latex are failed:

exported LaTeX:

Environment:
\begin{equation*}
\left|
\begin{matrix}
a_1 & b_1\\
a_2 & b_2
\end{matrix}
\right|
\end{equation*}

Display--math:
$\backslash$[
\left|
\begin{matrix}
a_1 & b_1\\
a_2 & b_2
\end{matrix}
\right|
$\backslash$]

Inline--math:
$\backslash$(
\left|
\begin{matrix}
a_1 & b_1\\
a_2 & b_2
\end{matrix}
\right|
$\backslash$)
\end{document}

org-mode preview:

Emacs 26.3

Org-mode 9.3.8

Answer 1

So, this is not a bug:

nested \begin{...} should not start the line and therefore is not seen as starting a new LaTeX block:

Inline-math:
\(
\left|\begin{matrix}
a_1 & b_1\\
a_2 & b_2
\end{matrix}\right|
\)