AucTeX redefines the binding for $ to be the command TeX-insert-dollar. That command checks whether you are in a math environment already and complains if you try to close it the "wrong" way. Doing C-h f TeX-insert-dollar shows you the doc string of the function:
Insert dollar sign.
If current math mode was not entered with a dollar, refuse to
insert one. Show matching dollar sign if this dollar sign ends
the TeX math mode and ‘blink-matching-paren’ is non-nil.
When outside math mode, the behavior is controlled by the variable
‘TeX-electric-math’.
With raw C-u prefix, insert exactly one dollar
sign. With optional ARG, insert that many dollar signs.
Note the part that says: If current math mode was not entered with a dollar, refuse to insert one.
BTW, it's not clear to me what the \[ ... \] delimiters are buying you. If you get rid of them, then AucTeX will not complain about the $ signs, and you still have a centered display. CORRECTION: It's not centered but you can do that with \begin{center}...\end{center}.
Alternatively, keep the display but change the font of the text of the nodes:
\[
\begin{tikzpicture}[every text node part/.style={font=\itshape}]
\node (X) {X};
\node[right of= X] (Y) {Y};
\draw[->] (X.east) -- (Y.west);
\end{tikzpicture}
\]