You ask about initiation of the selection by
M-x. I don't know whether there is a simple way to cause use of the dialog box even when you do not use a menu or tool-bar button (typically initiated with a mouse). (But it's also not clear to me what
M-x has to do with selecting a file, unless you mean
M-x find-file or some such.)
Anyway, the reason the file selection box is shown is likely because of the settings of options
use-dialog-box, which you can customize.
In addition, since you have GTK, you can customize options
x-gtk-file-dialog-help-text, which affect the behavior of the file selection box.
See the Emacs manual, node Dialog Boxes.
C-h v use-file-dialog tells us:
use-file-dialog is a variable defined in
C source code.
Its value is
nil means mouse commands use a file dialog to ask for files.
This applies to commands from menus and tool bar buttons even when
they are initiated from the keyboard. If
that disables the use of a file dialog, regardless of the value of
This variable was introduced, or its default value was changed, in
version 22.1 of Emacs.
To your second question: You can test whether Emacs is running in a window system (GUI) or in terminal mode, using predicate
C-h f tells us:
display-graphic-p is a compiled Lisp function in
(display-graphic-p &optional DISPLAY)
Return non-nil if
DISPLAY is a graphic display.
Graphical displays are those which are capable of displaying several
frames and several different fonts at once. This is true for displays
that use a window system such as X, and false for text-only terminals.
DISPLAY can be a display name, a frame, or
nil (meaning the selected