This behaviour is actually a known and documented limitation of this port, see README-mac (emphasis mine):
The Mac port doesn't support multi-tty with GUI. The developer has no
idea how to detach Emacs as a GUI application from Window Server or
Dock without separating a GUI process (not thread) from the main Emacs
(Lisp evaluator) process. TTY-only multi-tty is supposed to work.
In other words, it's impossible to attach a client TTY frame to a server running in a GUI frame.
You can run two different server instances, though, one for the GUI and another for TTY, by setting server-name accordingly, i.e.
(setq server-name "gui-server")
(server-start)
You have to explicitly give the socket file name to emacsclient now:
$ emacsclient -f "$TMPDIR/emacs-$(id -i)/gui-server …
Both instances will not be able to share any state at all, though, and they'll potentially override each other's state and history in ~/.emacs.d.
In other words, both have separate buffer lists, undo history, save place history, etc., the instance which exists lasts wins for all files in ~/.emacs.d (i.e. save place history, minibuffer history, projectile project list, etc.) unless you give each data file a unique name, and if you save custom variables in one instance and then in the other, the values from the first instance are lost.
Emacs is really not made two run independently in two long-running separate interactive instances.
Finally, please don't refer to it as “railwaycat” port. This variant of Emacs is actually maintained by Yamamoto Mitsuharu; the Github account railwaycat just maintains the Homebrew formula and keeps a Git mirror. “railwaycat port” doesn't really give proper attribution.
