Normally you simply avoid binding the command to any key maps except for the modes you want it to be used in. But that doesn't stop the user from using "m-x" to run the command by name.
The convention on function names for mode-specific commands is mode-whatever, such as dired-do-rename-file. It is unlikely that the user will type that name, unless they really want it.
When defining your mode-only command with defun, you can give a list of applicable modes to the interactive declaration; this will prevent command name completion so that "m-x" won't see it elsewhere. This feature doesn't get used very often. For example, it is not used by dired or org modes, which are full of commands that don't work properly outside of buffers in their respective modes.
Your function can also try to test which mode(s) it is in, and do nothing unless it's the right mode. But the approach of figuring out the mode can be problematic, and you don't want to have to include such boilerplate at the beginning of all your functions. That's not the emacs way.