To only enable line numbers in "buffers that have code", you could add a hook to prog-mode-hook:
(add-hook 'prog-mode-hook #'display-line-numbers-mode)
(or whatever line numbering mode/function you're using), and make sure you don't have global-display-line-numbers-mode
somewhere in your init. However, "buffers with filenames" and "buffers with code" don't mean the same thing - only buffers with prog-mode
derived major modes (which includes most major language modes) will get line numbers with this method.
You could add display-line-numbers-mode
to something like find-file-hook
to have it turned on whenever a file is visited.