To only enable line numbers in "buffers that have code", you could add a hook to prog-mode-hook:
(add-hook 'prog-mode-hook #'display-line-numbers-mode)
(or whatever line numbering mode/function you're using), and make sure you don't have
global-display-line-numbers-mode somewhere in your init. However, "buffers with filenames" and "buffers with code" don't mean the same thing - only buffers with
prog-mode derived major modes (which includes most major language modes) will get line numbers with this method.
You could add
display-line-numbers-mode to something like
find-file-hook to have it turned on whenever a file is visited.