You can define conditions for face customizations, like "terminal mode" or "minimal colors" in the customization editor:
When customizing a face, click the [State] button and select "Show all display specs". Then you can set up the conditions for further specialization of the face.
For me, this resulted in this snippetHere's a minimal variant to set the background color to black when the frame is a tty.:
(custom-set-faces
'(default (
(((type tty) (min-colors 256))
(:background "black"))
(t
(:background "#181a26")))
))