In general, you can't. The validity of a file name depends on the filesystem. A file name may be valid in one directory but not in another. There's no way to find out for sure without trying: on a network filesystem, the validity of the name may be enforced by the server without the client knowing the rules.
file-name-invalid-regexp allows you to filter out some invalid file names, which the operating system doesn't support at all. The value of
file-name-invalid-regexp just says “null bytes are not allowed” (the special character that's shown as a colored
^@ is a null byte). I think that null bytes are forbidden in file names on all the platforms that Emacs supports.
On most Unix-like operating systems, including Linux and macOS, every character other than a null byte is valid; a slash
/ cannot appear in a filename component but
file-name-invalid-regexp is about the whole path. Most Unix-like operating systems also have a maximum file name length: 255. Additional constraints may apply, for example when accessing a Windows filesystem.
On Windows, several ASCII characters are forbidden in file names (unlike Unix, Windows doesn't allow the special characters of its standard shell in file names). In addition, several base names are forbidden (regardless of the extension). And Windows has a maximum length as well. At least in the official 25.2 Windows binary,
file-name-invalid-regexp only reflects the forbidden characters.
Note that if you use Tramp to access files remotely, the file names are processed by Tramp, they bypass the local operating system completely, so
file-name-invalid-regexp doesn't apply. This is another reason why
file-name-invalid-regexp is pretty much useless.