Short answer: launch emacs directly from your shell ; chances are that both the latter and the former, in their current version, know how to handle ssh keyrings.
On Ubuntu 20.04.2 LTS / the latest ZSH and the latest Emacs, it works, Magit doesn't ask for the passphrase ; but if I use some menu system to launch Emacs, it then does.
When you defer the ...
And not to forget the hint in the Tramp manual:
Emacs computes the ‘dired’ options based on the local host but if
the remote host cannot understand the same ls command, then set
them with a hook as follows:
(when (file-remote-p default-directory)
(setq dired-actual-switches "-al"))))