ssh-agent and hoping for the best is not enough.
ssh-agent hands out the decrypted private key to everyone asking on a specific socket. The problem is that the socket is not always the same and therefore only those processes that know the currently used socket can benefit.
ssh-agent is started it outputs the socket and its pid.
SSH_AUTH_SOCK=/tmp/ssh-XX4LkMJS/agent.26916; export SSH_AUTH_SOCK;
SSH_AGENT_PID=26917; export SSH_AGENT_PID;
echo Agent pid 26917;
The intended use is
$ eval $(ssh-agent)
By evaluating the output some environment variables are set and exported so that all subprocesses can see them too.
The reason that a
git process started by Magit inside Emacs doesn't know the socket is that the
emacs instance is not a subprocess of any process whose environment contains values for the necessary variables.
You can confirm this by running
eval $(ssh-agent); ssh-add ~/.ssh/id_rsa in a terminal, and then in the same terminal
emacs &. From that
emacs instance you will be able to push without being asked for the passphrase again.
Now you have to arrange for these variables to be always set in
emacs's environment, regardless of how it was started.
The way I do this is that I start the agent and add the keys using the
keychain utility when starting an interactive shell. That causes the variables to be set in all shells, and if I did then type
emacs & in a shell, then
emacs would have access too. But that's not how I usually start Emacs, so I also use
keychain-environment.el, which looks for a file created by
keychain and then sets the environment variables it finds in there.
I recommend you read OpenSSH Key Management, Part 2 (the other parts too). It was written by the author of
keychain. Also read the description of
keychain-environment.el (In particular note that you still have enter the passphrase from outside Emacs once.)