I think I have found a solution, although It's quite hacky. Basically, I use the following loop to pass key presses to the fzf process:
(while (null fzf-result)
(command-execute (read-key-sequence nil)))
In the term exit handle I then set this variable, fzf-result, kill the buffer, and add one last key to the unread command events (this is to finally get ...