Finally, I abandoned relying on the current directory as it was too intrusive and other things like ido where ending being affected...
Instead, I use added my build directory to
(add-to-list 'compilation-search-path "/path/to/build")
Found on this question: How to adjust the path that Emacs' compile-goto-error gets from the compilation buffer?
The question also explains that the directory matcher can change the directory automatically if some particular strings are emitted:
Entering directory `...'
Leaving directory `...'
This is also customizable apparently.