defalias is useful when you want to give some existing function an alternative name. But you want to change what a certain key does, and that's a different problem.
You have to bind
"RET" to some command in some keymap, a problem that can be broken down into two smaller problems: "what keymap?" and "what command?".
To find the appropriate keymap you can use
rgrep in the directory containing the Magit source to search for
Turns out you actually have to modify two keymaps
magit-hunk-section-map and that instead of
"RET" you should use
[remap magit-visit-thing], though the former would work too.
(define-key magit-file-section-map [remap magit-visit-thing] '???)
(define-key magit-hunk-section-map [remap magit-visit-thing] '???)
Looking at the other matches and the code in close approximation suggests that no variation exists that behaves the way you want, but defining it is easy: copy
magit-diff-visit-file, replace the body with a call to that function and adjust the name, signature and
(defun magit-diff-visit-file-other-window (file)
(interactive (list (--if-let (magit-file-at-point)
(user-error "No file at point"))))
(magit-diff-visit-file file t))