Both the alt key and the esc key are setting the "Meta" modifier on the key.
Differentiating them without v significant collateral damage by breaking other meta binding is going to be difficult.
The easiest change to achieve what you desire might be to rebind something else to the scroll-down-command' (which is what command
M-v' (which is what M-v
is bound to by default).
A good option might be to define C-c v' to v
scroll-down-command' to scroll-down-command
From the manual:
Sequences consisting of C-c
C-c
and a letter (either upper or lower case) are reserved for users; they are the only sequences reserved for users, so do not block them.