Evil-mode remap universal-argument to C-f in normal-mode, without weird C-u behavior

Question

This .emacs file is sufficient to reproduce the problem.

(package-initialize)
(setq evil-want-C-u-scroll t)
(require 'evil)
(define-key evil-normal-state-map (kbd "C-f") #'universal-argument)
(evil-mode +1)

However, when I am applying multiple consecutive universal arguments while in normal mode, C-u is being treated as universal-argument (which is inappropriate). I can stack both C-u and C-f and get C-f C-f C-u C-f C-f in my modeline. The semantics appear to be the same 4^[number of universal argument things], but I'd like to disable this unwanted behavior and use just C-f for universal argument in normal mode.

EDIT:

So it turns out what is actually happening is that after C-f the universal-argument-map is invoked and in that map the C-u key is bound to universal-argument. It looks like the purpose of this map and the related transient mode universal-argument--mode is to enable digits and - to modify the universal argument after C-u has been pressed.

It seems like any solution to this problem would involve defining a function similar to universal-argument and binding C-f to that.

EDIT:

wasamasa's suggestion works great, but this is what I've settled on so as to avoid modifying the behavior of evil-emacs-state (or when evil-mode is deactivated)

(defvar normal-mode-universal-argument-map universal-argument-map
  "swap C-f and C-u in universal-argument-map")
(define-key normal-mode-universal-argument-map (kbd "C-u") nil)
(define-key normal-mode-universal-argument-map
  (kbd "C-f") #'universal-argument-more) 

(defun normal-mode-universal-argument ()
  "universal-argument with C-f and C-u swapped"
  (let ((universal-argument-map normal-mode-universal-argument-map))
    (universal-argument)))

Answer 1

Remap C-u in both the global keymap and the transient one. Don't forget to map the freed C-u in both the global keymap and Evil's normal state keymap to something more useful and to poke a hole in Evil's normal state keymap for the universal argument to be picked up from the global keymap:

(define-key global-map (kbd "C-f") 'universal-argument)
(define-key universal-argument-map (kbd "C-u") nil)
(define-key universal-argument-map (kbd "C-f") 'universal-argument-more)
(define-key global-map (kbd "C-u") 'kill-whole-line)
(eval-after-load 'evil-maps
  '(progn
     (define-key evil-motion-state-map (kbd "C-f") nil)
     (define-key evil-motion-state-map (kbd "C-u") 'evil-scroll-up)))

Answer 2

Try using define-key instead of the variable approach

(define-key evil-normal-state-map "\C-u" 'evil-scroll-up)