What I'm actually trying to do is ... calling read-number from a separate function (my-command) with a buffer-local variable as HIST.
As read-number doesn't have its own history list, you could let-bind minibuffer-history to your buffer-local history list around your call to read-number.
(defvar-local local-history nil
(defvar my-hist nil)
(defun my-function (hist)
(read-from-minibuffer "> " nil nil nil hist))
(defun my-command ()
(add-to-history 'my-hist (my-function 'my-hist)))
Don't quote hist when you pass it to read-from-minibuffer. You want to pass its value, e.g. the symbol my-hist, and not ...