Is it possible to set key bindings specific to buffer?

I have the following in one of my org files:

# Local Variables:
# eval: (local-set-key (kbd "<f10>") 'some-custom-defun-specific-to-this-buffer)
# End:

But this binding persists for other org-mode files too. If I open a file with a different major mode, then my default key binding kicks in.

It looks like local-set-key is setting bindings specific to the major-mode not specific to the buffer.

How can I have bindings that activate only in a particular buffer?