Monkey patch and restore builtin set-window-dedicated-p during advice

Question

I would like to make set-window-dedicated-p a NOP when called from xref--show-pos-in-buf. Below is an attempt to do so using advice. It doesn't work, I think because fset is not really copying the original function definition, just pointing things at each other. How can I either fix this approach to make it work, or use an alternative approach to accomplish my overall goal?

Overall goal: The way I have things set up, when I run an xref command like xref-find-references, buffer *xref* is shown in a full window which is how I like. But, when I hit RET to visit a reference, the location opens in a new frame. From what I can tell, this is because xref--show-pos-in-buf dedicates *xref* which seems to cause display-buffer to think the best solution is to display the buffer in a new frame. I never want a new frame, so my overall goal here is that *xref* and anything I select in it always open in the same frame I am already using.

(defun sham-set-window-dedicated-p (win flag)
  "Don't dedicate WIN, only return FLAG."
  flag)

(fset 'original-set-window-dedicated-p (symbol-function 'set-window-dedicated-p))

(defadvice xref--show-pos-in-buf (around my-xref--show-pos-in-buf activate)
  "Monkey patch set-window-dedicated-p so it doesn't do anything."
  (fset 'set-window-dedicated-p sham-set-window-dedicated-p))
  ad-do-it
  (fset 'set-window-dedicated-p (symbol-function 'original-set-window-dedicated-p)))

Answer 1

Both fset and defalias can point to a symbol or its current function definition, depending on whether you pass them the symbol or symbol-function applied to the symbol. The latter gets the value of symbol-function at the time of aliasing. The former gives you the value of symbol-function at the time the alias in invoked - it is essentially an indirection.

I think maybe you forgot to quote sham-set-window-dedicated-p here:

(fset 'set-window-dedicated-p sham-set-window-dedicated-p)

Answer 2

Notice that your code will misbehave if ad-do-it ends up signalling an error (so the subsequent fset is not executed any more to restore the normal state).

Here's a slightly more verbose way to get what you want, which I think works better (e.g. it lets C-h f show you what's going on, and it does not misbehave when you use the new threading facilities):

(defvar my-inhibit-set-window-dedicated nil)
(advice-add 'set-window-dedicated-p :around
  (lambda (orig-fun &rest args)
    "Add inhibitor variable `my-inhibit-set-window-dedicated'."
    (unless my-inhibit-set-window-dedicated
      (apply orig-fun args))))

(advice-add 'xref--show-pos-in-buf :around
  (lambda (orig-fun &rest args)
    "Inhibit `set-window-dedicated-p'."
    (let ((my-inhibit-set-window-dedicated t))
      (apply orig-fun args))))