Q1: If you look at the description of org-map-entries (with C-h f org-map-entries), you will see that it takes optional arguments, one of which is scope, which can take a bunch of different values, one of which is tree. The description says this:
SCOPE determines the scope of this command. It can be any of:
‘nil’
The current buffer, respecting the restriction, if any.
‘tree’
The subtree started with the entry at point.
...
Forgetting the agenda stuff, try changing the function to this:
(defun my/sort-subtree-org-entries ()
(interactive)
(let ((fun (lambda ()
(condition-case nil
(org-sort-entries nil ?d)
(user-error t)))))
(org-map-entries fun nil 'tree)))
Then position the cursor at the start of the subtree you want to sort and say M-x my/sort-subtree-org-entries.
Note that it is important to be at the right place when you invoke the function. If you place the cursor at the beginning of the "Unrelated tree" and invoke the function, the "Unrelated tree" will be sorted. It is up to you to make sure that you are at the right position in the buffer when you invoke the function.
Q2: Of course it is possible for a command to "span separate buffers" as you put it, but it does not happen magically: you (i.e. your function) have to make sure that the context is correct for each operation. You do that by selecting the correct buffer and moving point to the right place in the buffer (if necessary).
In particular, if you look at the doc string of org-agenda-redo with C-h f org-agenda-redo, you will see that it says Rebuild possibly ALL agenda view(s) **in the current buffer**. IOW, you have to make sure that you are in the agenda buffer when you evaluate the call to org-agenda-redo. Similarly, you have to be in the file buffer where the entries to be sorted are, before you invoke my/sort-subtree-org-entries. Moreover, you have to be in the right place in the file buffer (at the beginning of the subtree to be sorted) before you invoke it.
I'll do the easy part: you are in the file buffer and you want to sort a subtree and regenerate the agenda. I don't want to modify the sort function above to do two things, because the one thing that it does is useful and I might want to do that alone, not in combination with the agenda. Instead, I want to write another function that selects the agenda buffer and regenerates the agenda, and a third function that composes these two: it sorts the subtree by calling the first function and then regenerates the agenda by calling the second function. Here's the code:
(defun my/switch-to-agenda-and-redo ()
(with-current-buffer org-agenda-buffer
(org-agenda-redo)))
(defun my/sort-subtree-regenerate-agenda ()
(interactive)
(my/sort-subtree-org-entries)
(my/switch-to-agenda-and-redo))
Now you can put your cursor in the right place at the beginning of the subtree to be sorted and say M-x my/sort-subtree-regenerate-agenda.
I should point out that if you don't have an agenda buffer open, you are going to get an error: the my/switch-to-agend-and-redo function badly needs error handling, but the basic idea is there.
Going in the other direction (from the agenda, switch to the file buffer and move to the right point, sort and then regenerate the agenda) is harder, because you have to figure out what file buffer the entry came from, then figure out how far up the tree to go to get to the right place for sorting. I think there is enough information in the agenda to determine the file buffer, but I don't think there is enough information to figure out the subtree (let's say you want to sort the second level subtree, but the entry in the agenda is a fourth level entry: how will the function know to go up to the second level and not stop at the third?) So I think your specification of the problem is incomplete: you will somehow have to provide that information to the function.
BTW, the code is completely untested.
