Programmatically defined src-block execution doesn't expand nested noweb references on export

Question

I've run into a problem and I can't for the life of me figure out why this happens.

The idea is that I want to have a method of reaching into any named block of code, and grabbing out a a variable to print to the document at any point.

I've written an elisp src-block that does this by calling org-babel-execute-src-block directly on a coded python string. This programmatically defined block is then provided header vars and a prologue that is the output of the org-babel-expand-noweb-references function, which gives it access to any variable within the targeted source block. Finally this block is then ingested into the library of babel, and so can be accessed using the call_py-val-demo where ever it's needed.

The call_py-val-demo works correctly if I evaluate it interactively, but if I export the file and the call_ command occurs below the target source block, any nested noweb reference don't get expanded and the python code quietly fails if the variable depends on anything from the remaining noweb references.

Interestingly this doesn't occur just using call on a function within the file itself, only with a LOB function.

Am I making some subtle (or obvious) error in my code that causes this behaviour?

This is a demo file that can be used with the LOB function to demonstrate the effects I describe.

Version Info:
Emacs 28.1
Doom v3.0.0-pre
Org mode 9.6

LOB Block

#+NAME: py-val-demo
#+HEADER: :var blockName="", varName=""
#+HEADER: :noweb yes :results drawer :exports none
#+begin_src elisp
(org-babel-execute-src-block
        nil
        ;; the python code to be executed is the 2nd term of the list while the 3rd term are the header declarations
        (list "python" "returnMessageValue = \"{a}\".format(a=varName)"
                `((:results . "value drawer silent")
                (:var . ,(format "varName=\'%s\'" varName))
                (:prologue . ,(org-babel-expand-noweb-references (org-babel-lob--src-info blockName)))
                (:session . "none")
                (:return . "returnMessageValue")
                (:sync)
                (:noweb . "yes")
                (:cache . "no")
                (:hlines . "no")
                (:tangle . "no")
                (:exports . "results")
                )
                nil
                nil
                nil
                "(ref:%s)")
)
#+end_src

Test Document

#+title: Se Demo

#+BEGIN_SRC emacs-lisp :results file :exports none
(org-latex-export-to-pdf)
#+END_SRC

* setup :noexport:
#+NAME: nowebabove
#+BEGIN_SRC python
import numpy as np
someVars = 600
anotherVar = 93
#+END_SRC

* Some text
:PROPERTIES:
:header-args:python: :noweb yes :exports none :results output drawer
:END:

Context: above all but setup\\
call to demo - must chain noweb\\
call_py-val-demo(varName="tt", blockName="evenMoreWeb") {{{results(30)}}}\\
Call to demo - no chain\\
call_py-val-demo(varName="gg", blockName="evenMoreWeb") {{{results(1000000.0)}}}\\
call to demo - chain up overtop\\
call_py-val-demo(varName="someVars", blockName="evenMoreWeb") {{{results(600)}}}\\
\\
direct call to block\\
call_moreNoweb[:results value :return retVal]() {{{results(6.283185307179586)}}}


#+NAME: tryingNoweb
#+BEGIN_SRC python
<<nowebabove>>
tt = np.pi
fd = 4
print(fd)
#+END_SRC

Context: mid\\
call to demo - must chain noweb\\
call_py-val-demo(varName="tt", blockName="evenMoreWeb") {{{results(30)}}}\\
Call to demo - no chain\\
call_py-val-demo(varName="gg", blockName="evenMoreWeb") {{{results(1000000.0)}}}\\
call to demo - chain up overtop\\
call_py-val-demo(varName="someVars", blockName="evenMoreWeb") {{{results(600)}}}\\
\\
direct call to block\\
call_moreNoweb[:results value :return retVal]() {{{results(6.283185307179586)}}}

#+NAME: moreNoweb
#+BEGIN_SRC python
<<tryingNoweb>>
d = 40
retVal = 2*np.pi
#+END_SRC

#+NAME: evenMoreWeb
#+BEGIN_SRC python :tangle tangleTest.py
<<moreNoweb>>
gg = 1e6
#+END_SRC


Context: below all\\
call to demo - must chain noweb\\
call_py-val-demo(varName="tt", blockName="evenMoreWeb") {{{results(30)}}}\\
Call to demo - no chain\\
call_py-val-demo(varName="gg", blockName="evenMoreWeb") {{{results(1000000.0)}}}\\
call to demo - chain up overtop\\
call_py-val-demo(varName="someVars", blockName="evenMoreWeb") {{{results(600)}}}\\
\\
direct call to block\\
call_moreNoweb[:results value :return retVal]() {{{results(6.283185307179586)}}}

Answer 1

I've been able to come up with a workaround, but I haven't been able to figure out why nested nowebs don't expand if called from a LOB function with other source blocks around it.

The workaround was to:

1. create a temporary buffer that contained the contents of the file itself
2. Set org-default-header-args (locally) to contain :noweb yes
3. visit the target src-block
4. call org-babel-expand-noweb-references and then pass that string along to the :prologue

The upshot of this approach is that now I can even collect values from other files and have their noweb's properly expanded.

Looking at it this code feels kind of kludgy and inefficient and I worry the solution is unstable and may fail in a surprising way. So I'd love to know if anyone has a recommendation of an alternative approach or if there's an explanation for the previously described behaviour.

Updated LOB Block

#+NAME: py-val-demo
#+HEADER: :var fileName=(buffer-file-name)
#+HEADER: :var blockName="", varName=""
#+HEADER: :noweb yes :results drawer :exports none
#+begin_src elisp
(let* ((nowebstring "")
       (org-babel-default-header-args
            '((:session . "none")
              (:results . "replace")
              (:exports . "code")
              (:cache . "no")
              (:noweb . "yes")
              (:hlines . "no")
              (:tangle . "no"))))
        (with-current-buffer (create-file-buffer fileName)
                            (insert-file-contents fileName)
                            (org-babel-goto-named-src-block blockName)
                            (setq nowebstring (org-babel-expand-noweb-references))
                            (kill-buffer)
        )
        (org-babel-execute-src-block
                nil
                ;; the python code to be executed is the 2nd term of the list while the 3rd term are the header declarations
                (list "python" "returnMessageValue = \"{a}\".format(a=varName)"
                        `((:results . "value drawer silent")
                        (:var . ,(format "varName=\'%s\'" varName))
                        (:prologue . ,(format "%s" nowebstring))
                        (:session . "none")
                        (:return . "returnMessageValue")
                        (:sync)
                        (:noweb . "yes")
                        (:cache . "no")
                        (:hlines . "no")
                        (:tangle . "no")
                        (:exports . "results")
                        )
                        nil
                        nil
                        nil
                        "(ref:%s)")
        )
)
#+end_src