Managing path for reproductible scientific paper using session, org-babel and org-mode

Question

I'm starting using org-mode and org-babel to support my day by day scientific work and produce reproductibles papers.

Lot of people use rMarkdown/jupyter to do that, but i'm interested by investing some time to do more or less the same things using org-babel and org-mode.

Running chunk of code using org-babel is as simple as running chunk of code with RMarkdown/KnitR, and it seems org-babel is much powerfull on this side, so no problem with that actually

For me, problem arrives when i try to manage path, and start using :session property, especially with shell (sh/bash).

I define the workspace :dir for source code like this :

:PROPERTIES:
:export_file_name: reproductiblepaper
:header-args: :dir ~/home/xxx/reproductible_paper tangle:yes session: reproductible
:END:

It's common to use git or any cvs to download and run code producing result, so naively i wrote in my org code :

#+begin_src sh
git clone [email protected]:xxx/yyy/mygitproject.git
#+end_src

Folder appear in ~/home/xxx/reproductible_paper/mygitproject/ as expected.

After that, i don't want to rewrite my :dir path for each #+BEGIN_SRC ... #+END_SRC block, so i use the :session capacity, and i wrote

#+begin_src sh
cd mygitproject
#+end_src

BUT, using this strategy cannot work.

When you write and export org file multiple time in a writing/publishing workflow (write -> execute/publish -> write), each command rerun using the same shell session !, bypassing the :header-args: dir: defined in property.

The result was weird, and multiple run create an infinity of nested folder due to git clone and cd command.

Following your advice what is the best way to manage path in this use case ? Killing session each run ? Use environment variable without :session to define directory path ?

UPDATE 1

a) Using absolute path, env substitution, and session :

:PROPERTIES:
:export_file_name: reproductiblepaper
:header-args: :dir ~/home/xxx/reproductible_paper tangle:yes session: reproductible
:END:

#+NAME: absfolder
#+BEGIN_SRC sh
ABS="/home/xxx/reproductible_paper"
#+END_SRC

#+begin_src sh
git clone [email protected]:xxx/yyy/mygitproject.git
#+end_src

#+begin_src sh :noweb yes
<<absfolder>>
cd $ABS/mygitproject
pwd
#+end_src

If you use this solution, and killing each time the session, it works, but you cannot use interactive evaluation without creating an unstable state in your shell session. Why not.

But to do this in a clean way i need :

• a way/command to kill/reset session at each re-export.
• a starting script which clean previous state files/folders manipulation

Any advice do that in org ?

b) I suppose there is probably another way to do the same thing without using a session, using :var or :dir and using link dependency between each code block ?

Answer 1

Your use case is a bit unusual, but I think you can get what you want with a few tweaks.

As you've discovered, when you use the session argument, you will re-use the same active bash/R session each time you re-run your code. That's an issue if you use cd. You could protect against this by including a block resetting your directory at the start of your file:

#+begin_src sh
cd ~/home/xxx/reproductible_paper
#+end_src

This will allow you to continue using relative paths for the rest of your script. If you don't want this block to appear in your final document, you can hide it from the output with the header argument :exports none

Another thing that might help is to selectively exclude some code blocks from evaluation at export. You probably don't need to clone the same git repository every time you change some code further down in your document. You can control this with the eval header:

#+begin_src sh :eval noexport
git clone [email protected]:xxx/yyy/mygitproject.git
#+end_src

The noexport option will allow you to interactively run the code the first time you need it, or anytime you want to repeat the code (e.g., when the repository has been updated). But it won't get run automatically whenever you export your document.

Answer 2

I'm not sure how this is specific to Org. If you write a script that changes state dependent on current state, and don't unchange/reset the state elsewhere in the script, and repeatedly run the script, the state will be changed each time you run it.

What about cd ${project_root}/mygitproject?, where ${project_root} is absolute? If you use the absolute path, not the relative path, your problem may be solved.