org can't execute on remote machine

Question

I have a .org file on my laptop that I use to run code on a remote server. However, when I try to execute the code I'm getting Tramp errors.

My local org file has the following header:

# -*- org-export-babel-evaluate: nil -*-
#+PROPERTY: header-args:bash :dir /ssh:gpsc:./rubus

gpsc is a multihop connection which I've configured in Emacs as:

(add-to-list 'tramp-default-proxies-alist 
             '("gpsc" 
               nil 
               "/ssh:USER@REMOTEHOST:"))

This works fine if I want to open a file for editing, e.g. C-x C-f /ssh:gpsc:~/.bashrc.

However, if I try to execute an org code block:

#+begin_src bash 
  date
  ls -lh ./
#+end_src

it fails, with the following logged in my *Messages* buffer:

Executing Bash code block at position 439...
Tramp: Encoding local file ‘/home/smithty/.cache/emacs/tramp.Zr0vSS’ using ‘base64-encode-region’...done
Tramp: Decoding remote file ‘/ssh:gpsc:/tmp/ob-input-YvoIhU’ using ‘base64 -d -i >%s’...failed
tramp-error: Couldn’t write region to ‘/ssh:gpsc:/tmp/ob-input-YvoIhU’, decode using ‘base64 -d -i >%s’ failed

What's stranger, is that this same file initially worked, and now doesn't (with the errors described above), without my changing the underlying configuration.

I'm not sure where to look for the problem, can anyone help debug this?

Emacs 30.0.50
org 9.6.7
tramp 2.7.0-pre

Update

I've posted the full debug log when I set tramp-verbose to 6 below. The final bit with the error is:

tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:02.196152 tramp-sh-handle-write-region (3) # Encoding local file ‘/home/smithty/.cache/emacs/tramp.JD7UfG’ using ‘base64-encode-region’...
16:54:02.196517 tramp-sh-handle-write-region (3) # Encoding local file ‘/home/smithty/.cache/emacs/tramp.JD7UfG’ using ‘base64-encode-region’...done
16:54:02.196668 tramp-sh-handle-write-region (3) # Decoding remote file ‘/ssh:gpsc:/tmp/ob-input-4gzteN’ using ‘base64 -d -i >%s’...
16:54:02.196898 tramp-send-command (6) # base64 -d -i >/tmp/ob-input-4gzteN <<'e2712b2f73d6ede4a2606fb8401b3d91' 2>/dev/null; echo tramp_exit_status $?
ZWNobyBoZWxsbyEKbHMK
e2712b2f73d6ede4a2606fb8401b3d91
16:54:02.315325 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///b5b0382db50d125fa6e4b9883103175a#$
16:54:02.315612 tramp-error (1) # File error: Couldn’t write region to ‘/ssh:gpsc:/tmp/ob-input-4gzteN’, decode using ‘base64 -d -i >%s’ failed
16:54:02.315875 tramp-sh-handle-write-region (3) # Decoding remote file ‘/ssh:gpsc:/tmp/ob-input-4gzteN’ using ‘base64 -d -i >%s’...failed

Full log:

;; Emacs: 30.0.50 Tramp: 2.7.0-pre -*- mode: outline; coding: utf-8; -*-
16:54:00.702851 tramp-send-command (6) # echo are you awake
16:54:00.730322 tramp-wait-for-regexp (6) # 
are you awake
///b5b0382db50d125fa6e4b9883103175a#$
16:54:00.730398 tramp-send-command (6) # (if test -h "/tmp/ob-input-4gzteN"; then echo t; else echo nil; fi) && \readlink --canonicalize-missing /tmp/ob-input-4gzteN 2>/dev/null; echo tramp_exit_status $?
16:54:00.760359 tramp-wait-for-regexp (6) # 
nil
/tmp/ob-input-4gzteN
tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:00.760955 tramp-send-command (6) # test -e /tmp/ob-input-4gzteN 2>/dev/null; echo tramp_exit_status $?
16:54:00.885120 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///b5b0382db50d125fa6e4b9883103175a#$
16:54:00.885924 tramp-do-file-attributes-with-stat (5) # file attributes with stat: /tmp/.#ob-input-4gzteN
16:54:00.886054 tramp-send-command (6) # tramp_stat_file_attributes /tmp/.\#ob-input-4gzteN 2>/dev/null; echo tramp_exit_status $?
16:54:00.918034 tramp-wait-for-regexp (6) # 
nil
tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:00.919217 tramp-do-file-attributes-with-stat (5) # file attributes with stat: /tmp/ob-input-4gzteN
16:54:00.919315 tramp-send-command (6) # tramp_stat_file_attributes /tmp/ob-input-4gzteN 2>/dev/null; echo tramp_exit_status $?
16:54:00.951201 tramp-wait-for-regexp (6) # 
nil
tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:00.951719 tramp-send-command (6) # getfacl -ac /tmp/ob-input-4gzteN 2>/dev/null; echo tramp_exit_status $?
16:54:00.979370 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.010045 tramp-send-command (6) # cat </dev/null >/tmp/ob-input-4gzteN
16:54:01.129386 tramp-wait-for-regexp (6) # 
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.129992 tramp-do-file-attributes-with-stat (5) # file attributes with stat: /tmp
16:54:01.130103 tramp-send-command (6) # tramp_stat_file_attributes /tmp 2>/dev/null; echo tramp_exit_status $?
16:54:01.162860 tramp-wait-for-regexp (6) # 
(("‘/tmp’") 21 ("root" . 0) ("root" . 0) 1700776090 1700776441 1700776441 6660 "drwxrwxrwt" t 2 -1)
tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.163404 tramp-send-command (6) # chown 68615:66101 /tmp/ob-input-4gzteN
16:54:01.212055 tramp-wait-for-regexp (6) # 
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.212925 tramp-send-command (6) # test -d /tmp/.\#ob-input-4gzteN 2>/dev/null; echo tramp_exit_status $?
16:54:01.240604 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.240959 tramp-send-command (6) # rm -f /tmp/.\#ob-input-4gzteN 2>/dev/null; echo tramp_exit_status $?
16:54:01.372599 tramp-wait-for-regexp (6) # 
tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.374023 tramp-send-command (6) # (if test -h "/tmp/ob-error-HcNyB8"; then echo t; else echo nil; fi) && \readlink --canonicalize-missing /tmp/ob-error-HcNyB8 2>/dev/null; echo tramp_exit_status $?
16:54:01.403494 tramp-wait-for-regexp (6) # 
nil
/tmp/ob-error-HcNyB8
tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.404093 tramp-send-command (6) # test -e /tmp/ob-error-HcNyB8 2>/dev/null; echo tramp_exit_status $?
16:54:01.431348 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.431986 tramp-do-file-attributes-with-stat (5) # file attributes with stat: /tmp/.#ob-error-HcNyB8
16:54:01.432094 tramp-send-command (6) # tramp_stat_file_attributes /tmp/.\#ob-error-HcNyB8 2>/dev/null; echo tramp_exit_status $?
16:54:01.465948 tramp-wait-for-regexp (6) # 
nil
tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.467104 tramp-do-file-attributes-with-stat (5) # file attributes with stat: /tmp/ob-error-HcNyB8
16:54:01.467202 tramp-send-command (6) # tramp_stat_file_attributes /tmp/ob-error-HcNyB8 2>/dev/null; echo tramp_exit_status $?
16:54:01.501526 tramp-wait-for-regexp (6) # 
nil
tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.502041 tramp-send-command (6) # getfacl -ac /tmp/ob-error-HcNyB8 2>/dev/null; echo tramp_exit_status $?
16:54:01.615488 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.650533 tramp-send-command (6) # cat </dev/null >/tmp/ob-error-HcNyB8
16:54:01.683121 tramp-wait-for-regexp (6) # 
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.683943 tramp-send-command (6) # chown 68615:66101 /tmp/ob-error-HcNyB8
16:54:01.717223 tramp-wait-for-regexp (6) # 
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.718088 tramp-send-command (6) # test -d /tmp/.\#ob-error-HcNyB8 2>/dev/null; echo tramp_exit_status $?
16:54:01.852729 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.853251 tramp-send-command (6) # rm -f /tmp/.\#ob-error-HcNyB8 2>/dev/null; echo tramp_exit_status $?
16:54:01.883697 tramp-wait-for-regexp (6) # 
tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.884361 tramp-send-command (6) # test -x /home/tys000/bash 2>/dev/null; echo tramp_exit_status $?
16:54:01.916872 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.917756 tramp-send-command (6) # (if test -h "/tmp/ob-input-4gzteN"; then echo t; else echo nil; fi) && \readlink --canonicalize-missing /tmp/ob-input-4gzteN 2>/dev/null; echo tramp_exit_status $?
16:54:01.949701 tramp-wait-for-regexp (6) # 
nil
/tmp/ob-input-4gzteN
tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:01.950573 tramp-do-file-attributes-with-stat (5) # file attributes with stat: /tmp/.#ob-input-4gzteN
16:54:01.950686 tramp-send-command (6) # tramp_stat_file_attributes /tmp/.\#ob-input-4gzteN 2>/dev/null; echo tramp_exit_status $?
16:54:02.084126 tramp-wait-for-regexp (6) # 
nil
tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:02.085383 tramp-do-file-attributes-with-stat (5) # file attributes with stat: /tmp/ob-input-4gzteN
16:54:02.085491 tramp-send-command (6) # tramp_stat_file_attributes /tmp/ob-input-4gzteN 2>/dev/null; echo tramp_exit_status $?
16:54:02.122525 tramp-wait-for-regexp (6) # 
(("‘/tmp/ob-input-4gzteN’") 1 ("tys000" . 68615) ("aafc_orph" . 66101) 1700776441 1700776441 1700776441 0 "-rw-r--r--" t 583 -1)
tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:02.123099 tramp-send-command (6) # getfacl -ac /tmp/ob-input-4gzteN 2>/dev/null; echo tramp_exit_status $?
16:54:02.157733 tramp-wait-for-regexp (6) # 
user::rw-
group::r--
other::r--

tramp_exit_status 0
///b5b0382db50d125fa6e4b9883103175a#$
16:54:02.196152 tramp-sh-handle-write-region (3) # Encoding local file ‘/home/smithty/.cache/emacs/tramp.JD7UfG’ using ‘base64-encode-region’...
16:54:02.196517 tramp-sh-handle-write-region (3) # Encoding local file ‘/home/smithty/.cache/emacs/tramp.JD7UfG’ using ‘base64-encode-region’...done
16:54:02.196668 tramp-sh-handle-write-region (3) # Decoding remote file ‘/ssh:gpsc:/tmp/ob-input-4gzteN’ using ‘base64 -d -i >%s’...
16:54:02.196898 tramp-send-command (6) # base64 -d -i >/tmp/ob-input-4gzteN <<'e2712b2f73d6ede4a2606fb8401b3d91' 2>/dev/null; echo tramp_exit_status $?
ZWNobyBoZWxsbyEKbHMK
e2712b2f73d6ede4a2606fb8401b3d91
16:54:02.315325 tramp-wait-for-regexp (6) # 
tramp_exit_status 1
///b5b0382db50d125fa6e4b9883103175a#$
16:54:02.315612 tramp-error (1) # File error: Couldn’t write region to ‘/ssh:gpsc:/tmp/ob-input-4gzteN’, decode using ‘base64 -d -i >%s’ failed
16:54:02.315875 tramp-sh-handle-write-region (3) # Decoding remote file ‘/ssh:gpsc:/tmp/ob-input-4gzteN’ using ‘base64 -d -i >%s’...failed

Answer 1

I'm not sure if this is the correct solution, but it works for me:

org-babel provides the option Org Babel Remote Temporary Directory. Customizing this to ~/ instructs babel to use my home directory for temp files, which I always have proper access and permissions for. I'm not sure if permissions are in fact the problem, or even part of the problem, but for now I have a solution. More details on the tramp-devel mailing list.