Debugging Remote Scripts
In this blog post I'll show by example how you can
1. set a "breakpoint" to a Python program, shell script or Makefile, and then
2. get interactive access to the program - even if it runs on a remote host or in a container.
The idea is to make the script call home where you want it to stop for debugging.
Furthermore, you can make your Python program call home only in case of a crash (an unhandled exception). This enables remote post-mortem debugging.
First, start waiting for a home call: launch fmbt-debug at the HOME host. This works for debugging both Python 2 and 3. Launch fmbt-debug with
-p PORT_NUMBER to listen to.
HOME$ fmbt-debug -p 33720
Example 1: A Python program that always calls home on line 2. Replace
HOME with a hostname, or
localhost when running the Python program and fmbt-debug on the same host.
import fmbt fmbt.debug("HOME:33720") # this is a "breakpoint" that calls to fmbt-debug at HOME my_var = 3 for i in range(1, my_var / 2): print("loop...")
In case you do not want to include full utils/fmbt.py (for Python 2) or utils3/fmbt.py (for Python 3) file to your project, you can copy only the
debug() function from the library. It's self-contained: it has no external dependencies even inside
fmbt.py and it includes the
imports that it needs.
Example 2: A Python program that calls home in case of an unhandled exception (on line 4):
import fmbt fmbt.debug("HOME:33720", post_mortem=True) # call HOME in case of an unhandled exception, not now. my_var = 3 for i in range(1, my_var / 2): print("loop...")
When the program calls home, you will get Python debugger
(pdb) prompt through fmbt-debug. There you can print local variables (
p VARNAME), walk up (
u) and down (
d) in the stack, list code where the execution is going (
l), execute Python code, and finally either quit (
q) or continue (
c) the execution, for instance. See Python Debugger Commands for more information.
First, start waiting for a home call with netcat:
HOME$ nc -k -l -p 33720
Example 3: A shell script that always calls home at the breakpoint called
before-loop. If you want to debug the script remotely, replace
localhost with the hostname/address from which you want to do the debugging.
#!/bin/sh BP_HOME=localhost BP_PORT=33720 BP_CALLHOME='BP_FIFO=/tmp/$BP.$BP_HOME.$BP_PORT; (rm -f $BP_FIFO; mkfifo $BP_FIFO) && (echo "\"c\" continues"; echo -n "($BP) "; tail -f $BP_FIFO) | nc $BP_HOME $BP_PORT | while read cmd; do if test "$cmd" = "c" ; then echo -n "" >$BP_FIFO; sleep 0.1; fuser -k $BP_FIFO >/dev/null 2>&1; break; else eval $cmd >$BP_FIFO 2>&1; echo -n "($BP) " >$BP_FIFO; fi; done' my_var=3 BP=before-loop eval $BP_CALLHOME for i in $(seq 1 $my_var); do echo loop... done
HOME$ nc -k -l -p 33720 "c" continues (before-loop) echo $my_var 3 (before-loop) c
evalversion of the elegant unix trick on interactive shell prompt:
mkfifo /tmp/f; cat /tmp/f | sh -i 2>&1 | nc HOME PORT > /tmp/fUsing eval instead of new shell process gives access to local variables.
BP_CALLHOMEvariable early in the Makefile. (Replace "
localhost" by the name of the remote host in case you want to debug the Makefile from that host.)
BP_CALLHOME := 'BP_HOME=localhost; BP_PORT=33720; BP_FIFO=/tmp/$$BP.$$BP_HOME.$$BP_PORT; (rm -f $$BP_FIFO; mkfifo $$BP_FIFO) && (echo "\"c\" continues"; echo -n "($$BP) "; tail -f $$BP_FIFO) | nc $$BP_HOME $$BP_PORT | while read cmd; do if test "$$cmd" = "c" ; then echo -n "" >$$BP_FIFO; sleep 0.1; fuser -k $$BP_FIFO >/dev/null 2>&1; break; else eval $$cmd >$$BP_FIFO 2>&1; echo -n "($$BP) " >$$BP_FIFO; fi; done'
- Add the following command wherever you want to stop the Makefile for debugging. Using a descriptive
my-breakpoint-namewill help following where you have been stopped in case you have many breakpoints. You can use Makefile variables in the breakpoint name, too.
BP=my-breakpoint-name sh -c $(BP_CALLHOME)
Just like with the shell script above, first start listening to a connection from the Makefile (
$ nc -k -l -p 33720) and then let the
However, the "classic" version of nc (netcat-traditional package in Debian/Ubuntu) does not have an option for "keep listening" (-k) after disconnection. In case of using that, instead of nc -k -l -p 33720 use
socat STDIN TCP-LISTEN:33720,reuseaddr,fork
Thanks to Jarkko Sakkinen for mentioning potential issues with netcat flavors and preferring socat!