Sorry, you need to enable JavaScript to visit this website.

fMBT

fMBT generates and executes tests automatically. It very quickly finds and tests paths that would never be tested by human test designers. This increases test coverage and cuts down test maintenance efforts, when compared to traditional test automation.

Debugging Remote Scripts

BY Antti Kervinen ON Oct 03, 2019

In this blog post I'll show by example how you can

1. set a "breakpoint" to a Python programshell script or Makefile, and then

2. get interactive access to the program - even if it runs on a remote host or in a container.

The idea is to make the script call home where you want it to stop for debugging.

Furthermore, you can make your Python program call home only in case of a crash (an unhandled exception). This enables remote post-mortem debugging.

Python

First, start waiting for a home call: launch fmbt-debug at the HOME host. This works for debugging both Python 2 and 3. Launch fmbt-debug with -p PORT_NUMBER to listen to.

HOME$ fmbt-debug -p 33720

Example 1: A Python program that always calls home on line 2. Replace HOME with a hostname, or localhost when running the Python program and fmbt-debug on the same host.

import fmbt
fmbt.debug("HOME:33720") # this is a "breakpoint" that calls to fmbt-debug at HOME
my_var = 3
for i in range(1, my_var / 2):
    print("loop...")

In case you do not want to include full utils/fmbt.py (for Python 2) or utils3/fmbt.py (for Python 3) file to your project, you can copy only the debug() function from the library. It's self-contained: it has no external dependencies even inside fmbt.py and it includes the imports that it needs.

Example 2: A Python program that calls home in case of an unhandled exception (on line 4):

import fmbt
fmbt.debug("HOME:33720", post_mortem=True) # call HOME in case of an unhandled exception, not now.
my_var = 3
for i in range(1, my_var / 2):
    print("loop...")

When the program calls home, you will get Python debugger (pdb) prompt through fmbt-debug. There you can print local variables (p VARNAME), walk up (u) and down (d) in the stack, list code where the execution is going (l), execute Python code, and finally either quit (q) or continue (c) the execution, for instance. See Python Debugger Commands for more information.

Shell script

First, start waiting for a home call with netcat:

HOME$ nc -k -l -p 33720

Example 3: A shell script that always calls home at the breakpoint called before-loop. If you want to debug the script remotely, replace localhost with the hostname/address from which you want to do the debugging.

#!/bin/sh
BP_HOME=localhost BP_PORT=33720
BP_CALLHOME='BP_FIFO=/tmp/$BP.$BP_HOME.$BP_PORT; (rm -f $BP_FIFO; mkfifo $BP_FIFO) && (echo "\"c\" continues"; echo -n "($BP) "; tail -f $BP_FIFO) | nc $BP_HOME $BP_PORT | while read cmd; do if test "$cmd" = "c" ; then echo -n "" >$BP_FIFO; sleep 0.1; fuser -k $BP_FIFO >/dev/null 2>&1; break; else eval $cmd >$BP_FIFO 2>&1; echo -n "($BP) "  >$BP_FIFO; fi; done'

my_var=3

BP=before-loop eval $BP_CALLHOME

for i in $(seq 1 $my_var); do
    echo loop...
done
When you run the shell script and it enters a "breakpoint" (evaluates BP_CALLHOME), you will get a breakpoint prompt to the netcat:
HOME$ nc -k -l -p 33720
"c" continues
(before-loop) echo $my_var
3
(before-loop) c
The idea for a "shell breakpoint" is basically an eval version of the elegant unix trick on interactive shell prompt: mkfifo /tmp/f; cat /tmp/f | sh -i 2>&1 | nc HOME PORT > /tmp/f Using eval instead of new shell process gives access to local variables.

Makefile

Shell script debugging enables adding breakpoints to Makefile rules, too. That is, you can get a shell prompt between any commands in a Makefile to see what exactly is happening. And of course do it remotely, if needed:
 
  1. Introduce BP_CALLHOME variable early in the Makefile. (Replace "localhost" by the name of the remote host in case you want to debug the Makefile from that host.)

    BP_CALLHOME := 'BP_HOME=localhost; BP_PORT=33720; BP_FIFO=/tmp/$$BP.$$BP_HOME.$$BP_PORT; (rm -f $$BP_FIFO; mkfifo $$BP_FIFO) && (echo "\"c\" continues"; echo -n "($$BP) "; tail -f $$BP_FIFO) | nc $$BP_HOME $$BP_PORT | while read cmd; do if test "$$cmd" = "c" ; then echo -n "" >$$BP_FIFO; sleep 0.1; fuser -k $$BP_FIFO >/dev/null 2>&1; break; else eval $$cmd >$$BP_FIFO 2>&1; echo -n "($$BP) "  >$$BP_FIFO; fi; done'
     
  2. Add the following command wherever you want to stop the Makefile for debugging. Using a descriptive my-breakpoint-name will help following where you have been stopped in case you have many breakpoints. You can use Makefile variables in the breakpoint name, too.

    BP=my-breakpoint-name sh -c $(BP_CALLHOME)

Just like with the shell script above, first start listening to a connection from the Makefile ($ nc -k -l -p 33720) and then let the make run.

Update 2019-10-31:
Examples works with netcat (nc) flavors: OpenBSD, Ncat, Busybox.
However, the "classic" version of nc (netcat-traditional package in Debian/Ubuntu) does not have an option for "keep listening" (-k) after disconnection. In case of using that, instead of nc -k -l -p 33720 use
 
socat STDIN TCP-LISTEN:33720,reuseaddr,fork

Thanks to Jarkko Sakkinen for mentioning potential issues with netcat flavors and preferring socat!

Update 2020-08-19:

Makefile example added.