Sorry, you need to enable JavaScript to visit this website.


fMBT generates and executes tests automatically. It very quickly finds and tests paths that would never be tested by human test designers. This increases test coverage and cuts down test maintenance efforts, when compared to traditional test automation.

Debugging Remote Scripts

BY Antti Kervinen ON Oct 03, 2019

In this blog post I'll show by example how you can

1. set a "breakpoint" to your shell script or Python program, and then

2. get interactive access to your program - even if it runs on a remote host or in a container.

The idea is to make the script call home where you want it to stop for debugging.

Furthermore, you can make your Python program call home only in case of a crash (an unhandled exception). This enables remote post-mortem debugging.


First, start waiting for a home call: launch fmbt-debug at the HOME host. This works for debugging both Python 2 and 3. Launch fmbt-debug with -p PORT_NUMBER to listen to.

HOME$ fmbt-debug -p 33720

Example 1: A Python program that always calls home on line 2. Replace HOME with a hostname, or localhost when running the Python program and fmbt-debug on the same host.

import fmbt
fmbt.debug("HOME:33720") # this is a "breakpoint" that calls to fmbt-debug at HOME
my_var = 3
for i in range(1, my_var / 2):

In case you do not want to include full utils/ (for Python 2) or utils3/ (for Python 3) file to your project, you can copy only the debug() function from the library. It's self-contained: it has no external dependencies even inside and it includes the imports that it needs.

Example 2: A Python program that calls home in case of an unhandled exception (on line 4):

import fmbt
fmbt.debug("HOME:33720", post_mortem=True) # call HOME in case of an unhandled exception, not now.
my_var = 3
for i in range(1, my_var / 2):

When the program calls home, you will get Python debugger (pdb) prompt through fmbt-debug. There you can print local variables (p VARNAME), walk up (u) and down (d) in the stack, list code where the execution is going (l), execute Python code, and finally either quit (q) or continue (c) the execution, for instance. See Python Debugger Commands for more information.

Shell script

First, start waiting for a home call with netcat:

HOME$ nc -k -l -p 33720

Example 3: A shell script that always calls home at the breakpoint called before-loop. Replace HOME on the BP_HOME= line with a hostname, or localhost when running the script and the netcat on the same host.

BP_CALLHOME='BP_FIFO=/tmp/$BP.$BP_HOME.$BP_PORT; (rm -f $BP_FIFO; mkfifo $BP_FIFO) && (echo "\"c\" continues"; echo -n "($BP) "; tail -f $BP_FIFO) | nc $BP_HOME $BP_PORT | while read cmd; do if test "$cmd" = "c" ; then echo -n "" >$BP_FIFO; sleep 0.1; fuser -k $BP_FIFO >/dev/null 2>&1; break; else eval $cmd >$BP_FIFO 2>&1; echo -n "($BP) "  >$BP_FIFO; fi; done'


BP=before-loop eval $BP_CALLHOME

for i in $(seq 1 $my_var); do
    echo loop...
When you run the shell script and it enters a "breakpoint" (evaluates BP_CALLHOME), you will get a breakpoint prompt to the netcat:
HOME$ nc -k -l -p 33720
"c" continues
(before-loop) echo $my_var
(before-loop) c
The idea for a "shell breakpoint" is basically an eval version of the elegant unix trick on interactive shell prompt: mkfifo /tmp/f; cat /tmp/f | sh -i 2>&1 | nc HOME PORT > /tmp/f Using eval instead of new shell process gives access to local variables.
Update 2019-10-31:
Examples works with netcat (nc) flavors: OpenBSD, Ncat, Busybox.
However, the "classic" version of nc (netcat-traditional package in Debian/Ubuntu) does not have an option for "keep listening" (-k) after disconnection. In case of using that, instead of nc -k -l -p 33720 use
socat STDIN TCP-LISTEN:33720,reuseaddr,fork
Thanks to Jarkko Sakkinen for mentioning potential issues with netcat flavors and preferring socat!