Debugging Octave
Preliminaries
Since compilation of all the source from scratch can take long it is good to have a source folder where most of the source has been compiled. To do this, you can create a parallel build:
mkdir dbg-octave
cd dbg-octave
/path/to/octave/source/configure FFLAGS=-g CFLAGS=-g CXXFLAGS=-g --enable-address-sanitizer-flags --prefix=/opt/dbg-octave
make # or make -jN where N is the number of CPU cores
This will create a new build of Octave in a different directory without optimisations (no -O2 gcc parameter) and with debug symbols compiled in. This build is useful for debugging Octave itself.
Debugging Source files without GUI
Ubuntu introduced a patch to disallow ptracing of non-child processes by non-root users - ie. only a process which is a parent of another process can ptrace it for normal users - whilst root can still ptrace every process. That's why gdb won't be able to link to the octave process if you start gdb from within an Octave session using the command
octave> system (sprintf ("gnome-terminal --command 'gdb -p %d'", getpid ()), "async");
You can temporarily disable this restriction by doing:
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
and then reopen the gdb using the command mentioned above from within an Octave session or if you have admin right you can simply do:
octave> system (sprintf ("gnome-terminal --command 'sudo gdb -p %d'", getpid ()), "async");
Instead you can debug octave without gui using:
./run-octave -g --no-gui
Debugging oct-files
To debug oct-files, avoid making any optimization during compilation. Use export CXXFLAGS="-ggdb -Wall -O0" for C++ code or export CFLAGS="-ggdb -Wall -O0" for C code to suppress optimization. Compile the oct-file with the debug flag -g which enables debug symbols
mkoctfile -g file.cpp
In next step you will use GNU debugger or gdb. The symbols from your oct-file will only be available to gdb once the oct-file is loaded in Octave. To do that without executing any functions from the oct-file, you can ask Octave to display the help for the oct-file:
octave> help file
start now the GNU debugger with octave. On most Unixy systems, you can start gdb from within an Octave session by evaluating a command like
octave> system (sprintf ("gnome-terminal --command 'gdb -p %d'", getpid ()), "async");
at the Octave prompt. This command will open a terminal window running gdb attached to the Octave process. At this point, Octave will be stopped.
Now you can set a breakpoint in the line of interest of your oct-file in gdb prompt:
(gdb) b file.cpp:40
by typing c the execution of octave will continue and you can run your oct-file directly or via an m-script.
octave> x = file(y)
the debugger will stop on the above defined line and you can start debugging according to the manual of GNU debugger.
Tools for debugging
There are different tools for debugging. This article concentrates on describing how to use gdb.
If you run Octave from a build tree, execute ./run-octave -g to start a gdb session that is prepared to run Octave (with the necessary environment correctly set up).
Alternatively, you can attach gdb to a running Octave. For this, execute gdb in a shell. (On Windows, use the msys2 shell that can be started with the file cmdshell.bat in Octave's installation folder). Find the PID of the running Octave process and attach to it from the (gdb) prompt with attach *PID* (replace *PID* with the actual PID).
Yet another possibility is to start gdb using the system command at the Octave prompt. Depending on the used Linux shell, the command might look similar to (see the above comments wrt running this from the GUI in some distributions):
system (sprintf ("gnome-terminal -- gdb -p %d", getpid ()), false, "async");
On Windows, you can issue:
system (sprintf ("start gdb -p %d", getpid ()));
The last approach might not be useful for debugging some kinds of errors. The OS might kill the shell that runs gdb as soon as the spawning process (i.e. Octave) crashes.
Independent of how gdb was started and Octave was attached to it, it is now possible to issue gdb commands on the (gdb) prompt. See e.g. the gdb documentation. To return to Octave while gdb is still attached to it, execute continue (or c) at the (gdb) prompt.
Producing a stack trace
Sometimes Octave might crash, meaning, it terminates abruptly and returns control to the operating system shell. In these cases, it is very helpful to produce a stack trace to diagnose the problem. For this, it can be useful to (re)compile Octave with debugging symbols. Otherwise, the stack trace can be harder to read and optimizations might make debugging more difficult. (But it is also possible to produce a stack trace with a "standard" build.)
Attach gdb to Octave as described before and return execution to Octave. Then, execute whatever commands are necessary to cause Octave to crash. At that point, you will be back in the gdb session. Type where or bt at the gdb prompt to obtain a stack trace.
You could also get some help from your system tools. In most GNU/Linux systems whenever a crash happens in a software, a core dump will be generated. These core dumps are handled by a registered component of the system and finally might be stored somewhere in the directory tree. You should find them, view them and inspect them.
Where are core dumps stored?
It differs on each system. First you should see how core dumps are handled on your system. To do so, type this in a shell terminal:
$ cat /proc/sys/kernel/core_pattern
This may print a file name pattern along with a path where all core dumps will be saved ONLY if it does not start by a pipe or '|'. If it does, the kernel will treat the rest of the pattern as a command to run. The core dump will be written to the standard input of that program instead of to a file, and you need to consult that program's help or manual.
How to view a core dump?
To do this you should use gdb. Core dumps are saved under root user, so you may need to change owner of the core dump you are interested in if you are not logged in as root. After that type in the terminal:
gdb octave -c <Path to core dump>
Always expect some warnings from gdb in a few first times of doing this. Most likely gdb will tell you that:
1. The core dump file is not in the correct format. It is the case if the core dump handler of your system compresses core dumps before storing them, and you need to decompress the core dump first.
2. Core dump was generated by a-path-to/octave-gui. Then quit gdb and start it again by:
gdb octave-gui -c <Path to core dump>
3. Some debug info are missing. In this case gdb itself will tell you how to install them. Install them and start gdb again.
If everything worked fine, you can use where command in gdb prompt to see a full stack trace of the crash.
Helpful gdb commands
For debugging octave_value variables (e.g. my_octave_value) to find out what the variable actually is (instead of just it's base class):
(gdb) print *(my_octave_value.rep)