Since compilation of all the source from scratch can take long it is good to have a source folder where most of the source has been compiled. To do this, you can create a parallel build:
mkdir dbg-octave cd dbg-octave /path/to/octave/source/configure FFLAGS=-g CFLAGS=-g CXXFLAGS=-g --enable-bounds-check --prefix=/opt/dbg-octave make # or make -jN where N is the number of CPU cores
This will create a new build of Octave in a different directory without optimisations (no -O2 gcc parameter) and with debug symbols compiled in. This build is useful for debugging Octave itself.
 Debugging oct-files
To debug oct-files, avoid making any optimization during compilation. Use
export CXXFLAGS="-ggdb -Wall -O0" for C++ code or
export CFLAGS="-ggdb -Wall -O0" for C code to suppress optimization. Compile the oct-file with the debug flag
-g which enables debug symbols
mkoctfile -g file.cpp
start now the GNU debugger with octave. On most Unixy systems, you can start gdb from within an Octave session by evaluating a command like
octave> system (sprintf ("gnome-terminal --command 'gdb -p %d'", getpid ()), "async");
at the Octave prompt. This command will open a terminal window running gdb attached to the Octave process. At this point, Octave will be stopped. To tell Octave to continue, type
at the gdb prompt. The symbols from your oct-file will only be available to gdb once the oct-file is loaded in Octave. To do that without executing any functions from the oct-file, you can ask Octave to display the help for the oct-file:
octave> help file
Now halt execution of Octave by typing ctrl+c, you'll see again the gdb prompt. Set now a breakpoint in the line of interest
(gdb) b file.cpp:40
by typing c the execution of octave will continue and you can run your oct-file directly or via an m-script.
octave> x = file(y)
the debugger will stop on the above defined line and you can start debugging according to the manual of GNU debugger.
 Tools for debugging
 Producing a stack trace
Sometimes Octave will crash, meaning, it terminates abruptly and returns control to the operating system shell. In these cases, it is very helpful to produce a stack trace to diagnose the problem. To do so, you need to (re)compile Octave with debugging symbols and run the debugger, as explained above. Then execute whatever commands are necessary to cause Octave to crash. At that point, you will be back in a gdb session. Type
where at the gdb prompt to obtain a stack trace.