Many Octave functions are wrappers to optimized numerical libraries, notably BLAS and ATLAS. It is possible to achieve impressive performance gains by simply using a library tuned to your platform. One example is using OpenBLAS to replace the default BLAS implementation ([http://www.stat.cmu.edu/~nmv/2013/07/09/for-faster-r-use-openblas-instead-better-than-atlas-trivial-to-switch-to-on-ubuntu/ further details]).
On Linux, this just takes a few commands
sudo apt-get install libopenblas-base libatlas3gf-base
sudo update-alternatives --config libblas.so.3
select the openblas option.