Octave and separate toolchain

From Octave
Revision as of 15:22, 14 November 2016 by Siko1056 (talk | contribs) (Added category.)
Jump to navigation Jump to search

Building of octave together with a separate toolchain (custom built gcc/binutils/...).

This is required mainly when the base linux distribution ships too old a version of gcc. Unexperienced users should upgrade their distribution, though.

The main build directory will be called /usr/local/build. It was first created by root, and then changed to be owned by the user who will perform compilations. You might want change this to another directory in your $HOME for instance. 

export buildroot=/usr/local/build

We will need a directory to hold downloaded files 

export downloads=$buildroot/downloads

mkdir -p $buildroot/downloads


gcc toolchain

This part is largely inspired by this answer 

cd $buildroot
# work in a subshell, to isolate mistakes
bash
# check that PATH and LD_RUN_PATH are normal
echo $PATH
echo $LD_RUN_PATH

cd $downloads
wget ftp://ftp.gnu.org/gnu/binutils/binutils-2.27.tar.bz2
wget ftp://ftp.gnu.org/gnu/gcc/gcc-6.2.0/gcc-6.2.0.tar.bz2
wget ftp://ftp.gnu.org/gnu/mpfr/mpfr-3.1.5.tar.xz
wget ftp://ftp.gnu.org/gnu/mpc/mpc-1.0.3.tar.gz
wget ftp://ftp.gnu.org/gnu/gmp/gmp-6.1.1.tar.xz
wget http://isl.gforge.inria.fr/isl-0.17.tar.xz

cd $buildroot
tar --no-same-owner -xvf $downloads/binutils-2.27.tar.bz2
tar --no-same-owner -xvf $downloads/gcc-6.2.0.tar.bz2
tar --no-same-owner -xvf $downloads/mpfr-3.1.5.tar.xz
tar --no-same-owner -xvf $downloads/mpc-1.0.3.tar.gz
tar --no-same-owner -xvf $downloads/gmp-6.1.1.tar.xz
tar --no-same-owner -xvf $downloads/isl-0.17.tar.xz

cd $buildroot/gcc-6.2.0
ln -s ../mpfr-3.1.5 mpfr
ln -s ../mpc-1.0.3 mpc
ln -s ../gmp-6.1.1 gmp
ln -s ../isl-0.17/ isl
# Now symlink everything from the binutils dir which doesn't exist in the gcc dir,
# so anything which already exists will take priority but the
# binutils tools will look be visible to the build:
for file in ../binutils-2.27/* ; do ln -s "${file}" ; done
# errors ending with "File exists" are normal

Now we are ready to build the toolchain 

cd $buildroot
mkdir gcc-6.2.0_isl_build; cd $_
../gcc-6.2.0/configure --prefix=/usr/local/gcc-6.2.0_binutils-2.27_isl --disable-nls --enable-languages=c,c++,fortran,java --disable-multilib
make -j 6 bootstrap
sudo make install
# see what has been installed
ls /usr/local/gcc-6.2.0_binutils-2.27_isl/bin


OpenBLAS

We need an optimized BLAS library, such as OpenBLAS 

cd $downloads
wget -O OpenBLAS-0.2.19.tar.gz http://github.com/xianyi/OpenBLAS/archive/v0.2.19.tar.gz

cd $buildroot
tar --no-same-owner -xvf $downloads/OpenBLAS-0.2.19.tar.gz
cd OpenBLAS-0.2.19/

bash
export PATH=/usr/local/gcc-6.2.0_binutils-2.27_isl/bin:$PATH
# check
which gcc
# should be /usr/local/gcc-6.2.0_binutils-2.27_isl/bin/gcc
which gfortran
# should be /usr/local/gcc-6.2.0_binutils-2.27_isl/bin/gfortran

# do not use threads 
# (because I use pararrayfun from the parallel package)
make USE_THREAD=0 CPPFLAGS="-I/usr/local/gcc-6.2.0_binutils-2.27_isl/include" CFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" CXXFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" FFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" LDFLAGS=-Wl,-rpath,/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64

One should get 

OpenBLAS build complete. (BLAS CBLAS LAPACK LAPACKE)

 OS               ... Linux             
 Architecture     ... x86_64               
 BINARY           ... 64bit                 
 C compiler       ... GCC  (command line : gcc)
 Fortran compiler ... GFORTRAN  (command line : gfortran)
 Library Name     ... libopenblas_haswell-r0.2.19.a (Single threaded)

Checking the runpath: 

ldd libopenblas.so
       linux-vdso.so.1 (0x00007fff961fc000)
       libm.so.6 => /lib64/libm.so.6 (0x00007f3ee88cb000)
       libgfortran.so.3 => /usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/libgfortran.so.3 (0x00007f3ee85a3000)
       libc.so.6 => /lib64/libc.so.6 (0x00007f3ee81f9000)
       libquadmath.so.0 => /usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/../lib64/libquadmath.so.0 (0x00007f3ee7fb9000)
       libgcc_s.so.1 => /usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/../lib64/libgcc_s.so.1 (0x00007f3ee7da2000)
       /lib64/ld-linux-x86-64.so.2 (0x00007f3ee9ad6000)

Manual installation 

sudo cp libopenblas.a libopenblas_haswell-r0.2.19.a libopenblas_haswell-r0.2.19.so /usr/local/gcc-6.2.0_binutils-2.27_isl/lib64
cd /usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/
sudo ln -s libopenblas_haswell-r0.2.19.so libopenblas.so.0
sudo ln -s libopenblas.so.0 libopenblas.so

qrupdate

download the lastes version (1.1.2 as of this writing) to $downloads

https://sourceforge.net/projects/qrupdate/files/latest/download?source=typ_redirect

Start from a clean state 

exit
bash
export PATH=/usr/local/gcc-6.2.0_binutils-2.27_isl/bin:$PATH
which gfortran
# should be /usr/local/gcc-6.2.0_binutils-2.27_isl/bin/gfortran

cd $buildroot
tar --no-same-owner -xvf $downloads/qrupdate-1.1.2.tar.gz
cd qrupdate-1.1.2/

Save the following to $downloads/qrupdate-1.1.2_Makeconf.patch

Code: qrupdate-1.1.2_Makeconf.patch 
--- Makeconf.orig	2016-09-20 16:33:23.061756020 +0200
+++ Makeconf	2016-09-20 16:39:47.385547117 +0200
@@ -1,26 +1,26 @@
 # set this to your compiler's executable name (e.g. gfortran, g77)
 FC=gfortran
 # requested flags
-FFLAGS=-fimplicit-none -O3 -funroll-loops 
+FFLAGS=-fimplicit-none -march=native -O3 -funroll-loops -lgfortran
 # set if you need shared library
 FPICFLAGS=-fPIC
 
 # BLAS library (only required for tests)
-BLAS=-lblas
+BLAS=-lopenblas
 # LAPACK library (only required for tests)
-LAPACK=-llapack
+LAPACK=
 
 # Library version
 VERSION=1.1
 MAJOR=1
 
 # The default library dir
-LIBDIR=lib
+LIBDIR=lib64
 
 # Destination installation offset
 DESTDIR=
 
 # set default prefix to /usr/local
 ifeq ($(strip $(PREFIX)),)
-  PREFIX=/usr/local
+  PREFIX=/usr/local/gcc-6.2.0_binutils-2.27_isl
 endif

patch -p0 < $downloads/qrupdate-1.1.2.tar.gz
make solib

as root (su, not sudo, because we want PATH to contain /usr/local/gcc-6.2.0_binutils-2.27_isl/bin) 

make install-shlib

exit root session, then exit current bash, to return to a clean state

arpack-ng

https://github.com/opencollab/arpack-ng/releases 

cd $downloads
wget -O arpack-ng-3.4.0.tar.gz https://github.com/opencollab/arpack-ng/archive/3.4.0.tar.gz

Start from a clean state 

exit
bash
export PATH=/usr/local/gcc-6.2.0_binutils-2.27_isl/bin:$PATH
which gfortran
# should be /usr/local/gcc-6.2.0_binutils-2.27_isl/bin/gfortran

cd $buildroot
tar --no-same-owner -xvf $downloads/arpack-ng-3.4.0.tar.gz
cd arpack-ng-3.4.0
./bootstrap
./configure CPPFLAGS="-I/usr/local/gcc-6.2.0_binutils-2.27_isl/include" CFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" CXXFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" FFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" LDFLAGS=-Wl,-rpath,/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64 --prefix=/usr/local/gcc-6.2.0_binutils-2.27_isl

make -j 6
make check
 # TOTAL: 12
 # PASS:  12

# check libraries
ldd ./SRC/.libs/libarpack.so
       linux-vdso.so.1 (0x00007fff1f38c000)
       libopenblas.so.0 => /usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/libopenblas.so.0 (0x00007f80c79d8000)
       libgfortran.so.3 => /usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/libgfortran.so.3 (0x00007f80c76b0000)
       libm.so.6 => /lib64/libm.so.6 (0x00007f80c7368000)
       libgcc_s.so.1 => /usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/libgcc_s.so.1 (0x00007f80c7151000)
       libquadmath.so.0 => /usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/libquadmath.so.0 (0x00007f80c6f11000)
       libc.so.6 => /lib64/libc.so.6 (0x00007f80c6b67000)
       /lib64/ld-linux-x86-64.so.2 (0x00007f80c892e000)

as root (su, not sudo, because we want PATH to contain /usr/local/gcc-6.2.0_binutils-2.27_isl/bin) 

make install

exit root session, then exit current bash, to return to a clean state

SuiteSparse

http://faculty.cse.tamu.edu/davis/suitesparse.html 

cd $downloads
wget http://faculty.cse.tamu.edu/davis/SuiteSparse/SuiteSparse-4.5.3.tar.gz

Save the code below to $downloads/UMFPACK_Demo_Makefile-4.5.3.patch

Code: UMFPACK_Demo_Makefile-4.5.3.patch 
--- UMFPACK/Demo/Makefile.orig	2016-09-19 16:20:46.317044240 +0200
+++ UMFPACK/Demo/Makefile	2016-09-19 16:30:15.079302070 +0200
@@ -27,9 +27,9 @@
 C = $(CC) $(CF) $(UMFPACK_CONFIG) $(CONFIG_PARTITION) \
     -I../../include
 
-LIBS = $(LDLIBS) -L../../lib -lumfpack -lamd -lsuitesparseconfig \
+LIBS = $(LDFLAGS) $(LDLIBS) -L../../lib -lumfpack -lamd -lsuitesparseconfig \
 	$(LIB_WITH_CHOLMOD) $(LAPACK) $(BLAS)
 
 libs: metis
 	( cd ../../SuiteSparse_config ; $(MAKE) )
 	( cd ../../AMD ; $(MAKE) library )




cd $buildroot
tar --no-same-owner -xvf $downloads/SuiteSparse-4.5.3.tar.gz
mv SuiteSparse SuiteSparse-4.5.3
cd SuiteSparse-4.5.3/

patch -p0 < $downloads/UMFPACK_Demo_Makefile-4.5.3.patch
# should get: "patching file UMFPACK/Demo/Makefile"

If the following command 

find . -name Makefile -exec grep "LDLIBS +=" {} \; -print

yields LDLIBS lines not starting with override, then the patch sent upstream has not been applied, and we should do it here: 

find . -name Makefile -exec sed -i 's/LDLIBS +=/override LDLIBS +=/g' {} \;

Now we are ready to build SuiteSparse 

make -j6 LAPACK= CF="-march=native -O3 -fexceptions -fPIC -fopenmp" F77FLAGS="-march=native -O3" LDFLAGS="-L/usr/local/build/SuiteSparse-4.5.3/lib -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" LDLIBS="-lm -lrt -Wl,-rpath=/usr/local/build/SuiteSparse-4.5.3/lib:/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64"

fix RUN_PATH and install 

cp -Rdp lib lib_chrpath
patchelf --set-rpath /usr/local/gcc-6.2.0_binutils-2.27_isl/lib64 lib_chrpath/*
sudo cp lib_chrpath/* /usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/
sudo mkdir /usr/local/gcc-6.2.0_binutils-2.27_isl/include/suitesparse/
sudo cp include/* /usr/local/gcc-6.2.0_binutils-2.27_isl/include/suitesparse/


FFTW

http://www.fftw.org/ 

cd $downloads
wget http://www.fftw.org/fftw-3.3.5.tar.gz

Start from a clean state 

exit
bash
export PATH=/usr/local/gcc-6.2.0_binutils-2.27_isl/bin:$PATH
which gfortran
# should be /usr/local/gcc-6.2.0_binutils-2.27_isl/bin/gfortran

cd $buildroot
tar --no-same-owner -xvf $downloads/fftw-3.3.5.tar.gz
cd fftw-3.3.5/

Single precision

./configure --enable-shared --enable-sse2 --enable-avx --enable-openmp --enable-threads --enable-float CPPFLAGS="-I/usr/local/gcc-6.2.0_binutils-2.27_isl/include" CFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" CXXFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" FFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" LDFLAGS=-Wl,-rpath,/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64 PKG_CONFIG_PATH=/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/pkgconfig --prefix=/usr/local/gcc-6.2.0_binutils-2.27_isl
make -j6
make check

as root (su, not sudo, because we want PATH to contain /usr/local/gcc-6.2.0_binutils-2.27_isl/bin) 

make install
exit

Double precision

./configure --enable-shared --enable-sse2 --enable-avx --enable-openmp --enable-threads CPPFLAGS="-I/usr/local/gcc-6.2.0_binutils-2.27_isl/include" CFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" CXXFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" FFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" LDFLAGS=-Wl,-rpath,/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64 PKG_CONFIG_PATH=/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/pkgconfig --prefix=/usr/local/gcc-6.2.0_binutils-2.27_isl
make -j6
make check

as root (su, not sudo, because we want PATH to contain /usr/local/gcc-6.2.0_binutils-2.27_isl/bin) 

make install
exit

GraphicsMagick

download GraphicsMagick (here, GraphicsMagick-1.3.25.tar.xz) to $downloads

https://sourceforge.net/projects/graphicsmagick/files/graphicsmagick/

Start from a clean state 

exit
bash
export PATH=/usr/local/gcc-6.2.0_binutils-2.27_isl/bin:$PATH
which gfortran
# should be /usr/local/gcc-6.2.0_binutils-2.27_isl/bin/gfortran

cd $buildroot
tar --no-same-owner -xvf $downloads/GraphicsMagick-1.3.25.tar.xz
cd GraphicsMagick-1.3.25

Disable webp and jbig that fail tests with gcc-6.2.0 

./configure --without-webp --without-jbig --enable-static=no --enable-shared=yes --with-quantum-depth=16 CPPFLAGS="-I/usr/local/gcc-6.2.0_binutils-2.27_isl/include" CFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" CXXFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" FFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" LDFLAGS=-Wl,-rpath,/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64 PKG_CONFIG_PATH=/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/pkgconfig --prefix=/usr/local/gcc-6.2.0_binutils-2.27_isl

The summary should be fine. As comparison and source of relevant informations: GraphicsMagick 

make -j 6

make -j6 check
============================================================================
Testsuite summary for GraphicsMagick 1.3.25
============================================================================
# TOTAL: 1215
# PASS:  1169
# SKIP:  46
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0


as root (su, not sudo, because we want PATH to contain /usr/local/gcc-6.2.0_binutils-2.27_isl/bin) 

make install

exit root session, then exit current bash, to return to a clean state

Octave itself

cd $downloads
wget ftp://alpha.gnu.org/gnu/octave/octave-4.2.0-rc4.tar.lz

Start from a clean state 

exit
bash
export PATH=/usr/local/gcc-6.2.0_binutils-2.27_isl/bin:$PATH
which gfortran
# should be /usr/local/gcc-6.2.0_binutils-2.27_isl/bin/gfortran

cd $buildroot
tar --no-same-owner -xvf $downloads/octave-4.2.0-rc4.tar.lz
cd octave-4.2.0-rc4

mkdir build-; cd $_
../configure --with-blas="/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/libopenblas.so" CPPFLAGS="-I/usr/local/gcc-6.2.0_binutils-2.27_isl/include" CFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" CXXFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" FFLAGS="-O2 -march=native -L/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64" LDFLAGS=-Wl,-rpath,/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64 PKG_CONFIG_PATH=/usr/local/gcc-6.2.0_binutils-2.27_isl/lib64/pkgconfig JAVA_HOME=/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/ --prefix=/usr/local/gcc-6.2.0_binutils-2.27_isl

make -j 6

make check
# or, for nvidia users,
xvfb-run -s "-screen 0 640x480x24" make check LD_PRELOAD=/usr/lib64/libGL.so

To be continued...

Retrieved from "https://wiki.octave.org/wiki/index.php?title=Octave_and_separate_toolchain&oldid=9647"