Enable large arrays: Build octave such that it can use arrays larger than 2Gb.: Difference between revisions

From Octave
Jump to navigation Jump to search
(First version, copy paste)
 
(Add a note to see also section.)
 
(17 intermediate revisions by 9 users not shown)
Line 1: Line 1:
By default, the size of a single Octave array cannot exceed 2 GB of memory. Trying to create one will produce the following error:
{{Note|The following only applies to systems that have 64-bit pointers (64bit architecture).}}


<pre>
Starting with Octave 4.4.0, 64-bit indexing is the default for targets with 64-bit pointers. You can override that default by specifying <code>--disable-64</code> when configuring Octave.
octave:1> a = zeros(1024,1024,512);
error: memory exhausted or requested size too large for range of Octave's index type -- trying to return to prompt
</pre>
 
You will obtain this error even if your system has enough RAM to create this array (4 GB in the above case).


To use arrays larger than 2 GB, Octave has to be configured with the option --enable-64. This option is experimental and you are (as always) encouraged to submit bug reports if you find a problem. With this option, Octave will use internally 64-bit integers for array dimensions and indexing. However, all numerical libraries used by Octave will need to use also 64-bit integers for array dimensions and indexing, and in most cases they need to be compiled from source.
However, if the configure script determines that the BLAS library uses 32-bit integers, then operations using the following libraries are limited to arrays with dimensions that are smaller than 2^31 elements:
* BLAS
* LAPACK
* QRUPDATE
* SuiteSparse
* ARPACK


The following instructions are tested with Octave 3.2.0 and gcc/gfortran 4.2.4 on a 64-bit CPU. They might also be useful on a 32-bit CPU as long as you have more than 2 GB of memory.
Additionally, the following libraries use "int" internally, so maximum problem sizes are always limited:
* glpk
* Qhull


Useful information and projects are listed below in the [[#See also|See also]] section.


== BLAS ==
To determine the integer size of the BLAS library used by Octave, the following code can be executed:


Various implementations of the BLAS library exist. Once you compiled this library make sure that you use this library for compiling LAPACK, <nowiki>SuiteSparse</nowiki>, ARPACK and Octave. In the following we assume that you installed the BLAS library as $BLAS/lib/libblas.a.
<syntaxhighlight lang="Octave">
clear all;
N = 2^31;
## The following line requires about 8 GB of RAM!
a = b = ones (N, 1, "single");
c = a' * b
</syntaxhighlight>


=== Reference BLAS ===
If the BLAS library uses '''32-bit integers''', an error will be thrown:


To compile this library use the gfortran compiler options -fdefault-integer-8 and -fPIC (for 64-bit CPUs). Either add those options to the variable OPTS in make.inc, or use the following make command:
error: integer dimension or index out of range for Fortran INTEGER type


<pre>
Otherwise, if the BLAS library uses '''64-bit integers''', the result is:
make FORTRAN=gfortran OPTS="-fPIC -O3 -fdefault-integer-8" BLASLIB=libblas.a
</pre>


=== ATLAS ===
c = 2^31 = 2147483648


<i>Not yet tested.</i>
Note that the test case above usually requires twice the memory, if <code>a</code> and <code>b</code> are not assigned by <code>a = b = ...</code>.
Note further, that the data type "single" has a precision of about 23 binary bits.
In this particular example no rounding errors occur.


=== GotoBLAS ===
===Versions prior to Octave 4.4===
 
On previous versions of Octave, the default is that the size of a single Octave array cannot have more than approximately 2^31 elements, even on systems that use 64-bit pointers.  This is because array indices were limited to 32-bit signed integers by default.  Trying to create one will produce the following error:
In Makefile.rule, set your Fortran compiler (F_COMPILER) to GFORTRAN, uncomment the line with INTERFACE64 = 1 and BINARY64 = 1 (if you have a 64-bit CPU) or use the command:


<pre>
<pre>
make F_COMPILER=GFORTRAN BINARY64=1 INTERFACE64=1
>> a = zeros (1024*1024*1024*3, 1, 'int8');
error: out of memory or dimension too large for Octave's index type
</pre>
</pre>


Note that <nowiki>GotoBLAS</nowiki> is available free of charge for research but it is not free software.
You will obtain this error even if your system has enough RAM to create this array (3 GB in the above case).


To use arrays with more than (approximately) <math>2^{31}</math> elements, Octave has to be configured with the option <code>--enable-64</code>. This option is experimental and you are (as always) encouraged to submit bug reports if you find a problem.
With this option, Octave will use internally 64-bit integers for array dimensions and indexing. However, '''all numerical libraries''' used by Octave will need to use also 64-bit integers for array dimensions and indexing, and in most cases they need to be compiled from source.


== LAPACK ==
===See also===


Copy the file make.inc.example and name it make.inc. The options -fdefault-integer-8 and -fPIC (on 64-bit CPU) have to be added to the variable OPTS and NOOPT. Alternatively, the following command can be used (with some additional optimizations):
* [https://octave.org/doc/interpreter/Compiling-Octave-with-64_002dbit-Indexing.html GNU Octave manual] -- Details on how to compile some of Octave's library dependencies for 64-bit indices.
* [[MXE]] (M Cross Environment) which takes care to compile Octave's library dependencies for 64-bit indices.


<pre>
Two more lightweight solutions compared to [[MXE]] to compile Octave's library dependencies for 64-bit indices.
cp make.inc.example  make.inc
* https://gitlab.com/mtmiller/octave-blas64-builder
make FORTRAN=gfortran LOADER=gfortran OPTS="-funroll-all-loops -O3 -fPIC -fdefault-integer-8" NOOPT="-fPIC -fdefault-integer-8" PLAT=  \
* https://github.com/octave-de/GNU-Octave-enable-64
  LAPACKLIB=liblapack.a  BLASLIB=$BLAS/lib/libblas.a TIMER=INT_ETIME  lapacklib
</pre>


Once you compiled this library make sure that you use this library for compiling Suite Sparse and Octave.
[[Category:Building]]
In the following we assume that you installed the LAPACK library as $LAPACK/lib/liblapack.a.
 
 
== ARPACK ==
 
With gfortran 4.2.4 I had to apply the patch from: https://cvs.fedoraproject.org/viewvc/rpms/arpack/F-9/arpack-etime.patch?revision=1.1&view=markup&pathrev=arpack-2_1-10_fc9
 
In ARmake.inc set the following variables:
 
<pre>
home=path to directory ARPACK
FC=gfortran
FFLAGS=-fPIC -fdefault-integer-8
MAKE=/usr/bin/make
ARPACKLIB=$(home)/libarpack.a
DIRS=$(UTILdir) $(SRCdir)
</pre>
 
Alternatively, they can be defined as arguments of make:
 
<pre>
make FC=gfortran FFLAGS="-fPIC -fdefault-integer-8" MAKE=/usr/bin/make ARPACKLIB=$PWD/libarpack.a home=$PWD DIRS="$PWD/UTIL $PWD/SRC" lib
</pre>
 
Copy the library libarpack.a to $ARPACK/lib/libarpack.a.
 
== SuiteSparse ==
 
In UFconfig/UFconfig.mk use the following options for CFLAGS and F77FLAGS:
 
<pre>
CFLAGS=-fPIC -O -DLP64 -D'LONGBLAS=long int' -D'LONG=long int'
F77FLAGS=-fPIC -O -fdefault-integer-8
BLAS=-L$BLAS/lib -lblas -lgfortran -lpthread"
LAPACK=-L$LAPACK/lib -llapack"
</pre>
 
The library -lpthread for BLAS is only needed is you used the multithreaded <nowiki>GotoBLAS</nowiki> library.
Optionally, Suite Sparse needs metis. In metis-4.0/Makefile.in the following options have to be defined:
 
<pre>
OPTFLAGS = -O2 -fPIC
</pre>
 
In the following we assume that you installed all <nowiki>SuiteSparse</nowiki> libraries (and metis) in  $SUITESPARSE/lib and all header files in $SUITESPARSE/include/suitesparse. Since <nowiki>SuiteSparse</nowiki> makefile does not have an install target, you might use:
 
<pre>
cp {AMD,BTF,CAMD,CCOLAMD,CHOLMOD,COLAMD,CXSparse,UMFPACK}/Lib/lib*a metis-4.0/libmetis.a $SUITESPARSE/lib/
cp {AMD,BTF,CAMD,CCOLAMD,CHOLMOD,COLAMD,CXSparse,UMFPACK}/Include/*h UFconfig/UFconfig.h  $SUITESPARSE/include/suitesparse
</pre>
 
 
== QHull ==
 
<i>Suggestions on how to compile qhull will be most welcome.</i>
 
== Octave ==
 
Octave's 64-bit index support is activated with the configure option --enable-64.
 
<pre>
./configure \
  CPPFLAGS="-I$SUITESPARSE/include -I$QHULL/include " \
  LIBS="-L$SUITESPARSE/lib -lmetis -L$QHULL/lib  -L$ARPACK -larpack -L$LAPACK/lib -L$BLAS/lib -lblas -lgfortran -lm -lpthread" \
  FFLAGS=-fdefault-integer-8 \
  F77=gfortran --enable-64
</pre>

Latest revision as of 10:29, 9 February 2020

Info icon.svg
The following only applies to systems that have 64-bit pointers (64bit architecture).

Starting with Octave 4.4.0, 64-bit indexing is the default for targets with 64-bit pointers. You can override that default by specifying --disable-64 when configuring Octave.

However, if the configure script determines that the BLAS library uses 32-bit integers, then operations using the following libraries are limited to arrays with dimensions that are smaller than 2^31 elements:

  • BLAS
  • LAPACK
  • QRUPDATE
  • SuiteSparse
  • ARPACK

Additionally, the following libraries use "int" internally, so maximum problem sizes are always limited:

  • glpk
  • Qhull

Useful information and projects are listed below in the See also section.

To determine the integer size of the BLAS library used by Octave, the following code can be executed:

clear all;
N = 2^31;
## The following line requires about 8 GB of RAM!
a = b = ones (N, 1, "single");
c = a' * b

If the BLAS library uses 32-bit integers, an error will be thrown:

error: integer dimension or index out of range for Fortran INTEGER type

Otherwise, if the BLAS library uses 64-bit integers, the result is:

c = 2^31 = 2147483648

Note that the test case above usually requires twice the memory, if a and b are not assigned by a = b = .... Note further, that the data type "single" has a precision of about 23 binary bits. In this particular example no rounding errors occur.

Versions prior to Octave 4.4[edit]

On previous versions of Octave, the default is that the size of a single Octave array cannot have more than approximately 2^31 elements, even on systems that use 64-bit pointers. This is because array indices were limited to 32-bit signed integers by default. Trying to create one will produce the following error:

>> a = zeros (1024*1024*1024*3, 1, 'int8');
error: out of memory or dimension too large for Octave's index type

You will obtain this error even if your system has enough RAM to create this array (3 GB in the above case).

To use arrays with more than (approximately) elements, Octave has to be configured with the option --enable-64. This option is experimental and you are (as always) encouraged to submit bug reports if you find a problem. With this option, Octave will use internally 64-bit integers for array dimensions and indexing. However, all numerical libraries used by Octave will need to use also 64-bit integers for array dimensions and indexing, and in most cases they need to be compiled from source.

See also[edit]

  • GNU Octave manual -- Details on how to compile some of Octave's library dependencies for 64-bit indices.
  • MXE (M Cross Environment) which takes care to compile Octave's library dependencies for 64-bit indices.

Two more lightweight solutions compared to MXE to compile Octave's library dependencies for 64-bit indices.