NDpar package
Introduction
ndpar_[cell,array]fun are the par[cell,array]fun from package parallel 2.2.0, extended in order to handle N-dimensionnal arrays as input and output. Hope it will eventually make it to the parallel package, once its stability is proved. See also the Parallel package maintained by Olaf Till
There is an overhead for parallelization. Hence this package is not suited for very fast functions.
Installation
The ndpar package can be downloaded from SourceForge
To install, issue pkg install <path to the downloaded file>
Usage
To load the package, pkg load ndpar
setting CatDimensions and IdxDimensions - vectorized
Suppose we have a function that works along different dimensions of its input arguments, and returns arrays of different shapes like
function [x, y] = f(u, v)
x = u + v.';
y = x.';
endfunction
This function is used only to illustrate the syntax, not the usefulness of the extension.
Admittedly such a simple case could be handled easily either by changing the function or making a small wrapper to it. But it becomes really interesting when we are dealing with arrays of many and different dimensions (not shown here yet).
Applying this function to arrays u and v can be straightforwardly parallelized :
Code: setting CatDimensions and IdxDimensions - vectorized |
u = [1:10; 2:11]; v = u.'; [x, y] = ndpar_arrayfun(2, @f, u, v, "Vectorized", true, "ChunksPerProc", 2, "VerboseLevel", 1, "CatDimensions", [2 1],"IdxDimensions", [2 1] ); |
Here is the meaning of the options
"IdxDimensions", [2 1]
The parallelization (or slicing, or indexing) should be done along the 2nd dimension ofu
and 1st dimension ofv
. A value of 0 means no indexing (no slicing), so the argument would be passed "as is"."CatDimensions", [2 1]
The outputs from each slice should be concatenated along the 2nd dimension of the first output and 1st dimension of the second output"Vectorized", true
Use only if the function is vectorized along the "indexing" dimensions."ChunksPerProc", 2
It means that each process should make 2 chunks (2 calls tof
with"Vectorized", true
). Increase this number to minimize memory usage for instance. Increasing this number is also useful if function executions can have very different durations. If a process is finished, it can take over jobs from another process that is still busy.
Output in cell arrays
The following sample code was an answer to this question. The goal was to diagonalize matrices. For this example, 2x2 matrices contained as rows of a 2d array (each row of the array being a flattened 2x2 matrix). Note that in this case the function is too fast, so the performance penalty is too high to gain anything.
Code: diagonalize NxN matrices contained in an array |
A = [0.6060168 0.8340029 0.0064574 0.7133187; 0.6325375 0.0919912 0.5692567 0.7432627; 0.8292699 0.5136958 0.4171895 0.2530783; 0.7966113 0.1975865 0.6687064 0.3226548; 0.0163615 0.2123476 0.9868179 0.1478827]; N = 2; [eigenvectors, eigenvalues] = ndpar_arrayfun(nproc, @(row) eig(reshape(row, N, N)), A, "IdxDimensions", 1, "UniformOutput", false) |
With "UniformOutput", false
, the outputs are contained in cell arrays (one cell per slice). In the sample above, both eigenvectors
and eigenvalues
are 1x5
cell arrays.
Issues
Please report any issue using tickets.