NDpar package: Difference between revisions

From Octave
Jump to navigation Jump to search
(add link to tickets)
(Back to Category:Packages.)
Tag: Undo
 
(6 intermediate revisions by 2 users not shown)
Line 2: Line 2:
ndpar_[cell,array]fun are the par[cell,array]fun from package parallel 2.2.0, extended in order to handle N-dimensionnal arrays as input and output.  
ndpar_[cell,array]fun are the par[cell,array]fun from package parallel 2.2.0, extended in order to handle N-dimensionnal arrays as input and output.  
Hope it will eventually make it to the parallel package, once its stability is proved.
Hope it will eventually make it to the parallel package, once its stability is proved.
See also package parallel, maintained by Olaf Till
See also the [[Parallel package]] maintained by Olaf Till
 
There is an overhead for parallelization. Hence this package is not suited for very fast functions.


== Installation ==
== Installation ==
Line 41: Line 43:
* {{codeline|"ChunksPerProc", 2}} It means that each process should make 2 chunks (2 calls to {{codeline|f}} with {{codeline|"Vectorized", true}}). Increase this number to minimize memory usage for instance. Increasing this number is also useful if function executions can have very different durations. If a process is finished, it can take over jobs from another process that is still busy.
* {{codeline|"ChunksPerProc", 2}} It means that each process should make 2 chunks (2 calls to {{codeline|f}} with {{codeline|"Vectorized", true}}). Increase this number to minimize memory usage for instance. Increasing this number is also useful if function executions can have very different durations. If a process is finished, it can take over jobs from another process that is still busy.


=== Output in cell arrays ===
The following sample code was an answer to [http://stackoverflow.com/questions/27422219/for-every-row-reshape-and-calculate-eigenvectors-in-a-vectorized-way this question]. The goal was to diagonalize matrices. For this example, 2x2 matrices contained as rows of a 2d array (each row of the array being a flattened 2x2 matrix). Note that in this case the function is too fast, so the performance penalty is too high to gain anything.
{{code|diagonalize NxN matrices contained in an array|
<pre>
A = [0.6060168 0.8340029 0.0064574 0.7133187;
0.6325375 0.0919912 0.5692567 0.7432627;
0.8292699 0.5136958 0.4171895 0.2530783;
0.7966113 0.1975865 0.6687064 0.3226548;
0.0163615 0.2123476 0.9868179 0.1478827];
N = 2;
[eigenvectors, eigenvalues] = ndpar_arrayfun(nproc,
                                @(row) eig(reshape(row, N, N)),
                                A, "IdxDimensions", 1, "UniformOutput", false)
</pre>
}}
With {{codeline|"UniformOutput", false}}, the outputs are contained in cell arrays (one cell per slice). In the sample above, both {{codeline|eigenvectors}} and {{codeline|eigenvalues}} are {{codeline|1x5}} cell arrays.


== Issues ==
== Issues ==


Please report any issue using [https://sourceforge.net/p/ndpar/tickets/ tickets].
Please report any issue using [https://sourceforge.net/p/ndpar/tickets/ tickets].
[[Category:Packages]]

Latest revision as of 05:35, 29 September 2020

Introduction[edit]

ndpar_[cell,array]fun are the par[cell,array]fun from package parallel 2.2.0, extended in order to handle N-dimensionnal arrays as input and output. Hope it will eventually make it to the parallel package, once its stability is proved. See also the Parallel package maintained by Olaf Till

There is an overhead for parallelization. Hence this package is not suited for very fast functions.

Installation[edit]

The ndpar package can be downloaded from SourceForge

To install, issue pkg install <path to the downloaded file>


Usage[edit]

To load the package, pkg load ndpar

setting CatDimensions and IdxDimensions - vectorized[edit]

Suppose we have a function that works along different dimensions of its input arguments, and returns arrays of different shapes like

function [x, y] = f(u, v)
  x = u + v.';
  y = x.';
endfunction

This function is used only to illustrate the syntax, not the usefulness of the extension. Admittedly such a simple case could be handled easily either by changing the function or making a small wrapper to it. But it becomes really interesting when we are dealing with arrays of many and different dimensions (not shown here yet).

Applying this function to arrays u and v can be straightforwardly parallelized :

Code: setting CatDimensions and IdxDimensions - vectorized
u = [1:10; 2:11];
v = u.';
[x, y] = ndpar_arrayfun(2, @f, u, v, "Vectorized", true, "ChunksPerProc", 2, "VerboseLevel", 1, "CatDimensions", [2 1],"IdxDimensions", [2 1] );

Here is the meaning of the options

  • "IdxDimensions", [2 1] The parallelization (or slicing, or indexing) should be done along the 2nd dimension of u and 1st dimension of v. A value of 0 means no indexing (no slicing), so the argument would be passed "as is".
  • "CatDimensions", [2 1] The outputs from each slice should be concatenated along the 2nd dimension of the first output and 1st dimension of the second output
  • "Vectorized", true Use only if the function is vectorized along the "indexing" dimensions.
  • "ChunksPerProc", 2 It means that each process should make 2 chunks (2 calls to f with "Vectorized", true). Increase this number to minimize memory usage for instance. Increasing this number is also useful if function executions can have very different durations. If a process is finished, it can take over jobs from another process that is still busy.

Output in cell arrays[edit]

The following sample code was an answer to this question. The goal was to diagonalize matrices. For this example, 2x2 matrices contained as rows of a 2d array (each row of the array being a flattened 2x2 matrix). Note that in this case the function is too fast, so the performance penalty is too high to gain anything.

Code: diagonalize NxN matrices contained in an array
A = [0.6060168 0.8340029 0.0064574 0.7133187;
0.6325375 0.0919912 0.5692567 0.7432627;
0.8292699 0.5136958 0.4171895 0.2530783;
0.7966113 0.1975865 0.6687064 0.3226548;
0.0163615 0.2123476 0.9868179 0.1478827];

N = 2;
[eigenvectors, eigenvalues] = ndpar_arrayfun(nproc, 
                                @(row) eig(reshape(row, N, N)), 
                                A, "IdxDimensions", 1, "UniformOutput", false)

With "UniformOutput", false, the outputs are contained in cell arrays (one cell per slice). In the sample above, both eigenvectors and eigenvalues are 1x5 cell arrays.

Issues[edit]

Please report any issue using tickets.