Pythonic: Difference between revisions

From Octave
Jump to navigation Jump to search
(→‎Python Objects in Octave: add link to the bookmark)
(Updating with link to reference API)
 
(14 intermediate revisions by 5 users not shown)
Line 3: Line 3:
== Features ==
== Features ==


Features and capabilities of Octave's Python interface may include:
At a high level, the features and capabilities of Octave's Python interface allow a user to:


* Import and call Python modules and functions from the Octave interpreter
* Import and call any Python module or function from the Octave interpreter
* Automatically convert basic Octave and Python types seamlessly between the two environments
* Automatically convert basic Octave and Python types seamlessly between the two operating environments
* Be able to handle arbitrary unknown Python objects (print their repr, store in a variable, pass back in to a Python function)
* Assign any Python object to an Octave variable, view its properties, and invoke methods on it
* Store references to Python functions (and other "callables") and be able to call them as if they were function handles
* Assign any Python function or callable object to an Octave variable, and call it as if it were a function handle
* Perform element indexing on lists and other sequence objects using curly bracket indexing syntax
* Perform key indexing on dicts and other mapping objects using curly bracket indexing syntax
 
Some features that have not yet been implemented, but have been planned for, include:
 
* Perform slice indexing on lists and other sequence objects using parentheses indexing syntax
* Operate on Python objects using standard Octave arithmetic and logical operators
* Load and save Python objects to Octave data files using the standard load/save commands


== Development ==
== Development ==


Project development is ongoing among a small group of developers. Communication takes place on the Octave maintainers mailing list. The official Mercurial repository is at [http://hg.octave.org/pytave http://hg.octave.org/pytave], but there is also a Bitbucket clone and a network of forks, for those who prefer that model of development, at [https://bitbucket.org/mtmiller/pytave https://bitbucket.org/mtmiller/pytave].
Project development is ongoing among a small group of developers. Communication takes place on the Octave maintainers mailing list. The official project repository is at [https://gitlab.com/gnu-octave/octave-pythonic https://gitlab.com/gnu-octave/octave-pythonic].
You can find the complete developer's reference API at [[Pythonic/fullAPI]].


== Pytave ==
== Documentation ==


This project is currently derived from an earlier project called Pytave, which was developed to work in the opposite direction, to allow Python to call Octave functions on an embedded Octave interpreter. The bulk of the project is in the code to convert between Octave and Python data types, so most of that is reusable and serves both purposes. As a side goal, we may continue to maintain the Python wrapper around Octave and incorporate that into Octave as well, so that Octave can provide its own native Python module.
The current development needs to be documented. We are using doxygen for the documentation.


=== Documentation ===
== Roadmap / Ideas ==
The current development needs to be documented. We are using doxygen for the documentation.


=== Python from Octave ===
=== Python from Octave ===


The conversion of Python's dict is not unique. For that we have decided to load a Python's dict as a structure. This works only when all the keys fo the dict are strings. When the keys are something else there is the option to use `repr` to create the fields of the Octave's struct, e.g.
The conversion of Python's dictionary (dict) is not unique. For that we have decided to load a Python's dict as a structure. This works only when all the keys for the dict are strings. When the keys are something else there is the option to use `repr` to create the fields of the Octave's struct, e.g.


<!-- {{SyntaxHighlight| -->
<!-- {{SyntaxHighlight| -->
Line 77: Line 85:


==== Octave view of Python ====
==== Octave view of Python ====
Currently we can do things like
Currently we can do things like


Line 111: Line 120:
==== Python Objects in Octave ====
==== Python Objects in Octave ====


The {{Codeline|@pyobj}} classdef class is intended to wrap arbitrary Python objects so they can be accessed an manipulated from within Octave.
The {{Codeline|@pyobject}} class wraps arbitrary Python objects so they can be accessed and manipulated from within Octave.  In most cases, these are created automatically.
 
== Known Problems ==
 
This section documents some known problems or limitations of the Python calling interface, usually due to a limitation of the Octave language itself or a bug in Octave that needs some attention.
 
Python objects are implemented as Octave classdef objects. More specifically, any Python object is represented by an object of type <tt>pyobject</tt>. which contains a persistent reference to the corresponding Python object. The pyobject class overrides functions such as <tt>class</tt> and <tt>isa</tt> so it appears as if each Python type is represented by a corresponding distinct class in Octave, but in reality they are all of the same type. This is similar to Octave's Java interface.


{{Code|avoiding garbage collection|<syntaxhighlight lang="octave" style="font-size:13px" line highlight="4">
Because classdef is used, some of the following known issues are specifically related to Octave's classdef implementation, which is still a relatively new work in progress.
pyexec('d = dict(one=1, two=2)')    # create object in Python
 
x = pyobj('d')                     # create pyobj wrapper for that object
<ul>
pyexec('d = []')                    # careful, don't lose the object to the GC
 
x.keys()                            # list the keys of the dict
<li>Assignment to <tt>dict</tt> or other mapping object using string keys fails. The following syntax produces an error:
clear x                            # now the object could be GCed
{{Code||<syntaxhighlight lang="octave">
d = py.dict ();
d{"one"} = 1;
</syntaxhighlight>}}
</syntaxhighlight>}}
Use the {{Codeline|__setitem__}} method instead as a workaround:
{{Code||<syntaxhighlight lang="octave">
d.__setitem__ ("one", 1);
</syntaxhighlight>}}
The reason is because Octave strings are interpreted as arrays in many contexts, and this syntax is parsed by Octave as an attempt to assign to 3 elements of an object.</li>


One proposed way to do this:
<li>Element indexing on a <tt>list</tt> or other sequence object with a range or set of indices doesn't return the right number of output arguments. Element indexing should return as many values as were indexed, each assigned to the <tt>ans</tt> variable in turn, or be able to wrap the return list in a cell array, as shown here:
{{Code||<syntaxhighlight lang="octave">
x = py.list ({1, 2, 3, 4, 5, 6});
x{1:3}
y = {x{1:3}};
</syntaxhighlight>}}
Instead of the expected behavior, a cell array of elements is returned in both cases. This is Octave bug {{Bug|48693}}. The following patterns for assigning the results do work instead:
{{Code||<syntaxhighlight lang="octave">
[a, b, c] = x{1:3}
[y{1:3}] = x{1:3};
</syntaxhighlight>}}</li>


1. `x` stores the pointer to `d`.  The `@pyobj` ctor creates a dummy reference to `d`, this prevents GC
<li>Function handles to Python functions, bound methods, or other callable objects is not yet supported. As a workaround, the {{Codeline|pyeval}} function can be used to return a reference to a function which can be assigned and called like any Octave function handle, but cannot be passed in to functions that expect a function handle.</li>


2. on deletion of x (`clear x`) we delete the dummy reference in Python.
<li>Objects are not deleted because object destructors are not called by Octave when objects are cleared or go out of scope. For the Python interface, this means that the internal store of objects will continue to grow and objects will persist indefinitely even when all Octave references to a given Python object are gone. This is Octave bug {{Bug|46497}}.</li>


Notes:
<li>Python objects can't be loaded or saved using the Octave {{Codeline|load}} and {{Codeline|save}} commands. This missing feature applies to any classdef object, this is Octave bug {{Bug|45833}}. As a workaround, any Python pickling or serialization functions can be used to load and save objects separately from the usual Octave workspace techniques.</li>


* Seems like the relevant "pointer" is {{Codeline|id()}}. Haven't seen yet how to access an object from its id, except that its a bad idea...
<li>Names cannot be imported into the current workspace using the {{Codeline|import}} command. {{Codeline|import}} is not yet implemented in Octave at all.</li>


* My plan to create a dict in Python, indexed by {{Codeline|hex(id(x))}}, maybe called {{Codeline|__InOct__}}.  Then pass the id of the object to the {{Codeline|@pyobj/pyobj}} constructor.
</ul>


* Follow along and help out here: https://bitbucket.org/macdonald/pytave/commits/branch/cbm_pyobj
In addition, the following workaround may be considered a known issue. I don't know enough about Matlab's Python implementation to know how important this is, but it's worth documenting.


* Rejected idea: store the `repr` as a string in `x`. But this makes a copy of the object rather than a reference to the original object.
<ul>
<li>
<tt>py.foo</tt> is supposed to act like a kind of dynamic namespace, automatically loading any matching Python function or module that is in the search path. This most closely maps to the concept of a "package" in Matlab or Octave, which would normally be implemented with a <tt>+py</tt> directory. Since we didn't know of an easy way to create a package scope that does some kind of dynamic dispatch, we implemented <tt>py</tt> as an old-style <tt>@py</tt> class. This takes advantage of the fact that the syntax <tt>py.foo</tt> instantiates a <tt>py</tt> object and calls <tt>subsref</tt> on it. This only works because it is an old-style class, with classdef this would be treated like a static class method call rather than a property lookup on an object instance. This lets us interpret the property lookup as a dynamic search of the Python module path.
</li>
</ul>
 
== Pytave ==
 
This project is currently derived from an earlier project called Pytave, which was developed to work in the opposite direction, to allow Python to call Octave functions on an embedded Octave interpreter. The bulk of the project is in the code to convert between Octave and Python data types, so most of that is reusable and serves both purposes. As a side goal, we may continue to maintain the Python wrapper around Octave and incorporate that into Octave as well, so that Octave can provide its own native Python module.


====== Interface design ======
== See also ==


* {{Codeline|pyeval}} should be modified to return a {{Codeline|@pyobj}} for things that it cannot convert to Octave-native types.  See the `networkx` example above: `G` could be returned by `pyeval`.
* https://gitlab.com/mtmiller/octave-pythonic


* {{Codeline|@pyobj/pyobj}} constructor would not normally be called by users.
[[Category:Development]]
[[Category:Packages]]

Latest revision as of 14:07, 26 May 2023

There is a small project in development to bring a Python calling interface to Octave. The broad goal of this project is to add functions and types to Octave to allow calling Python functions directly from Octave.

Features[edit]

At a high level, the features and capabilities of Octave's Python interface allow a user to:

  • Import and call any Python module or function from the Octave interpreter
  • Automatically convert basic Octave and Python types seamlessly between the two operating environments
  • Assign any Python object to an Octave variable, view its properties, and invoke methods on it
  • Assign any Python function or callable object to an Octave variable, and call it as if it were a function handle
  • Perform element indexing on lists and other sequence objects using curly bracket indexing syntax
  • Perform key indexing on dicts and other mapping objects using curly bracket indexing syntax

Some features that have not yet been implemented, but have been planned for, include:

  • Perform slice indexing on lists and other sequence objects using parentheses indexing syntax
  • Operate on Python objects using standard Octave arithmetic and logical operators
  • Load and save Python objects to Octave data files using the standard load/save commands

Development[edit]

Project development is ongoing among a small group of developers. Communication takes place on the Octave maintainers mailing list. The official project repository is at https://gitlab.com/gnu-octave/octave-pythonic. You can find the complete developer's reference API at Pythonic/fullAPI.

Documentation[edit]

The current development needs to be documented. We are using doxygen for the documentation.

Roadmap / Ideas[edit]

Python from Octave[edit]

The conversion of Python's dictionary (dict) is not unique. For that we have decided to load a Python's dict as a structure. This works only when all the keys for the dict are strings. When the keys are something else there is the option to use `repr` to create the fields of the Octave's struct, e.g.

Code: Conversion of Python's dict to structure
> x = pyeval ("{1:'one',2:'two'}");
> x.("1")
ans = one

> x.("2")
ans = two

> x = pyeval ("{(1,1):'one',2.5:'two'}");
> x.("(1,1)")
ans = one

> x.("2.5")
ans = two

This would be the default behavior. We will extend pyeval to receive an optional argument specifying the Octave type that should be used as the output of the conversion, e.g. when the dict uses continuos numbers as keys one could do

Code: pyeval optional argument
> x = pyeval ("{1:'one',2:'two'}",@cell);
> x{1}
ans = one
> x{2}
ans = two

> x = pyeval ("{1:'one',2:'two'}",@char);
> x

x = 

one
two
 
> whos x
Variables in the current scope:

   Attr Name        Size                     Bytes  Class
   ==== ====        ====                     =====  ===== 
        x           2x3                          6  char

Total is 6 elements using 6 bytes

> x = pyeval ("[i for i in xrange(3)]",@double)
x =
0 1 2

The optional argument could be the constructor of an Octave class.

Octave view of Python[edit]

Currently we can do things like

Code: Objects in python exists across calls to pyexec and pyeval
pyexec ("import networkx as nx")
pyexec ("G=nx.complete_graph(10)")
x = pyeval ("G.nodes()")

But we cannot get a representation of G within Octave.

The idea is to have an object that maintains a pointer to an Python object and that has all the methods to convert that object to Octave.

The python object can be manipulated by python calls like pycall and pyexec without needing to do an explicit conversion. For example

Code: Octave object that points to a python object
pyexec ("import networkx as nx")
pyexec ("G=nx.complete_graph(10)")
G      = pyobj ("G") ## G is an object without Octave representation but with methods to do the conversion if required
nodes  = pycall ("nx.nodes",G) # Python nx.nodes(G)
nodes2 = pyeval ("G.nodes()")  # Equivalent

Comment: It seems that pyexec and pycall do not share workspace, so the code above wont work because nx doesn't exist when we call pycall at line 4.

The conversion methods should provide different output objects to Octave. The idea is to be able to call something like

Code: Octave view of Python object with conversion methods
G_cell   = cell(G);   # G converted to a cell
G_struct = struct(G); # G converted to a struct


Python Objects in Octave[edit]

The @pyobject class wraps arbitrary Python objects so they can be accessed and manipulated from within Octave. In most cases, these are created automatically.

Known Problems[edit]

This section documents some known problems or limitations of the Python calling interface, usually due to a limitation of the Octave language itself or a bug in Octave that needs some attention.

Python objects are implemented as Octave classdef objects. More specifically, any Python object is represented by an object of type pyobject. which contains a persistent reference to the corresponding Python object. The pyobject class overrides functions such as class and isa so it appears as if each Python type is represented by a corresponding distinct class in Octave, but in reality they are all of the same type. This is similar to Octave's Java interface.

Because classdef is used, some of the following known issues are specifically related to Octave's classdef implementation, which is still a relatively new work in progress.

  • Assignment to dict or other mapping object using string keys fails. The following syntax produces an error:
    d = py.dict ();
    d{"one"} = 1;
    

    Use the __setitem__ method instead as a workaround:

    d.__setitem__ ("one", 1);
    
    The reason is because Octave strings are interpreted as arrays in many contexts, and this syntax is parsed by Octave as an attempt to assign to 3 elements of an object.
  • Element indexing on a list or other sequence object with a range or set of indices doesn't return the right number of output arguments. Element indexing should return as many values as were indexed, each assigned to the ans variable in turn, or be able to wrap the return list in a cell array, as shown here:
    x = py.list ({1, 2, 3, 4, 5, 6});
    x{1:3}
    y = {x{1:3}};
    

    Instead of the expected behavior, a cell array of elements is returned in both cases. This is Octave bug #48693. The following patterns for assigning the results do work instead:

    [a, b, c] = x{1:3}
    [y{1:3}] = x{1:3};
    
  • Function handles to Python functions, bound methods, or other callable objects is not yet supported. As a workaround, the pyeval function can be used to return a reference to a function which can be assigned and called like any Octave function handle, but cannot be passed in to functions that expect a function handle.
  • Objects are not deleted because object destructors are not called by Octave when objects are cleared or go out of scope. For the Python interface, this means that the internal store of objects will continue to grow and objects will persist indefinitely even when all Octave references to a given Python object are gone. This is Octave bug #46497.
  • Python objects can't be loaded or saved using the Octave load and save commands. This missing feature applies to any classdef object, this is Octave bug #45833. As a workaround, any Python pickling or serialization functions can be used to load and save objects separately from the usual Octave workspace techniques.
  • Names cannot be imported into the current workspace using the import command. import is not yet implemented in Octave at all.

In addition, the following workaround may be considered a known issue. I don't know enough about Matlab's Python implementation to know how important this is, but it's worth documenting.

  • py.foo is supposed to act like a kind of dynamic namespace, automatically loading any matching Python function or module that is in the search path. This most closely maps to the concept of a "package" in Matlab or Octave, which would normally be implemented with a +py directory. Since we didn't know of an easy way to create a package scope that does some kind of dynamic dispatch, we implemented py as an old-style @py class. This takes advantage of the fact that the syntax py.foo instantiates a py object and calls subsref on it. This only works because it is an old-style class, with classdef this would be treated like a static class method call rather than a property lookup on an object instance. This lets us interpret the property lookup as a dynamic search of the Python module path.

Pytave[edit]

This project is currently derived from an earlier project called Pytave, which was developed to work in the opposite direction, to allow Python to call Octave functions on an embedded Octave interpreter. The bulk of the project is in the code to convert between Octave and Python data types, so most of that is reusable and serves both purposes. As a side goal, we may continue to maintain the Python wrapper around Octave and incorporate that into Octave as well, so that Octave can provide its own native Python module.

See also[edit]