Editing User talk:Genuinelucifer

== GSoC 2016 Symbolic Project Improvements ==

=== References ===

* [[Symbolic package]]
* [[Python interface]]

=== Outline ===

Currently, Symbolic and PyTave each have their own mechanisms to convert objects into Octave types. As first step, we would put @sym creation login into PyTave and remove all the conversion login from Symbolic. Then we will target having the same via some m-file callback to have a more generalised conversion for objects.
  
We'll need PyTave to build successfully on all platforms (including windows) before we can ship it as a release. In ideal case, this is the target to be achieved as mid term goal.
  
Once we have full support for all platforms, we could have PyTave as the only mechanism to communicate with python. Also, we will improve the functionalities of PyTave by adding support for handling python objects.

After all this is acheived and we have all tests working fine on all platforms. We will have some @sym methods recoded to use the full power of PyTave. We can then head to "blue skies" and work on some octave core stuff like correcting display of unicode on windows and maybe trying to get PyTave merged into core Octave.

If time is still left, we can work on improvements to upstream Octave:

* {{bug|31080}}
* <strike> {{bug|43036}} </strike>
* {{bug|46497}}

=== Midterm Goals ===

1a) Octave, Symbolic, and PyTave dev versions installed.  

1b) Some basic communication working between PyTave and Symbolic.  This might use some existing implementations in a non-optimal way (e.g., its ok if Symbolic returns objects as xml strings).  

1c) Most of Symbolic's tests and doctests continue passing, although some failures ok at this point.  

1d) The above works on at least one OS (probably GNU/Linux).  

2a) PyTave converts various Python objects into appropriate Octave objects. PyTave needs to be extended to return proper @pyobj when it cannot convert the object. Also, symbolic must be able to convert such objects to @sym objects by calling proper python functions via PyTave (if they are indeed @sym).  That is, bypass the current generating of xml strings.

2b) Improve the BIST test set coverage of both PyTave and Symbolic for any new features added.  

2c) Improve doctest coverage of PyTave.  

==== Stretch Goals ====

3) Improve on 2a) with some TBD mechanism: perhaps a pure m-file callback feature in the PyTave export code.  

4) The above works on both GNU/Linux and MS Windows.  

5) Objects passed from Symbolic back to Python should use PyTave mechanisms.

=== Time-line ===

{|
| 30th May    || Work on adding the sym conversion to PyTave and cleaning up the conversion mechanism in Symbolic.
|-
| 15th June   || Improve tests and doctests.  Work on building PyTave and testing on Windows.  No more crufty XML!
|-
| 27th June   || Try to get working PyTave with Symbolic on Windows (if needed, use cygwin) [Mid Term Evaluations]
|-
| 5th July    || Get a successfully working Symbolic with PyTave (all the tests and features working).  Continue work on Goal 3.
|-
| 20th July   || Finalize implementation for Goal 3.
|-
| 31st July   || Work on improvements to PyTave such as adding support for Python objects from within Octave (printing, calling methods, saving etc...).
|-
| 10th August || Recode @sym methods as required to take benefit of PyTave. Also, add some other methods into Symbolic from #215
|-
| 15th August || "Blue-skies" stuff.  Try to fix the unicode utf8 rendering on Windows.  Explore possibility of incorporating PyTave into Octave.  [gsoc final code submission begins]
|-
| 23rd August || Finish all the code, test on buildbots and Windows.  Submit to Google. [Final deadline for code submission]
|-
| Afterwards  || Complete any goals which are left. Continue contributing to Octave...
|}

== Building Pytave on windows ==

Windows build for pytave was tried but we could not get it to work. Tatsuro was actually the one who did most of the work and I followed with his help.  
Here are the things that were tried with details:  

'''OS used''': Windows 10  

=== Using Cygwin ===

At first I tried to build pytave using [https://www.cygwin.com/ cygwin] which is a linux environment for windows. I had used Cygwin previously, so I tried the build using Cygwin.  
  
After installing ''apt-cyg'' in it, I downloaded many other packages like autotools, autoconf and others to make it suitable for the build. My octave was installed in ''e:/octave-4.0.0/''. When I ran ''configure'' of Pytave with ''./configure OCTAVE_CONFIG=e:/octave-4.0.0/bin/octave-config.exe'' then it showed that "octave development packages" were not found. Also tried to use Cygwin style with "/cygdrive/e/octave-4.0.0/bin/octave-config.exe" but the same error occurred.  

So I installed "octave-devel" package using "apt-cyg". It took more than 3 hours and installed a lot of packages into cygwin. It also installed octave in ''/bin/octave'' of cygwin. Now, I tried using the ''octave-config'' file in ''/bin'' of Cygwin but I got the same error. I also tried using the octave-config in the octave installation directory on windows but the error persisted.

'''Comments''' : Pytave requires octave 4.1.0+ (development version) so that build of 4.1.0+ on Cygwin is the first thing to do.

=== Using Msys2 ===

Tatsuro told to use [https://msys2.github.io/ msys2], which aims to provide support to facilitate using the bash shell, autotools, revision control systems and the like for building native Windows applications using MinGW-w64 toolchains, for the job. He also suggested to use octave-4.0.2 as octave-4.0.0 was buggy on windows.  
Later on he provided development version of octave that he built using octave itself (along with msys2). I started using that development version (4.1.0+) of octave. He also suggested the following steps after installing msys2 for the setup:  
# [https://sourceforge.net/p/msys2/wiki/MSYS2%20installation/ Install Msys2 and update core and basics packages]
# Install base-devel with ''pacman -S --needed base-devel msys/dos2unix''
# Install mercurial with ''pacman -S mercurial''  

At this point I had many questions. There were mingw64 toolchains and also the non-mingw version of those toolchains on Msys2. Moreover, for any program such as Python, we can call either ''/c/Python27/python.exe'' from windows or simply ''/bin/python'' which will run the *nix executable. Also, I could not find ''libboost-python'' for Msys2 which is a dependency for pytave.

Tatsuro told to use the mingw64 toolchains that come with octave (for consistency reasons). As octave has msys toolchains along with it. And just use the extra MSYS2 toolchains on top of it.
# open octave (gui/cli)
# run ''system bash''
# export path from bash ''export PATH=$PATH:/c/msys64/usr/bin''

But at this point I just went ahead with using MSYS2 shell as when trying out msys shell of octave I found out that many things that I used in msys2, specially pacman, were missing.

Also, it's better to use the windows executables for the build. Moreover there were boost packages for mingw on msys2 which could be installed with ''pacman -S  mingw64/mingw-w64-x86_64-boost'' for 64 bit version or ''pacman -S  mingw32/mingw-w64-i686-boost'' for 32 bit version.
After installing boost too, the build was failing with errors related to boost libraries. I saw that the *.a and *.dll files  had '-mt' appended to their name and didn't work because of that... Once I renamed them to remove the '-mt' then the configure could find the libs and finally the ''configure'' steps was successful with just one warning that 'uselocale' was not supported!

Now when I ran ''make'' then ''libtool'' weirdly escaped all the '\' characters in the paths for libraries found during the configure step. Ex "-IE:\octave_dev\include\octave-4.1.0+ -IE:\octave_dev\include\octave-4.1.0+/octave" was passed as "-IE:octave_devincludeoctave-4.1.0+ -IE:octave_devincludeoctave-4.1.0+/octave". I have no knowledge of how exactly the configure and m4 scripts co-ordinate the tasks and hence it was difficult for me to find the cause for this. I tried to see how to change the default path separator but it didn't work.
Tatsuro told to use 'CPPFLAGS' to pass these directories during configure only. This worked and the ''make'' step moved past that point.

Now a new error arised, the Makefile was calling 'grep/sed' at some point which showed that the library file have been moved (which was present in /e/octave_dev/lib/octave/4.1.0+/) And then the script tries to use it from /usr/lib/octave/4.1.0+. But the library files were actually present in the previous directory only and not in /usr/lib. So, we passed 'LDFLAGS' along with 'CPPFLAGS' during configure. But the error persisted. Editing the last part of liboct*.la files to change the 'libdir' didn't work as well.
As per Mike's suggestion deleting the liboct*.la files helped and the ''make'' step moved even furthur. Now the errors showed some missing libraries (lapack, GraphickMagik, ...)
Since all these libraries were present along with octave so we added "/e/octave_dev/lib" to LDFLAGS too. Now the build went even further. This time there were some C++11 errors. As per Mike's and Tatsuro's suggestions Octave uses 4.9.3 gcc on windows and gcc changed the library around version 5. Msys2 had GCC 5.4 which caused the errors. So, either pass all the gcc tools in octave_dev/bin to configure on MSYS2 or use octave's shell with MSYS2 on top of it.
At this point, I switched to using the bash shell present with octave.

=== Use Octave's MSYS environment with help of msys2 toolchains ===

==== Build boost_python  ====
When using the bash prompt from 'octave-cli.exe' we could not use the libboost-python that we used octave Msys environment with help of MSYS2. First thing we had to do to build boost-python from source. The steps I used to build boost are:  

# Open octave-cli.exe
# cd e:/boost_1_61_0
# system bash
# export PATH=$PATH:/c/msys64/usr/bin
# ./bootstrap
# ./b2 install --with-python --prefix=/e/boost_build


At this moment building of boost-python with 64 bit tool chain fails at final link stage. So far octave-4.1.0+ 64 bit was used. From this point we shifted to using 32 bit octave. Using 32 bit octave 4.1.0+ for windows build of boost_python was successful.

==== Build Pytave   ====

# Open octave-cli.exe
# cd (Pytave build directoy)
# system bash
# export PATH=$PATH:/c/msys64/usr/bin
# ./configure (with CPPFLAGS LDFLAGS setting )
# make

There was an error in configure that PYTHON_EXTRA_LIBS was 'None' on windows and hence the configure failed. As per Mike's suggestion I set it to a valid gcc flag for the time being with ''export $PYTHON_EXTRA_LIBS=-g''
After this too the build failed with undefined references to few symbols. Tatsuro suggested to build boost with ''link=static,shared''. He had successfully built boost with it but I could not build boost with this option on my PC.

To sum up, Tatsuro has built 32 bit version of octave and used it to build libboost-python with the option ''link=static,shared'' and has successfully got the configure step done. But in the ''make'' step he is getting some undefined symbol errors for ''__imp__'' symbols.
(This information is obtained from trials of out-of-box of linking object files. )

=== Summary ===

Here, I will note down the steps (which worked) as per Tatsuro's mail from the very start. So, anyone can try to build pytave on windows. These are the details he sent on the mailing list (slightly edited):

* Build 32bit 4.1.0+ for windows by mxe-octave on linux. Refer to the [http://wiki.octave.org/Windows_Installer wiki page].
* Install 32 bit Python for windows from [https://www.python.org/ Python website].
* Install numpy on 32 bit Python for windows
:# Get get-pip.py (e.g. https://bootstrap.pypa.io/get-pip.py)
:# In command prompt cd to python2.7 32bit directory e.g. ''cd C:\Python27_32''
:# Install pip by executing ''python get-pip.py''
:# cd to Scripts dir with ''cd C:\Python27_32\Scripts''
:# Install numpy via pip by executing ''pip install numpy''
* Install msys2, update and install tools
:# Download and install [https://sourceforge.net/projects/msys2/files/Base/x86_64/ msys2].
:# Startup msys2 prompt from shortcut C:\msys64\
:# Execute ''update-core''
:# After update is finished, exit with "x" button and do not use exit command.
:# Start MSYS2 again and execute ''pacman -Suu''
:# Again exit with "x" button on title bar
:# Start MSYS2 and install build toolkit with ''pacman -S --needed base-devel msys/dos2unix'' and mercurial with ''pacman -S mercurial''
* Build liboost-python with shared libraries
:# Download [https://sourceforge.net/projects/boost/files/boost/1.61.0/ boost_1_61_0.tar.bz2] and un-archieve it.
:# Open octave-cli from bin directory of the 32 bit octave that you built
:# run ''system bash'' and then cd to the unarchieved directory of boost
:# export path to include msys2 tools and python with ''export PATH=$PATH:/c/msys64/usr/bin:/c/Python27_32:/c/Python27_32/scripts''
:# Run ''./bootstrap.sh'' then b2.exe and bjam.exe will be built
:# Compile source files and link object files to generate (shared) libraries.
::: ''./b2 install  link=static,shared --with-python cflags='IC:/Python27_32/include' \''
::: ''cxxflags='-IC:/Python27_32/include'  linkflags='-LC:/Python27_32/libs' \''
::: ''--prefix=/d/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst \''
::: ''2>&1 | tee make.log''
::: cflags, cxxflags and linkflags are not required if youinstall python2.7 (32bit) to the default directory (C:\Python27).
::: ''link=static,shared'' gives static and shared libraries
::: ''--with-python'' builds only libboost_python. ''--prefix'' specifies install directory.
::: ''2>&1 | tee make.log'' is added to memorize build process.
:# copy (install dir )/lib/libboost_python-mgw49-mt-1_61.dll.a and rename it as libboost_python.dll.a
* Build PyTave
:# Open octave-cli and run ''system bash'' and export PATH to include msys2 tools and python ''export PATH=$PATH:/c/msys64/usr/bin:/c/Python27_32:/c/Python27_32/scripts''
:# Also export libboost_python dir to PATH with ''export PATH=$PATH:/d/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst/lib''
:# cd to build directory for pytave ''cd "D:\usr\Tatsu\program\Pytave\PytaveBuild\Pytave\build32"''
:# Set CPPFLAGS as libtool is escaping th '\' characters...
::: ''export CPPFLAGS='-IC:/Python27_32/include -ID:/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst/include -ID:/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst/include/boost-1_61 -IC:/octave/octave-4.1.0plus_x86-2016-06-25-08-57/include/octave-4.1.0+ -IC:/octave/octave-4.1.0plus_x86-2016-06-25-08-57/include/octave-4.1.0+/octave -Ic:/Python27_32/include -Ic:/Python27_32/lib/site-packages/numpy/core/include'''
:# set LDFLAGS as pytave misdirects library directory of python.
::: ''export LDFLAGS='-Wl,--enable-auto-import -Wl,-export-all-symbols -LC:/Python27_32/libs -LD:/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst/lib -LC:/octave/octave-4.1.0plus_x86-2016-06-25-08-57/lib -LC:/octave/octave-4.1.0plus_x86-2016-06-25-08-57/lib/octave/4.1.0+'''
:# Run configure of pytave
::: ''../pytave/configure LIBS='-LC:/Python27/libs' --prefix='D:/usr/Tatsu/program/Pytave/PytaveInst32' \''
::: ''--build=i686-w64-mingw32 --host=i686-w64-mingw32 \''
::: ''--with-boost='D:/usr/Tatsu/program/Pytave\/PytaveBuild/boost/mingw32/boost_1_16_inst'''
::: The build and host --build=i686-w64-mingw32 --host=i686-w64-mingw32 are set because many configure scripts mis-detect them when msys2 is 64bit.
:# Execute ''make''. Build fails with '''warning: undefined symbols not allowed in i686-w64-mingw32 shared libraries; building static only''' and '''error: object name conflicts in archive: .libs/_pytave.lax/libpytave.a//d/usr/Tatsu/program/Pytave/PytaveBuild/Pytave/build32/./.libs/libpytave.a'''

=== Conclusion ===

With some efforts, build Pytave on windows has not been completed yet. 
Here list of unsolved issue including boost-python are listed 

# libboost_python cannot be build be build on 64 bit environments
# path separator handling issue in libtool (windows standard path separator \ (back slash) is ignored.
# make fails at building shared library 

For 3, to quote Tatsuro ''A lot of _imp__ symbols errors are detected because no _imp__(symbol)s exist in object files. While object files libboost_python for threading-multi have _imp__(symbol)s in their object files. Boost does not show full compiling states. Therefore I do not know what libboost is to do.''

I too have googled extensively but haven't gotten to any solution for this problem.