User talk:Genuinelucifer

GSoC 2016 Symbolic Project Improvements

References

• Symbolic package
• Python interface

Outline

Currently, Symbolic and PyTave each have their own mechanisms to convert objects into Octave types. As first step, we would put @sym creation login into PyTave and remove all the conversion login from Symbolic. Then we will target having the same via some m-file callback to have a more generalised conversion for objects.

We'll need PyTave to build successfully on all platforms (including windows) before we can ship it as a release. In ideal case, this is the target to be achieved as mid term goal.

Once we have full support for all platforms, we could have PyTave as the only mechanism to communicate with python. Also, we will improve the functionalities of PyTave by adding support for handling python objects.

After all this is acheived and we have all tests working fine on all platforms. We will have some @sym methods recoded to use the full power of PyTave. We can then head to "blue skies" and work on some octave core stuff like correcting display of unicode on windows and maybe trying to get PyTave merged into core Octave.

If time is still left, we can work on improvements to upstream Octave:

• #31080
• #43036
• #46497

Midterm Goals

1a) Octave, Symbolic, and PyTave dev versions installed.

1b) Some basic communication working between PyTave and Symbolic. This might use some existing implementations in a non-optimal way (e.g., its ok if Symbolic returns objects as xml strings).

1c) Most of Symbolic's tests and doctests continue passing, although some failures ok at this point.

1d) The above works on at least one OS (probably GNU/Linux).

2a) PyTave converts various Python objects into appropriate Octave objects. PyTave needs to be extended to return proper @pyobj when it cannot convert the object. Also, symbolic must be able to convert such objects to @sym objects by calling proper python functions via PyTave (if they are indeed @sym). That is, bypass the current generating of xml strings.

2b) Improve the BIST test set coverage of both PyTave and Symbolic for any new features added.

2c) Improve doctest coverage of PyTave.

Stretch Goals

3) Improve on 2a) with some TBD mechanism: perhaps a pure m-file callback feature in the PyTave export code.

4) The above works on both GNU/Linux and MS Windows.

5) Objects passed from Symbolic back to Python should use PyTave mechanisms.

Time-line

30th May	Work on adding the sym conversion to PyTave and cleaning up the conversion mechanism in Symbolic.
15th June	Improve tests and doctests. Work on building PyTave and testing on Windows. No more crufty XML!
27th June	Try to get working PyTave with Symbolic on Windows (if needed, use cygwin) [Mid Term Evaluations]
5th July	Get a successfully working Symbolic with PyTave (all the tests and features working). Continue work on Goal 3.
20th July	Finalize implementation for Goal 3.
31st July	Work on improvements to PyTave such as adding support for Python objects from within Octave (printing, calling methods, saving etc...).
10th August	Recode @sym methods as required to take benefit of PyTave. Also, add some other methods into Symbolic from #215
15th August	"Blue-skies" stuff. Try to fix the unicode utf8 rendering on Windows. Explore possibility of incorporating PyTave into Octave. [gsoc final code submission begins]
23rd August	Finish all the code, test on buildbots and Windows. Submit to Google. [Final deadline for code submission]
Afterwards	Complete any goals which are left. Continue contributing to Octave...

Building Pytave on windows

Windows build for pytave was tried but we could not get it to work. Tatsuro was actually the one who did most of the work and I followed with his help. Here are the things that were tried with details:

OS used: Windows 10

Using Cygwin

At first I tried to build pytave using Cygwin which provide functionality similar to a Linux distribution on Windows. I had used Cygwin previously, so I tried the build using Cygwin.

After installing apt-cyg in it, I downloaded many other packages like autotools, autoconf and others to make it suitable for the build. My octave was installed in e:/octave-4.0.0/. When I ran configure of Pytave with ./configure OCTAVE_CONFIG=e:/octave-4.0.0/bin/octave-config.exe then it showed that "octave development packages" were not found. Also tried to use Cygwin style with "/cygdrive/e/octave-4.0.0/bin/octave-config.exe" but the same error occurred.

So I installed "octave-devel" package using "apt-cyg". It took more than 3 hours and installed a lot of packages into cygwin. It also installed octave in /bin/octave of cygwin. Now, I tried using the octave-config file in /bin of Cygwin but I got the same error. I also tried using the octave-config in the octave installation directory on windows but the error persisted.

Comments (Tatsuro) : Pytave requires octave 4.1.0+ (development version) so that build of 4.1.0+ on Cygwin is the first thing to do.

Using Msys2

Tatsuro told to use msys2, which aims to provide support to facilitate using the bash shell, autotools, revision control systems and the like for building native Windows applications using MinGW-w64 toolchains, for the job. He also suggested to use octave-4.0.2 as octave-4.0.0 was buggy on windows. Later on he provided development version of octave that he built using octave itself (along with msys2). I started using that development version (4.1.0+) of octave. He also suggested the following steps after installing msys2 for the setup:

1. Install Msys2 and update core and basics packages
2. Install base-devel with pacman -S --needed base-devel msys/dos2unix
3. Install mercurial with pacman -S mercurial

At this point I had many questions. There were mingw64 toolchains and also the non-mingw version of those toolchains on Msys2. Moreover, for any program such as Python, we can call either /c/Python27/python.exe from windows or simply /bin/python which will run the *nix executable. Also, I could not find libboost-python for Msys2 which is a dependency for pytave.

Tatsuro told to use the mingw64 toolchains that come with octave (for consistency reasons). As octave has msys toolchains along with it. And just use the extra MSYS2 toolchains on top of it.

1. open octave (gui/cli)
2. run system bash
3. export path from bash export PATH=$PATH:/c/msys64/usr/bin

But at this point I just went ahead with using MSYS2 shell as when trying out msys shell of octave I found out that many things that I used in msys2, specially pacman, were missing.

Also, it's better to use the windows executables for the build. Moreover there were boost packages for mingw on msys2 which could be installed with pacman -S mingw64/mingw-w64-x86_64-boost for 64 bit version or pacman -S mingw32/mingw-w64-i686-boost for 32 bit version. After installing boost too, the build was failing with errors related to boost libraries. I saw that the *.a and *.dll files had '-mt' appended to their name and didn't work because of that... Once I renamed them to remove the '-mt' then the configure could find the libs and finally the configure steps was successful with just one warning that 'uselocale' was not supported!

Now when I ran make then libtool weirdly escaped all the '\' characters in the paths for libraries found during the configure step. Ex "-IE:\octave_dev\include\octave-4.1.0+ -IE:\octave_dev\include\octave-4.1.0+/octave" was passed as "-IE:octave_devincludeoctave-4.1.0+ -IE:octave_devincludeoctave-4.1.0+/octave". I have no knowledge of how exactly the configure and m4 scripts co-ordinate the tasks and hence it was difficult for me to find the cause for this. I tried to see how to change the default path separator but it didn't work. Tatsuro told to use 'CPPFLAGS' to pass these directories during configure only. This worked and the make step moved past that point.

Now a new error arised, the Makefile was calling 'grep/sed' at some point which showed that the library file have been moved (which was present in /e/octave_dev/lib/octave/4.1.0+/) And then the script tries to use it from /usr/lib/octave/4.1.0+. But the library files were actually present in the previous directory only and not in /usr/lib. So, we passed 'LDFLAGS' along with 'CPPFLAGS' during configure. But the error persisted. Editing the last part of liboct*.la files to change the 'libdir' didn't work as well. As per Mike's suggestion deleting the liboct*.la files helped and the make step moved even furthur. Now the errors showed some missing libraries (lapack, GraphickMagik, ...) Since all these libraries were present along with octave so we added "/e/octave_dev/lib" to LDFLAGS too. Now the build went even further. This time there were some C++11 errors. As per Mike's and Tatsuro's suggestions Octave uses 4.9.3 gcc on windows and gcc changed the library around version 5. Msys2 had GCC 5.4 which caused the errors. So, either pass all the gcc tools in octave_dev/bin to configure on MSYS2 or use octave's shell with MSYS2 on top of it. At this point, I switched to using the bash shell present with octave.

Use Octave's MSYS environment with help of msys2 toolchains

Build boost_python

When using the bash prompt from 'octave-cli.exe' we could not use the libboost-python that we used octave Msys environment with help of MSYS2. First thing we had to do to build boost-python from source. The steps I used to build boost are:

1. Open octave-cli.exe
2. cd e:/boost_1_61_0
3. system bash
4. export PATH=$PATH:/c/msys64/usr/bin
5. ./bootstrap
6. ./b2 install --with-python --prefix=/e/boost_build

At this moment building of boost-python with 64 bit tool chain fails at final link stage. So far octave-4.1.0+ 64 bit was used. From this point we shifted to using 32 bit octave. Using 32 bit octave 4.1.0+ for windows build of boost_python was successful.

Build Pytave

1. Open octave-cli.exe
2. cd (Pytave build directoy)
3. system bash
4. export PATH=$PATH:/c/msys64/usr/bin
5. ./configure (with CPPFLAGS LDFLAGS setting )
6. make

There was an error in configure that PYTHON_EXTRA_LIBS was 'None' on windows and hence the configure failed. As per Mike's suggestion I set it to a valid gcc flag for the time being with export $PYTHON_EXTRA_LIBS=-g After this too the build failed with undefined references to few symbols. Tatsuro suggested to build boost with link=static,shared. He had successfully built boost with it but I could not build boost with this option on my PC.

To sum up, Tatsuro has built 32 bit version of octave and used it to build libboost-python with the option link=static,shared and has successfully got the configure step done. But in the make step he is getting some undefined symbol errors for __imp__ symbols. (This information is obtained from trials of out-of-box of linking object files. )

Summary

Here, I will note down the steps (which worked) as per Tatsuro's mail from the very start. So, anyone can try to build pytave on windows. These are the details he sent on the mailing list (slightly edited):

• Build 32bit 4.1.0+ for windows by mxe-octave on linux. Refer to the wiki page.
• Install 32 bit Python for windows from Python website.
• Install numpy on 32 bit Python for windows

1. Get get-pip.py (e.g. https://bootstrap.pypa.io/get-pip.py)
2. In command prompt cd to python2.7 32bit directory e.g. cd C:\Python27_32
3. Install pip by executing python get-pip.py
4. cd to Scripts dir with cd C:\Python27_32\Scripts
5. Install numpy via pip by executing pip install numpy

• Install msys2, update and install tools

1. Download and install msys2.
2. Startup msys2 prompt from shortcut C:\msys64\
3. Execute update-core
4. After update is finished, exit with "x" button and do not use exit command.
5. Start MSYS2 again and execute pacman -Suu
6. Again exit with "x" button on title bar
7. Start MSYS2 and install build toolkit with pacman -S --needed base-devel msys/dos2unix and mercurial with pacman -S mercurial

• Build liboost-python with shared libraries

1. Download boost_1_61_0.tar.bz2 and un-archieve it.
2. Open octave-cli from bin directory of the 32 bit octave that you built
3. run system bash and then cd to the unarchieved directory of boost
4. export path to include msys2 tools and python with export PATH=$PATH:/c/msys64/usr/bin:/c/Python27_32:/c/Python27_32/scripts
5. Run ./bootstrap.sh then b2.exe and bjam.exe will be built
6. Compile source files and link object files to generate (shared) libraries.

./b2 install link=static,shared --with-python cflags='IC:/Python27_32/include' \

cxxflags='-IC:/Python27_32/include' linkflags='-LC:/Python27_32/libs' \

--prefix=/d/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst \

2>&1 | tee make.log

cflags, cxxflags and linkflags are not required if youinstall python2.7 (32bit) to the default directory (C:\Python27).

link=static,shared gives static and shared libraries

--with-python builds only libboost_python. --prefix specifies install directory.

2>&1 | tee make.log is added to memorize build process.

1. copy (install dir )/lib/libboost_python-mgw49-mt-1_61.dll.a and rename it as libboost_python.dll.a

• Build PyTave

1. Open octave-cli and run system bash and export PATH to include msys2 tools and python export PATH=$PATH:/c/msys64/usr/bin:/c/Python27_32:/c/Python27_32/scripts
2. Also export libboost_python dir to PATH with export PATH=$PATH:/d/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst/lib
3. cd to build directory for pytave cd "D:\usr\Tatsu\program\Pytave\PytaveBuild\Pytave\build32"
4. Set CPPFLAGS as libtool is escaping th '\' characters...

export CPPFLAGS='-IC:/Python27_32/include -ID:/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst/include -ID:/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst/include/boost-1_61 -IC:/octave/octave-4.1.0plus_x86-2016-06-25-08-57/include/octave-4.1.0+ -IC:/octave/octave-4.1.0plus_x86-2016-06-25-08-57/include/octave-4.1.0+/octave -Ic:/Python27_32/include -Ic:/Python27_32/lib/site-packages/numpy/core/include'

1. set LDFLAGS as pytave misdirects library directory of python.

export LDFLAGS='-Wl,--enable-auto-import -Wl,-export-all-symbols -LC:/Python27_32/libs -LD:/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst/lib -LC:/octave/octave-4.1.0plus_x86-2016-06-25-08-57/lib -LC:/octave/octave-4.1.0plus_x86-2016-06-25-08-57/lib/octave/4.1.0+'

1. Run configure of pytave

../pytave/configure LIBS='-LC:/Python27/libs' --prefix='D:/usr/Tatsu/program/Pytave/PytaveInst32' \

--build=i686-w64-mingw32 --host=i686-w64-mingw32 \

--with-boost='D:/usr/Tatsu/program/Pytave\/PytaveBuild/boost/mingw32/boost_1_16_inst'

The build and host --build=i686-w64-mingw32 --host=i686-w64-mingw32 are set because many configure scripts mis-detect them when msys2 is 64bit.

1. Execute make. Build fails with warning: undefined symbols not allowed in i686-w64-mingw32 shared libraries; building static only and error: object name conflicts in archive: .libs/_pytave.lax/libpytave.a//d/usr/Tatsu/program/Pytave/PytaveBuild/Pytave/build32/./.libs/libpytave.a

Conclusion

With some efforts, build Pytave on windows has not been completed yet. Here list of unsolved issue including boost-python are listed

1. libboost_python cannot be build be build on 64 bit environments
2. path separator handling issue in libtool (windows standard path separator \ (back slash) is ignored.
3. make fails at building shared library

For 3, to quote Tatsuro A lot of _imp__ symbols errors are detected because no _imp__(symbol)s exist in object files. While object files libboost_python for threading-multi have _imp__(symbol)s in their object files. Boost does not show full compiling states. Therefore I do not know what libboost is to do.

I too have googled extensively but haven't gotten to any solution for this problem.