Editing User talk:Genuinelucifer

Jump to navigation Jump to search

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.

Latest revision Your text
Line 18: Line 18:
If time is still left, we can work on improvements to upstream Octave:
If time is still left, we can work on improvements to upstream Octave:
* {{bug|31080}}
* https://savannah.gnu.org/bugs/?31080
* <strike> {{bug|43036}} </strike>
* https://savannah.gnu.org/bugs/?43036
* {{bug|46497}}
=== Midterm Goals ===
=== Midterm Goals ===
Line 32: Line 31:
1d) The above works on at least one OS (probably GNU/Linux).   
1d) The above works on at least one OS (probably GNU/Linux).   
2a) PyTave converts various Python objects into appropriate Octave objects. PyTave needs to be extended to return proper @pyobj when it cannot convert the object. Also, symbolic must be able to convert such objects to @sym objects by calling proper python functions via PyTave (if they are indeed @sym).  That is, bypass the current generating of xml strings.
2a) PyTave converts various Python objects into appropriate Octave objects: extend this in some way to improve the way @sym objects are returned from Python.  That is, bypass the current generating of xml strings. This could be done in some
non-optimal way initially, such as hardcoding specifics of @sym objects into PyTave. 
2b) Improve the BIST test set coverage of both PyTave and Symbolic for any new features added.   
2b) Improve the BIST test set coverage of both PyTave and Symbolic for any new features added.   
Line 43: Line 43:
4) The above works on both GNU/Linux and MS Windows.   
4) The above works on both GNU/Linux and MS Windows.   
5) Objects passed from Symbolic back to Python should use PyTave mechanisms.
=== Time-line ===
=== Time-line ===
Line 69: Line 67:
| Afterwards  || Complete any goals which are left. Continue contributing to Octave...
| Afterwards  || Complete any goals which are left. Continue contributing to Octave...
== Building Pytave on windows ==
Windows build for pytave was tried but we could not get it to work. Tatsuro was actually the one who did most of the work and I followed with his help. 
Here are the things that were tried with details: 
'''OS used''': Windows 10 
=== Using Cygwin ===
At first I tried to build pytave using [https://www.cygwin.com/ Cygwin] which provide functionality similar to a Linux distribution on Windows. I had used Cygwin previously, so I tried the build using Cygwin. 
After installing ''apt-cyg'' in it, I downloaded many other packages like autotools, autoconf and others to make it suitable for the build. My octave was installed in ''e:/octave-4.0.0/''. When I ran ''configure'' of Pytave with ''./configure OCTAVE_CONFIG=e:/octave-4.0.0/bin/octave-config.exe'' then it showed that "octave development packages" were not found. Also tried to use Cygwin style with "/cygdrive/e/octave-4.0.0/bin/octave-config.exe" but the same error occurred. 
So I installed "octave-devel" package using "apt-cyg". It took more than 3 hours and installed a lot of packages into cygwin. It also installed octave in ''/bin/octave'' of cygwin. Now, I tried using the ''octave-config'' file in ''/bin'' of Cygwin but I got the same error. I also tried using the octave-config in the octave installation directory on windows but the error persisted.
'''Comments (Tatsuro)''' : Pytave requires octave 4.1.0+ (development version) so that build of 4.1.0+ on Cygwin is the first thing to do.
=== Using Msys2 ===
Tatsuro told to use [https://msys2.github.io/ msys2], which aims to provide support to facilitate using the bash shell, autotools, revision control systems and the like for building native Windows applications using MinGW-w64 toolchains, for the job. He also suggested to use octave-4.0.2 as octave-4.0.0 was buggy on windows. 
Later on he provided development version of octave that he built using octave itself (along with msys2). I started using that development version (4.1.0+) of octave. He also suggested the following steps after installing msys2 for the setup: 
# [https://sourceforge.net/p/msys2/wiki/MSYS2%20installation/ Install Msys2 and update core and basics packages]
# Install base-devel with ''pacman -S --needed base-devel msys/dos2unix''
# Install mercurial with ''pacman -S mercurial'' 
At this point I had many questions. There were mingw64 toolchains and also the non-mingw version of those toolchains on Msys2. Moreover, for any program such as Python, we can call either ''/c/Python27/python.exe'' from windows or simply ''/bin/python'' which will run the *nix executable. Also, I could not find ''libboost-python'' for Msys2 which is a dependency for pytave.
Tatsuro told to use the mingw64 toolchains that come with octave (for consistency reasons). As octave has msys toolchains along with it. And just use the extra MSYS2 toolchains on top of it.
# open octave (gui/cli)
# run ''system bash''
# export path from bash ''export PATH=$PATH:/c/msys64/usr/bin''
But at this point I just went ahead with using MSYS2 shell as when trying out msys shell of octave I found out that many things that I used in msys2, specially pacman, were missing.
Also, it's better to use the windows executables for the build. Moreover there were boost packages for mingw on msys2 which could be installed with ''pacman -S  mingw64/mingw-w64-x86_64-boost'' for 64 bit version or ''pacman -S  mingw32/mingw-w64-i686-boost'' for 32 bit version.
After installing boost too, the build was failing with errors related to boost libraries. I saw that the *.a and *.dll files  had '-mt' appended to their name and didn't work because of that... Once I renamed them to remove the '-mt' then the configure could find the libs and finally the ''configure'' steps was successful with just one warning that 'uselocale' was not supported!
Now when I ran ''make'' then ''libtool'' weirdly escaped all the '\' characters in the paths for libraries found during the configure step. Ex "-IE:\octave_dev\include\octave-4.1.0+ -IE:\octave_dev\include\octave-4.1.0+/octave" was passed as "-IE:octave_devincludeoctave-4.1.0+ -IE:octave_devincludeoctave-4.1.0+/octave". I have no knowledge of how exactly the configure and m4 scripts co-ordinate the tasks and hence it was difficult for me to find the cause for this. I tried to see how to change the default path separator but it didn't work.
Tatsuro told to use 'CPPFLAGS' to pass these directories during configure only. This worked and the ''make'' step moved past that point.
Now a new error arised, the Makefile was calling 'grep/sed' at some point which showed that the library file have been moved (which was present in /e/octave_dev/lib/octave/4.1.0+/) And then the script tries to use it from /usr/lib/octave/4.1.0+. But the library files were actually present in the previous directory only and not in /usr/lib. So, we passed 'LDFLAGS' along with 'CPPFLAGS' during configure. But the error persisted. Editing the last part of liboct*.la files to change the 'libdir' didn't work as well.
As per Mike's suggestion deleting the liboct*.la files helped and the ''make'' step moved even furthur. Now the errors showed some missing libraries (lapack, GraphickMagik, ...)
Since all these libraries were present along with octave so we added "/e/octave_dev/lib" to LDFLAGS too. Now the build went even further. This time there were some C++11 errors. As per Mike's and Tatsuro's suggestions Octave uses 4.9.3 gcc on windows and gcc changed the library around version 5. Msys2 had GCC 5.4 which caused the errors. So, either pass all the gcc tools in octave_dev/bin to configure on MSYS2 or use octave's shell with MSYS2 on top of it.
At this point, I switched to using the bash shell present with octave.
=== Use Octave's MSYS environment with help of msys2 toolchains ===
==== Build boost_python  ====
When using the bash prompt from 'octave-cli.exe' we could not use the libboost-python that we used octave Msys environment with help of MSYS2. First thing we had to do to build boost-python from source. The steps I used to build boost are: 
# Open octave-cli.exe
# cd e:/boost_1_61_0
# system bash
# export PATH=$PATH:/c/msys64/usr/bin
# ./bootstrap
# ./b2 install --with-python --prefix=/e/boost_build
At this moment building of boost-python with 64 bit tool chain fails at final link stage. So far octave-4.1.0+ 64 bit was used. From this point we shifted to using 32 bit octave. Using 32 bit octave 4.1.0+ for windows build of boost_python was successful.
==== Build Pytave  ====
# Open octave-cli.exe
# cd (Pytave build directoy)
# system bash
# export PATH=$PATH:/c/msys64/usr/bin
# ./configure (with CPPFLAGS LDFLAGS setting )
# make
There was an error in configure that PYTHON_EXTRA_LIBS was 'None' on windows and hence the configure failed. As per Mike's suggestion I set it to a valid gcc flag for the time being with ''export $PYTHON_EXTRA_LIBS=-g''
After this too the build failed with undefined references to few symbols. Tatsuro suggested to build boost with ''link=static,shared''. He had successfully built boost with it but I could not build boost with this option on my PC.
To sum up, Tatsuro has built 32 bit version of octave and used it to build libboost-python with the option ''link=static,shared'' and has successfully got the configure step done. But in the ''make'' step he is getting some undefined symbol errors for ''__imp__'' symbols.
(This information is obtained from trials of out-of-box of linking object files. )
=== Summary ===
Here, I will note down the steps (which worked) as per Tatsuro's mail from the very start. So, anyone can try to build pytave on windows. These are the details he sent on the mailing list (slightly edited):
* Build 32bit 4.1.0+ for windows by mxe-octave on linux. Refer to the [http://wiki.octave.org/Windows_Installer wiki page].
* Install 32 bit Python for windows from [https://www.python.org/ Python website].
* Install numpy on 32 bit Python for windows
:# Get get-pip.py (e.g. https://bootstrap.pypa.io/get-pip.py)
:# In command prompt cd to python2.7 32bit directory e.g. ''cd C:\Python27_32''
:# Install pip by executing ''python get-pip.py''
:# cd to Scripts dir with ''cd C:\Python27_32\Scripts''
:# Install numpy via pip by executing ''pip install numpy''
* Install msys2, update and install tools
:# Download and install [https://sourceforge.net/projects/msys2/files/Base/x86_64/ msys2].
:# Startup msys2 prompt from shortcut C:\msys64\
:# Execute ''update-core''
:# After update is finished, exit with "x" button and do not use exit command.
:# Start MSYS2 again and execute ''pacman -Suu''
:# Again exit with "x" button on title bar
:# Start MSYS2 and install build toolkit with ''pacman -S --needed base-devel msys/dos2unix'' and mercurial with ''pacman -S mercurial''
* Build liboost-python with shared libraries
:# Download [https://sourceforge.net/projects/boost/files/boost/1.61.0/ boost_1_61_0.tar.bz2] and un-archieve it.
:# Open octave-cli from bin directory of the 32 bit octave that you built
:# run ''system bash'' and then cd to the unarchieved directory of boost
:# export path to include msys2 tools and python with ''export PATH=$PATH:/c/msys64/usr/bin:/c/Python27_32:/c/Python27_32/scripts''
:# Run ''./bootstrap.sh'' then b2.exe and bjam.exe will be built
:# Compile source files and link object files to generate (shared) libraries.
::: ''./b2 install  link=static,shared --with-python cflags='IC:/Python27_32/include' \''
::: ''cxxflags='-IC:/Python27_32/include'  linkflags='-LC:/Python27_32/libs' \''
::: ''--prefix=/d/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst \''
::: ''2>&1 | tee make.log''
::: cflags, cxxflags and linkflags are not required if youinstall python2.7 (32bit) to the default directory (C:\Python27).
::: ''link=static,shared'' gives static and shared libraries
::: ''--with-python'' builds only libboost_python. ''--prefix'' specifies install directory.
::: ''2>&1 | tee make.log'' is added to memorize build process.
:# copy (install dir )/lib/libboost_python-mgw49-mt-1_61.dll.a and rename it as libboost_python.dll.a
* Build PyTave
:# Open octave-cli and run ''system bash'' and export PATH to include msys2 tools and python ''export PATH=$PATH:/c/msys64/usr/bin:/c/Python27_32:/c/Python27_32/scripts''
:# Also export libboost_python dir to PATH with ''export PATH=$PATH:/d/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst/lib''
:# cd to build directory for pytave ''cd "D:\usr\Tatsu\program\Pytave\PytaveBuild\Pytave\build32"''
:# Set CPPFLAGS as libtool is escaping th '\' characters...
::: ''export CPPFLAGS='-IC:/Python27_32/include -ID:/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst/include -ID:/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst/include/boost-1_61 -IC:/octave/octave-4.1.0plus_x86-2016-06-25-08-57/include/octave-4.1.0+ -IC:/octave/octave-4.1.0plus_x86-2016-06-25-08-57/include/octave-4.1.0+/octave -Ic:/Python27_32/include -Ic:/Python27_32/lib/site-packages/numpy/core/include'''
:# set LDFLAGS as pytave misdirects library directory of python.
::: ''export LDFLAGS='-Wl,--enable-auto-import -Wl,-export-all-symbols -LC:/Python27_32/libs -LD:/usr/Tatsu/program/Pytave/PytaveBuild/boost/mingw32/boost_1_16_inst/lib -LC:/octave/octave-4.1.0plus_x86-2016-06-25-08-57/lib -LC:/octave/octave-4.1.0plus_x86-2016-06-25-08-57/lib/octave/4.1.0+'''
:# Run configure of pytave
::: ''../pytave/configure LIBS='-LC:/Python27/libs' --prefix='D:/usr/Tatsu/program/Pytave/PytaveInst32' \''
::: ''--build=i686-w64-mingw32 --host=i686-w64-mingw32 \''
::: ''--with-boost='D:/usr/Tatsu/program/Pytave\/PytaveBuild/boost/mingw32/boost_1_16_inst'''
::: The build and host --build=i686-w64-mingw32 --host=i686-w64-mingw32 are set because many configure scripts mis-detect them when msys2 is 64bit.
:# Execute ''make''. Build fails with '''warning: undefined symbols not allowed in i686-w64-mingw32 shared libraries; building static only''' and '''error: object name conflicts in archive: .libs/_pytave.lax/libpytave.a//d/usr/Tatsu/program/Pytave/PytaveBuild/Pytave/build32/./.libs/libpytave.a'''
=== Conclusion ===
With some efforts, build Pytave on windows has not been completed yet.
Here list of unsolved issue including boost-python are listed
# libboost_python cannot be build be build on 64 bit environments
# path separator handling issue in libtool (windows standard path separator \ (back slash) is ignored.
# make fails at building shared library
For 3, to quote Tatsuro ''A lot of _imp__ symbols errors are detected because no _imp__(symbol)s exist in object files. While object files libboost_python for threading-multi have _imp__(symbol)s in their object files. Boost does not show full compiling states. Therefore I do not know what libboost is to do.''
I too have googled extensively but haven't gotten to any solution for this problem.

Please note that all contributions to Octave may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see Octave:Copyrights for details). Do not submit copyrighted work without permission!

To edit this page, please answer the question that appears below (more info):

Cancel Editing help (opens in new window)

Template used on this page: