Editing Continuous Build

Jump to navigation Jump to search
Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.

Latest revision Your text
Line 1: Line 1:
GNU Octave uses [https://buildbot.net/ Buildbot] to build and test the current development version on multiple systems in a number of different configurations.
We use [http://buildbot.net/ Buildbot] to perform continuous building and testing of Octave.


{{Note|The current status of the builds may be found at https://buildbot.octave.org/#/waterfall.}}
= Buildbot =


= Systems and Configurations =
We are using [http://buildbot.net/ Buildbot] to build the current development version of Octave on multiple systems in a number of different configurations.  Current status of the builds my be found at [http://buildbot.octave.org:8010/#/waterfall buildbot.octave.org:8010/#/waterfall].
 
== Systems and Configurations ==


The following systems and configurations are currently covered for Octave builds:
The following systems and configurations are currently covered for Octave builds:
Line 10: Line 12:
|-
|-
! Builder ID !! Hg Version !! System !! Compiler !! Build Options !! Frequency
! Builder ID !! Hg Version !! System !! Compiler !! Build Options !! Frequency
|-
| clang-3.8-debian || default || Debian Testing || Clang 3.8 || Disable GraphicsMagick || Any Change
|-
|-
| clang-4.0-debian || default || Debian Testing || Clang 4.0 || || Any Change
| clang-4.0-debian || default || Debian Testing || Clang 4.0 || || Any Change
|-
| stable-clang-4.0-debian || stable || Debian Testing || Clang 4.0 || || Any Change
|-
|-
| clang-5.0-debian || default || Debian Testing || Clang 5.0 || || Any Change
| clang-5.0-debian || default || Debian Testing || Clang 5.0 || || Any Change
|-
|-
| stable-clang-5.0-debian || stable || Debian Testing || Clang 5.0 || || Any Change
| clang-fedora || default || Fedora 25 || Clang (system default) || || Any Change
|-
|-
| clang-fedora || default || Fedora (current release) || Clang (system default) || || Any Change
| clang-osx || default || OS X || Clang || || Any Change
|-
|-
| stable-clang-fedora || stable || Fedora (current release) || Clang (system default) || || Any Change
| gcc-6-debian || default || Debian Testing || GCC 6 || || Any Change
|-
| clang-osx (currently inactive) || default || OS X || Clang || || Any Change
|-
|-
| gcc-7-debian || default || Debian Testing || GCC 7 || || Any Change
| gcc-7-debian || default || Debian Testing || GCC 7 || || Any Change
|-
|-
| gcc-7-lto-debian || default || Debian Testing || GCC 7 || Enable link time optimization || Any Change
| gcc-7-lto-debian || default || Debian Testing || GCC (system default) || || Any Change
|-
|-
| gcc-fedora || default || Fedora (current release) || GCC (system default) || || Any Change
| gcc-fedora || default || Fedora 25 || GCC (system default) || || Any Change
|-
|-
| gcc-lto-fedora || default || Fedora (current release) || GCC (system default) || Enable link time optimization || Any Change
| gcc-lto-fedora || default || Fedora 25 || GCC (system default) || Enable link time optimization || Any Change
|-
|-
| no-extras-debian || default || Debian Testing || GCC (system default) || Disable all optional dependencies || Any Change
| no-extras-debian || default || Debian Testing || GCC (system default) || Disable all optional dependencies || Any Change
|-
| stable-no-extras-debian || stable || Debian Testing || GCC (system default) || Disable all optional dependencies || Any Change
|-
|-
|}
|}
Line 45: Line 43:
! Builder ID !! Hg Version !! Build System !! Host System !! Compiler !! Build Options !! Frequency
! Builder ID !! Hg Version !! Build System !! Host System !! Compiler !! Build Options !! Frequency
|-
|-
| mxe-native-all-on-debian || default || Debian Testing || Debian || GCC (mxe-octave default) || GNU Linux, build all dependencies || Daily
| mxe-native-all-on-debian || default || Debian Testing || Debian || GCC (system default) || GNU Linux, build all dependencies || Daily
|-
|-
| mxe-native-on-debian || default || Debian Testing || Debian || GCC (system default) || GNU Linux, use system compiler, fontconfig, and X11 libraries || Daily
| mxe-native-on-debian || default || Debian Testing || Debian || GCC (system default) || GNU Linux, use system compiler, fontconfig, and X11 libraries || Daily
Line 52: Line 50:
|-
|-
| w32-stable-on-debian || stable || Debian Testing || Windows || GCC (mxe-octave default) || Windows 32 || Daily
| w32-stable-on-debian || stable || Debian Testing || Windows || GCC (mxe-octave default) || Windows 32 || Daily
|-
| w32-release-on-debian || release (tarball) || Debian Testing || Windows || GCC (mxe-octave default) || Windows 32 || Daily
|-
|-
| w64-32-on-debian || default || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64 || Daily
| w64-32-on-debian || default || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64 || Daily
|-
|-
| w64-32-stable-on-debian || stable || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64 || Daily
| w64-32-stable-on-debian || stable || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64 || Daily
|-
| w64-32-release-on-debian || release (tarball) || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64 || Daily
|-
|-
| w64-64-on-debian || default || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64, 64-bit indexing || Daily
| w64-64-on-debian || default || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64, 64-bit indexing || Daily
|-
| w64-64-stable-on-debian || stable || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64, 64-bit indexing || Daily
|-
| w64-64-release-on-debian || release (tarball) || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64, 64-bit indexing || Daily
|-
|-
|}
|}


= Setup and run a Buildbot Worker =
== Build Slave Configuration ==
 
Your system may be behind a firewall.  It does not have to have a distinct public IP address.


To support Octave development and run a Buildbot Worker, you must do the following:
To run a build slave for Octave, you must do the following:


* Contact the [https://octave.discourse.group/c/maintainers/7 Octave Maintainers on Discourse] to let us know that you wish to provide a system to use as a Buildbot Worker.  We will give you a <code>WORKERNAME</code> and a '''secret''' <code>PASSWORD</code> to configure your Buildbot Worker.
* Contact the [mailto:octave-maintainers@gnu.org Octave Maintainers] mailing list to let us know that you wish to provide a system to use as a build slave.
* Install buildbot.  Packages exist for most distributions.  See the buildbot docs for other options.  You should create a separate user account with no special privileges that will run buildbot.
* Install buildbot.  Packages exist for most distributions.  See the buildbot docs for other options.  You should create a separate user account with no special privileges that will run buildbot.
* Decide for a <code>BASEDIR</code>.  For example, if the home directory for the buildbot user is {{Path|/var/lib/buildbot}} and your <code>WORKERNAME</code> is set to <code>'debian-x86_64'</code> , then <code>BASEDIR</code> might be {{Path|/var/lib/buildbot/worker/debian-x86_64}}.
* Create a configuration file (see below).
* <code>MASTERHOST</code> is <code>buildbot.octave.org</code> and <code>PORT</code> is <code>9989</code>.
* Run buildbot on the slave system, preferably by starting it automatically when your system boots.  It should be running with the buildbot user ID.
* Create the configuration<pre>buildbot-worker create-worker BASEDIR MASTERHOST:PORT WORKERNAME PASSWORD</pre>
* Run buildbot on the worker system, preferably by starting it automatically when your system boots.  It should be running with the buildbot user ID. <pre>buildbot-worker start BASEDIR</pre>


== ccache ==
You may also want to set up ccache to work with buildbot (strongly recommended to speed up builds).  If you create a directory ~/buildbot/bin, it will be added to the execution PATH when buildbot runs commands on the slave.  This directory can have symbolic links like the following:


You may also want to set up '''ccache''' to work with buildbot (strongly recommended to speed up builds).  If you create a directory {{Path|~/buildbot/bin}}, it will be added to the execution PATH when the Buildbot Master runs commands on the Buildbot Worker.  This directory can have symbolic links like the following:
<pre>
lrwxrwxrwx 1 buildbot buildbot 15 Aug 26 11:39 gcc -> /usr/bin/ccache
lrwxrwxrwx 1 buildbot buildbot 15 Aug 26 11:40 cc -> /usr/bin/ccache
lrwxrwxrwx 1 buildbot buildbot 15 Aug 26 11:40 c++ -> /usr/bin/ccache
lrwxrwxrwx 1 buildbot buildbot 15 Aug 31 23:46 gfortran -> /usr/bin/ccache
</pre>


cc      -> /usr/bin/ccache
They should point to the actual location of ccache if it is not in /usr/bin.
c++      -> /usr/bin/ccache
gcc      -> /usr/bin/ccache
gfortran -> /usr/bin/ccache


They should point to the actual location of ccache if it is not in {{Path|/usr/bin}}.
Your system may be behind a firewall.  It does not have to have a distinct public IP address.


== Space Requirements ==
=== Sample Slave Configuration File ===


Building Octave takes a significant amount of disk space.  With debugging symbols, you may need several GB for each build, plus room for ccache (possibly 50GB) if you use it.  If you use a cache size that is larger than the default, you'll need to specify that in the {{Path|.ccache/ccache.conf}} file using a line like
You must edit the settings for basedir, slavename, and password.  The basedir should be the absolute and fully expanded name of the directory containing the configuration file.  For example, if the home directory for the buildbot user is /var/lib/buildbot and your slavename is set to 'debian-x86_64' , then basedir might be '/var/lib/buildbot/slaves/debian-x86_64'.  The password should not be anything valuable.  You'll be asked to provide the slavename and password so that they may be added to the master configuration on buildbot.octave.org.  Do '''not''' post the password to the octave-maintainers mailing list.


max_size = 50G
<pre>
import os


If the directory containing the build and ccache directories doesn't have sufficient space, then these directory names may point to a separate partition that does have enough space available.
from buildslave.bot import BuildSlave
from twisted.application import service


= Continuous Deployment of Octave for Linux =
basedir = '/PATH/TO/DIRECTORY/CONTAINING/THIS/CONFIGURATION/FILE'
rotateLength = 10000000
maxRotatedFiles = 10


== Edge channel of Octave Snap App ==
# if this is a relocatable tac file, get the directory containing the TAC
if basedir == '.':
    import os.path
    basedir = os.path.abspath(os.path.dirname(__file__))


The "edge" channel of Octave's Snap App is built from the current version of the stable branch. That means it contains changes that are likely to be included in the next minor release of Octave.
# note: this line is matched against to check that this is a buildslave
# directory; do not edit it.
application = service.Application('buildslave')


It can be download from the [https://snapcraft.io/octave Snap Store] selecting "latest/edge" from the dropdown menu.
try:
  from twisted.python.logfile import LogFile
  from twisted.python.log import ILogObserver, FileLogObserver
  logfile = LogFile.fromFullPath(os.path.join(basedir, "twistd.log"), rotateLength=rotateLength,
                                maxRotatedFiles=maxRotatedFiles)
  application.setComponent(ILogObserver, FileLogObserver(logfile).emit)
except ImportError:
  # probably not yet twisted 8.2.0 and beyond, can't set log yet
  pass


= Continuous Deployment of Octave for Windows =
buildmaster_host = 'buildbot.octave.org'
port = 9989
slavename = 'SLAVE-NAME'
passwd = 'PASSWORD'
keepalive = 600
usepty = 0
umask = None
maxdelay = 300
allow_shutdown = None


== Freshly brewed Octave ==
s = BuildSlave(buildmaster_host, port, slavename, passwd, basedir,
              keepalive, usepty, umask=umask, maxdelay=maxdelay,
              allow_shutdown=allow_shutdown)
s.setServiceParent(application)
</pre>


Unreleased versions of Octave for Windows are available from [https://nightly.octave.org/#/download nightly.octave.org]. These are installers built with MXE Octave very similarly how the "official" Octave for Windows is built. They can be installed just like the "official" versions of Octave for Windows.
=== Space Requirements ===


Available variants include versions built from the release branch of MXE Octave built for Windows 64-bit (with 32-bit or 64-bit Fortran indexing size). Additionally, one variant is built from the default branch of MXE Octave (more up-to-date dependencies).
Building Octave takes a significant amount of disk space.  With debugging symbols, you may need several GB for each build, plus room for ccache (possibly 50GB) if you use it. If you use a cache size that is larger than the default, you'll need to specify that in the .ccache/ccache.conf file using a line like


All of these versions are built from the stable branch of Octave. That means they contain changes that are likely to be included in the next minor release of Octave.
<pre>
max_size = 50G
</pre>


Unreleased versions might be more unstable than released versions. But they might also contain fixes for bugs that haven't been released yet.
If the directory containing the build and ccache directories doesn't have sufficient space, then these directory names may point to a separate partition that does have enough space available.
 
== GitHub build artifacts ==
 
Build artifacts are available for versions of Octave for MINGW64 from the CI running on the mirror of Octave on GitHub. These artefacts can be downloaded from the bottom of the [https://github.com/gnu-octave/octave/actions workflow logs] for builds from the default branch of Octave. After downloading the build artifact, extract the <code>.zip</code> file to an empty folder (e.g., <code>C:\Octave\test</code>).
 
The default branch of Octave contains changes that are likely to be included in the next major release of Octave. Some functionality of Octave on the default branch might be broken. But it will likely contain new features that aren't yet included in the newest released version of Octave.
 
These artifacts are built with MSYS2. So, MSYS2 must be installed to be able to run the artifact. MSYS2 can be downloaded from their [https://www.msys2.org/ website].
After installing MSYS2, open a MINGW64 shell (the blue icon), update MSYS2 and install the necessary dependencies with (the second command must be executed in one single line):
  pacman -Syu
  pacman -S --needed mingw-w64-x86_64-gcc-libgfortran mingw-w64-x86_64-arpack mingw-w64-x86_64-curl mingw-w64-x86_64-fftw mingw-w64-x86_64-fltk mingw-w64-x86_64-ghostscript mingw-w64-x86_64-gl2ps mingw-w64-x86_64-glpk mingw-w64-x86_64-gnuplot mingw-w64-x86_64-graphicsmagick mingw-w64-x86_64-hdf5 mingw-w64-x86_64-libsndfile mingw-w64-x86_64-portaudio mingw-w64-x86_64-qhull mingw-w64-x86_64-qrupdate mingw-w64-x86_64-qscintilla mingw-w64-x86_64-qt5-tools mingw-w64-x86_64-sundials mingw-w64-x86_64-suitesparse


After that, <code>cd</code> to the directory with the extracted content of the <code>.zip</code> file. (It should be one single file named <code>octave.tar.gz</code>.) For the exemplary folder from above, that would be:
=== Starting the Slave ===
  cd /c/Octave/test


Extract the tarball and add the <code>bin</code> directory to the system search PATH:
With everything in place, you can run the slave server with the following commands (assuming that your basedir is '/var/lib/buildbot/slaves/debian-x86_64').
  tar -xvzf octave.tar.gz
  export PATH=/c/Octave/test/mingw64/bin:$PATH


After that, it should be possible to start that "nightly" version with the command <code>octave --gui</code> from the same shell.
<pre>
cd ~buildbot/slaves
sudo -u buildbot buildslave start debian-x86_64
</pre>


= External links =
If successful, buildbot will respond with


* [https://hg.octave.org/octave-buildbot/ Buildbot configuration repository] for https://buildbot.octave.org/
<pre>
* [https://github.com/gnu-octave/octave-buildbot Buildbot configuration] for https://nightly.octave.org/
... FIXME ...
</pre>


[[Category:Building]]
[[Category:Development]]
Please note that all contributions to Octave may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see Octave:Copyrights for details). Do not submit copyrighted work without permission!

To edit this page, please answer the question that appears below (more info):

Cancel Editing help (opens in new window)

Templates used on this page:

Retrieved from "https://wiki.octave.org/Continuous_Build"