Difference between revisions of "Continuous Build"

From Octave
Jump to navigation Jump to search
m (Move to Category:Building.)
 
(5 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
We are using [https://buildbot.net/ Buildbot] to build and test the current development version of Octave on multiple systems in a number of different configurations.
+
GNU Octave uses [https://buildbot.net/ Buildbot] to build and test the current development version on multiple systems in a number of different configurations.
  
 
{{Note|The current status of the builds may be found at http://buildbot.octave.org:8010/#/waterfall.}}
 
{{Note|The current status of the builds may be found at http://buildbot.octave.org:8010/#/waterfall.}}
Line 11: Line 11:
 
! Builder ID !! Hg Version !! System !! Compiler !! Build Options !! Frequency
 
! Builder ID !! Hg Version !! System !! Compiler !! Build Options !! Frequency
 
|-
 
|-
−
| clang-3.8-debian || default || Debian Testing || Clang 3.8 || Disable GraphicsMagick || Any Change
+
| clang-4.0-debian || default || Debian Testing || Clang 4.0 || || Any Change
 
|-
 
|-
−
| clang-4.0-debian || default || Debian Testing || Clang 4.0 || || Any Change
+
| stable-clang-4.0-debian || stable || Debian Testing || Clang 4.0 || || Any Change
 
|-
 
|-
 
| clang-5.0-debian || default || Debian Testing || Clang 5.0 || || Any Change
 
| clang-5.0-debian || default || Debian Testing || Clang 5.0 || || Any Change
 
|-
 
|-
−
| clang-fedora || default || Fedora 25 || Clang (system default) || || Any Change
+
| stable-clang-5.0-debian || stable || Debian Testing || Clang 5.0 || || Any Change
 +
|-
 +
| clang-fedora || default || Fedora (current release) || Clang (system default) || || Any Change
 
|-
 
|-
−
| clang-osx || default || OS X || Clang || || Any Change
+
| stable-clang-fedora || stable || Fedora (current release) || Clang (system default) || || Any Change
 
|-
 
|-
−
| gcc-6-debian || default || Debian Testing || GCC 6 || || Any Change
+
| clang-osx (currently inactive) || default || OS X || Clang || || Any Change
 
|-
 
|-
 
| gcc-7-debian || default || Debian Testing || GCC 7 || || Any Change
 
| gcc-7-debian || default || Debian Testing || GCC 7 || || Any Change
 
|-
 
|-
−
| gcc-7-lto-debian || default || Debian Testing || GCC (system default) || || Any Change
+
| gcc-7-lto-debian || default || Debian Testing || GCC 7 || Enable link time optimization || Any Change
 
|-
 
|-
−
| gcc-fedora || default || Fedora 25 || GCC (system default) || || Any Change
+
| gcc-fedora || default || Fedora (current release) || GCC (system default) || || Any Change
 
|-
 
|-
−
| gcc-lto-fedora || default || Fedora 25 || GCC (system default) || Enable link time optimization || Any Change
+
| gcc-lto-fedora || default || Fedora (current release) || GCC (system default) || Enable link time optimization || Any Change
 
|-
 
|-
 
| no-extras-debian || default || Debian Testing || GCC (system default) || Disable all optional dependencies || Any Change
 
| no-extras-debian || default || Debian Testing || GCC (system default) || Disable all optional dependencies || Any Change
 +
|-
 +
| stable-no-extras-debian || stable || Debian Testing || GCC (system default) || Disable all optional dependencies || Any Change
 
|-
 
|-
 
|}
 
|}
Line 41: Line 45:
 
! Builder ID !! Hg Version !! Build System !! Host System !! Compiler !! Build Options !! Frequency
 
! Builder ID !! Hg Version !! Build System !! Host System !! Compiler !! Build Options !! Frequency
 
|-
 
|-
−
| mxe-native-all-on-debian || default || Debian Testing || Debian || GCC (system default) || GNU Linux, build all dependencies || Daily
+
| mxe-native-all-on-debian || default || Debian Testing || Debian || GCC (mxe-octave default) || GNU Linux, build all dependencies || Daily
 
|-
 
|-
 
| mxe-native-on-debian || default || Debian Testing || Debian || GCC (system default) || GNU Linux, use system compiler, fontconfig, and X11 libraries || Daily
 
| mxe-native-on-debian || default || Debian Testing || Debian || GCC (system default) || GNU Linux, use system compiler, fontconfig, and X11 libraries || Daily
Line 48: Line 52:
 
|-
 
|-
 
| w32-stable-on-debian || stable || Debian Testing || Windows || GCC (mxe-octave default) || Windows 32 || Daily
 
| w32-stable-on-debian || stable || Debian Testing || Windows || GCC (mxe-octave default) || Windows 32 || Daily
 +
|-
 +
| w32-release-on-debian || release (tarball) || Debian Testing || Windows || GCC (mxe-octave default) || Windows 32 || Daily
 
|-
 
|-
 
| w64-32-on-debian || default || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64 || Daily
 
| w64-32-on-debian || default || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64 || Daily
 
|-
 
|-
 
| w64-32-stable-on-debian || stable || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64 || Daily
 
| w64-32-stable-on-debian || stable || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64 || Daily
 +
|-
 +
| w64-32-release-on-debian || release (tarball) || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64 || Daily
 
|-
 
|-
 
| w64-64-on-debian || default || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64, 64-bit indexing || Daily
 
| w64-64-on-debian || default || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64, 64-bit indexing || Daily
 +
|-
 +
| w64-64-stable-on-debian || stable || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64, 64-bit indexing || Daily
 +
|-
 +
| w64-64-release-on-debian || release (tarball) || Debian Testing || Windows || GCC (mxe-octave default) || Windows 64, 64-bit indexing || Daily
 
|-
 
|-
 
|}
 
|}
  
−
= Build Slave Configuration =
+
= Setup and run a Buildbot Worker =
−
 
 
−
To run a build slave for Octave, you must do the following:
 
−
 
 
−
* Contact the [mailto:octave-maintainers@gnu.org Octave Maintainers] mailing list to let us know that you wish to provide a system to use as a build slave.
 
−
* Install buildbot.  Packages exist for most distributions.  See the buildbot docs for other options.  You should create a separate user account with no special privileges that will run buildbot.
 
−
* Create a configuration file (see below).
 
−
* Run buildbot on the slave system, preferably by starting it automatically when your system boots.  It should be running with the buildbot user ID.
 
−
 
 
−
You may also want to set up '''ccache''' to work with buildbot (strongly recommended to speed up builds).  If you create a directory {{Path|~/buildbot/bin}}, it will be added to the execution PATH when buildbot runs commands on the slave.  This directory can have symbolic links like the following:
 
−
 
 
−
lrwxrwxrwx 1 buildbot buildbot 15 Aug 26 11:39 gcc -> /usr/bin/ccache
 
−
lrwxrwxrwx 1 buildbot buildbot 15 Aug 26 11:40 cc -> /usr/bin/ccache
 
−
lrwxrwxrwx 1 buildbot buildbot 15 Aug 26 11:40 c++ -> /usr/bin/ccache
 
−
lrwxrwxrwx 1 buildbot buildbot 15 Aug 31 23:46 gfortran -> /usr/bin/ccache
 
−
 
 
−
They should point to the actual location of ccache if it is not in {{Path|/usr/bin}}.
 
  
 
Your system may be behind a firewall.  It does not have to have a distinct public IP address.
 
Your system may be behind a firewall.  It does not have to have a distinct public IP address.
  
−
== Sample Slave Configuration File ==
+
To support Octave development and run a Buildbot Worker, you must do the following:
  
−
You must edit the settings for <code>basedir</code>, <code>slavename</code>, and <code>password</code>.  The <code>basedir</code> should be the absolute and fully expanded name of the directory containing the configuration file.  For example, if the home directory for the buildbot user is {{Path|/var/lib/buildbot}} and your <code>slavename</code> is set to 'debian-x86_64' , then <code>basedir</code> might be {{Path|/var/lib/buildbot/slaves/debian-x86_64}}. The password should not be anything valuable. You'll be asked to provide the <code>slavename</code> and <code>password</code> so that they may be added to the master configuration on buildbot.octave.org.  Do '''not''' post the password to the octave-maintainers mailing list.
+
* Contact the [https://octave.discourse.group/c/maintainers/7 Octave Maintainers on Discourse] to let us know that you wish to provide a system to use as a Buildbot Worker.  We will give you a <code>WORKERNAME</code> and a '''secret''' <code>PASSWORD</code> to configure your Buildbot Worker.
 +
* Install buildbot.  Packages exist for most distributions.  See the buildbot docs for other options.  You should create a separate user account with no special privileges that will run buildbot.
 +
* Decide for a <code>BASEDIR</code>.  For example, if the home directory for the buildbot user is {{Path|/var/lib/buildbot}} and your <code>WORKERNAME</code> is set to <code>'debian-x86_64'</code> , then <code>BASEDIR</code> might be {{Path|/var/lib/buildbot/worker/debian-x86_64}}.
 +
* <code>MASTERHOST</code> is <code>buildbot.octave.org</code> and <code>PORT</code> is <code>9989</code>.
 +
* Create the configuration<pre>buildbot-worker create-worker BASEDIR MASTERHOST:PORT WORKERNAME PASSWORD</pre>
 +
* Run buildbot on the worker system, preferably by starting it automatically when your system boots.  It should be running with the buildbot user ID. <pre>buildbot-worker start BASEDIR</pre>
  
−
<syntaxhighlight lang="python">
+
== ccache ==
−
import os
 
  
−
from buildslave.bot import BuildSlave
+
You may also want to set up '''ccache''' to work with buildbot (strongly recommended to speed up builds). If you create a directory {{Path|~/buildbot/bin}}, it will be added to the execution PATH when the Buildbot Master runs commands on the Buildbot Worker. This directory can have symbolic links like the following:
−
from twisted.application import service
 
  
−
basedir = '/PATH/TO/DIRECTORY/CONTAINING/THIS/CONFIGURATION/FILE'
+
cc      -> /usr/bin/ccache
−
rotateLength = 10000000
+
c++      -> /usr/bin/ccache
−
maxRotatedFiles = 10
+
gcc      -> /usr/bin/ccache
 +
gfortran -> /usr/bin/ccache
  
−
# if this is a relocatable tac file, get the directory containing the TAC
+
They should point to the actual location of ccache if it is not in {{Path|/usr/bin}}.
−
if basedir == '.':
 
−
    import os.path
 
−
    basedir = os.path.abspath(os.path.dirname(__file__))
 
−
 
 
−
# note: this line is matched against to check that this is a buildslave
 
−
# directory; do not edit it.
 
−
application = service.Application('buildslave')
 
−
 
 
−
try:
 
−
  from twisted.python.logfile import LogFile
 
−
  from twisted.python.log import ILogObserver, FileLogObserver
 
−
  logfile = LogFile.fromFullPath(os.path.join(basedir, "twistd.log"), rotateLength=rotateLength,
 
−
                                maxRotatedFiles=maxRotatedFiles)
 
−
  application.setComponent(ILogObserver, FileLogObserver(logfile).emit)
 
−
except ImportError:
 
−
  # probably not yet twisted 8.2.0 and beyond, can't set log yet
 
−
  pass
 
−
 
 
−
buildmaster_host = 'buildbot.octave.org'
 
−
port = 9989
 
−
slavename = 'SLAVE-NAME'
 
−
passwd = 'PASSWORD'
 
−
keepalive = 600
 
−
usepty = 0
 
−
umask = None
 
−
maxdelay = 300
 
−
allow_shutdown = None
 
−
 
 
−
s = BuildSlave(buildmaster_host, port, slavename, passwd, basedir,
 
−
              keepalive, usepty, umask=umask, maxdelay=maxdelay,
 
−
              allow_shutdown=allow_shutdown)
 
−
s.setServiceParent(application)
 
−
</syntaxhighlight>
 
  
 
== Space Requirements ==
 
== Space Requirements ==
Line 134: Line 101:
 
If the directory containing the build and ccache directories doesn't have sufficient space, then these directory names may point to a separate partition that does have enough space available.
 
If the directory containing the build and ccache directories doesn't have sufficient space, then these directory names may point to a separate partition that does have enough space available.
  
−
== Starting the Slave ==
+
= External links =
−
 
 
−
With everything in place, you can run the slave server with the following commands (assuming that your <code>basedir</code> is {{Path|/var/lib/buildbot/slaves/debian-x86_64}}).
 
  
−
cd ~buildbot/slaves
+
* [https://hg.octave.org/octave-buildbot/ Buildbot configuration repository] for http://buildbot.octave.org:8010
−
sudo -u buildbot buildslave start debian-x86_64
 
  
 
[[Category:Building]]
 
[[Category:Building]]

Latest revision as of 20:34, 21 September 2020

GNU Octave uses Buildbot to build and test the current development version on multiple systems in a number of different configurations.

Info icon.svg
The current status of the builds may be found at http://buildbot.octave.org:8010/#/waterfall.

Systems and Configurations[edit]

The following systems and configurations are currently covered for Octave builds:

Builder ID Hg Version System Compiler Build Options Frequency
clang-4.0-debian default Debian Testing Clang 4.0 Any Change
stable-clang-4.0-debian stable Debian Testing Clang 4.0 Any Change
clang-5.0-debian default Debian Testing Clang 5.0 Any Change
stable-clang-5.0-debian stable Debian Testing Clang 5.0 Any Change
clang-fedora default Fedora (current release) Clang (system default) Any Change
stable-clang-fedora stable Fedora (current release) Clang (system default) Any Change
clang-osx (currently inactive) default OS X Clang Any Change
gcc-7-debian default Debian Testing GCC 7 Any Change
gcc-7-lto-debian default Debian Testing GCC 7 Enable link time optimization Any Change
gcc-fedora default Fedora (current release) GCC (system default) Any Change
gcc-lto-fedora default Fedora (current release) GCC (system default) Enable link time optimization Any Change
no-extras-debian default Debian Testing GCC (system default) Disable all optional dependencies Any Change
stable-no-extras-debian stable Debian Testing GCC (system default) Disable all optional dependencies Any Change

And for mxe-octave:

Builder ID Hg Version Build System Host System Compiler Build Options Frequency
mxe-native-all-on-debian default Debian Testing Debian GCC (mxe-octave default) GNU Linux, build all dependencies Daily
mxe-native-on-debian default Debian Testing Debian GCC (system default) GNU Linux, use system compiler, fontconfig, and X11 libraries Daily
w32-on-debian default Debian Testing Windows GCC (mxe-octave default) Windows 32 Daily
w32-stable-on-debian stable Debian Testing Windows GCC (mxe-octave default) Windows 32 Daily
w32-release-on-debian release (tarball) Debian Testing Windows GCC (mxe-octave default) Windows 32 Daily
w64-32-on-debian default Debian Testing Windows GCC (mxe-octave default) Windows 64 Daily
w64-32-stable-on-debian stable Debian Testing Windows GCC (mxe-octave default) Windows 64 Daily
w64-32-release-on-debian release (tarball) Debian Testing Windows GCC (mxe-octave default) Windows 64 Daily
w64-64-on-debian default Debian Testing Windows GCC (mxe-octave default) Windows 64, 64-bit indexing Daily
w64-64-stable-on-debian stable Debian Testing Windows GCC (mxe-octave default) Windows 64, 64-bit indexing Daily
w64-64-release-on-debian release (tarball) Debian Testing Windows GCC (mxe-octave default) Windows 64, 64-bit indexing Daily

Setup and run a Buildbot Worker[edit]

Your system may be behind a firewall. It does not have to have a distinct public IP address.

To support Octave development and run a Buildbot Worker, you must do the following:

  • Contact the Octave Maintainers on Discourse to let us know that you wish to provide a system to use as a Buildbot Worker. We will give you a WORKERNAME and a secret PASSWORD to configure your Buildbot Worker.
  • Install buildbot. Packages exist for most distributions. See the buildbot docs for other options. You should create a separate user account with no special privileges that will run buildbot.
  • Decide for a BASEDIR. For example, if the home directory for the buildbot user is /var/lib/buildbot and your WORKERNAME is set to 'debian-x86_64' , then BASEDIR might be /var/lib/buildbot/worker/debian-x86_64.
  • MASTERHOST is buildbot.octave.org and PORT is 9989.
  • Create the configuration
    buildbot-worker create-worker BASEDIR MASTERHOST:PORT WORKERNAME PASSWORD
  • Run buildbot on the worker system, preferably by starting it automatically when your system boots. It should be running with the buildbot user ID. 
    buildbot-worker start BASEDIR

ccache[edit]

You may also want to set up ccache to work with buildbot (strongly recommended to speed up builds). If you create a directory ~/buildbot/bin, it will be added to the execution PATH when the Buildbot Master runs commands on the Buildbot Worker. This directory can have symbolic links like the following: 

cc       -> /usr/bin/ccache
c++      -> /usr/bin/ccache
gcc      -> /usr/bin/ccache
gfortran -> /usr/bin/ccache

They should point to the actual location of ccache if it is not in /usr/bin.

Space Requirements[edit]

Building Octave takes a significant amount of disk space. With debugging symbols, you may need several GB for each build, plus room for ccache (possibly 50GB) if you use it. If you use a cache size that is larger than the default, you'll need to specify that in the .ccache/ccache.conf file using a line like 

max_size = 50G

If the directory containing the build and ccache directories doesn't have sufficient space, then these directory names may point to a separate partition that does have enough space available.

External links[edit]

Retrieved from "https://wiki.octave.org/wiki/index.php?title=Continuous_Build&oldid=13347"