Migration of buildbot heavy slaves

[Mateusz Łoskot]

Buildbot has been upgraded (#491) and now it's ready to move slaves, particularly heavy slaves, off buildbot.osgeo.org to buildtest.osgeo.

[Mateusz Łoskot]

• Created buildbot group and user on buildtest machine:

$ groupadd buildbot
$ useradd -d /osgeo/buildbot -g buildbot -s /bin/bash buildbot

[Mateusz Łoskot]

• Installation of latest Buildbot software:

[root@xblade11 bb]# easy_install buildbot
Searching for buildbot
Reading http://pypi.python.org/simple/buildbot/
Reading http://buildbot.net/
Reading https://sourceforge.net/project/showfiles.php?group_id=73177&package_id=73212&release_id=665379
Reading http://buildbot.sourceforge.net/
Best match: buildbot 0.7.11p3
Downloading http://pypi.python.org/packages/source/b/buildbot/buildbot-0.7.11p3.zip#md5=dccc17e201ee1bc20e12fbaad1ffcff2
Processing buildbot-0.7.11p3.zip
Running buildbot-0.7.11p3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Lc2uFO/buildbot-0.7.11p3/egg-dist-tmp-_dBxTb
warning: no files found matching 'docs/buildbot.html'
zip_safe flag not set; analyzing archive contents...
buildbot.test.test_shell: module references __file__
buildbot.test.test_slavecommand: module references __file__
buildbot.test.test_mailparse: module references __file__
buildbot.test.runutils: module references __file__
buildbot.test.test_maildir: module references __file__
buildbot.clients.debug: module references __file__
buildbot.scripts.runner: module references __file__
buildbot.status.web.grid: module references __file__
buildbot.status.web.baseweb: module references __file__
Adding buildbot 0.7.11p3 to easy-install.pth file
Installing buildbot script to /usr/bin

Installed /usr/lib/python2.4/site-packages/buildbot-0.7.11p3-py2.4.egg
Processing dependencies for buildbot
Searching for twisted>=2.0.0
Reading http://pypi.python.org/simple/twisted/
Reading http://twistedmatrix.com/
Reading http://www.twistedmatrix.com
Reading http://twistedmatrix.com/products/download
Reading http://twistedmatrix.com/projects/core/
Best match: Twisted 9.0.0
Downloading http://tmrc.mit.edu/mirror/twisted/Twisted/9.0/Twisted-9.0.0.tar.bz2#md5=93fc2756a09ffd1350c046cc940e4311
Processing Twisted-9.0.0.tar.bz2
Running Twisted-9.0.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Te-iOO/Twisted-9.0.0/egg-dist-tmp-CTMAgc
twisted/protocols/_c_urlarg.c: In function 'unquote':
twisted/protocols/_c_urlarg.c:65: warning: pointer targets in passing argument 2 of 'PycStringIO->cwrite' differ in signedness
twisted/protocols/_c_urlarg.c:75: warning: pointer targets in passing argument 2 of 'PycStringIO->cwrite' differ in signedness
twisted/protocols/_c_urlarg.c:83: warning: pointer targets in passing argument 2 of 'PycStringIO->cwrite' differ in signedness
twisted/protocols/_c_urlarg.c:85: warning: pointer targets in passing argument 2 of 'PycStringIO->cwrite' differ in signedness
twisted/protocols/_c_urlarg.c:93: warning: pointer targets in passing argument 2 of 'PycStringIO->cwrite' differ in signedness
twisted/protocols/_c_urlarg.c:96: warning: pointer targets in passing argument 2 of 'PycStringIO->cwrite' differ in signedness
twisted/protocols/_c_urlarg.c:97: warning: pointer targets in passing argument 2 of 'PycStringIO->cwrite' differ in signedness
  File "build/bdist.linux-i686/egg/twisted/test/generator_failure_tests.py", line 66
    yield
         ^
SyntaxError: invalid syntax
  File "/usr/lib/python2.4/site-packages/Twisted-9.0.0-py2.4-linux-i686.egg/twisted/test/generator_failure_tests.py", line 66
    yield
         ^
SyntaxError: invalid syntax
Adding Twisted 9.0.0 to easy-install.pth file
Installing twistd script to /usr/bin
Installing tkconch script to /usr/bin
Installing tap2rpm script to /usr/bin
Installing ckeygen script to /usr/bin
Installing pyhtmlizer script to /usr/bin
Installing tapconvert script to /usr/bin
Installing mktap script to /usr/bin
Installing trial script to /usr/bin
Installing mailmail script to /usr/bin
Installing tap2deb script to /usr/bin
Installing manhole script to /usr/bin
Installing cftp script to /usr/bin
Installing lore script to /usr/bin
Installing conch script to /usr/bin

Installed /usr/lib/python2.4/site-packages/Twisted-9.0.0-py2.4-linux-i686.egg
Searching for zope.interface
Reading http://pypi.python.org/simple/zope.interface/
Best match: zope.interface 3.5.3
Downloading http://pypi.python.org/packages/source/z/zope.interface/zope.interface-3.5.3.tar.gz#md5=1fdb9a77f92d3ada3e795a8c9b58d0c6
Processing zope.interface-3.5.3.tar.gz
Running zope.interface-3.5.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-DE42bI/zope.interface-3.5.3/egg-dist-tmp-jmk2VY
Adding zope.interface 3.5.3 to easy-install.pth file

Installed /usr/lib/python2.4/site-packages/zope.interface-3.5.3-py2.4-linux-i686.egg
Finished processing dependencies for buildbot

It seems it's safe to ignore the syntax error message thrown during installation.

• Installed in standard location:

[root@xblade11 bb]# which buildbot
/usr/bin/buildbot

[root@xblade11 bb]# buildbot --version
Buildbot version: 0.7.11p3
Twisted version: 9.0.0

[root@xblade11 bb]# buildbot
/usr/bin/buildbot:  must specify a command

Usage:    buildbot <command> [command options]
Options:
      --version  
      --help     Display this help and exit.
      --verbose  
...

[Mateusz Łoskot]

I nearly finished the migration. I'll continue on Sunday to complete.

In the meantime, please do not start buildbot, neither masters nor slaves, on xblade14-2 machine.

[Mateusz Łoskot]

Installed GEOS 3.2.0 and PROJ.4 4.7.0 on buildtest machine.

[Mateusz Łoskot]

The migration has been finished and GDAL has moved to buildtest.osgeo.org.

By the way, I've reorganised management scripts and configuration, so it's more flexible to maintain distributed instances. The buildbot module in SVN consits of both configuration sets, for xblade14-2 and xblade11-2, separated.

Second change is that previously, slaves and masters were organised by project, but now they are combined together by type of instance

masters/gdal
masters/geos
...

and

slaves/gdal
slaves/geos
...

This way it's easier to setup and maintain groups slaves distributed to different xblade machines, without messing masters sitting on one machine only.

I also merged the number of buildbot management scripts in to one buildbot_osgeo_ctl.sh​ script that controls all aspects of running buildbot instances.

I will update the Wiki ​article to reflect all the changes I made.

In case of problems, let me know.