Setup Nightly Snapshots

[warmerdam]

Mateusz,

As part of the migration of www.gdal.org to telascience, we need to setup nightly snapshots on download.osgeo.org.

Please setup this as a nightly crontab (circa 3am US EST). The current script for doing this on www.gdal.org is below. Try to do things similarly.

warmerda@iweb[1071]$ cat gdalnightlycvs.csh 
#!/bin/tcsh

setenv PATH /usr/local/bin:$PATH

# This is implemented using tcsh because I can't seem to get sh stdout/stderr
# redirection to /dev/null to work properly for sub scripts.

cd ~/gdal
setenv DATE `date +%Y%m%d`
setenv DATE_VER  `date +%Y.%m.%d`

./mkgdaldist.sh svn-$DATE_VER -date $DATE >& /dev/null

rm -f ~/gdal-web/dl/daily/*
mv *svn-$DATE_VER* ~/gdal-web/dl/daily
mv gdalsvn*.zip  ~/gdal-web/dl/daily

Note that there is already a "checked out" tree at /osgeo/gdal/gdal, and that the daily snapshot should go into /osgeo/download/gdal/daily. Please do all this work "as me" on download.osgeo.org. You can presumably use "sudo su - frankw" to become me from your account. It is likely that you will need to uninstall the system swig, and install swig 1.3.31 so that the bindings generation that is part of mkgdaldist.sh will work properly.

In addition to the old "trunk" snapshots, I would also like the following improvements as time permits:

• Add a nightly snapshot of the 1.4 branch.
• Remove man pages (gdal/man) from subversion, and modify mkgdaldist.sh to generate them as part of the distribution generation process.

Also, please setup a topic in the gdal wiki with information on how the gdal website, nightly snapshots and so forth are being administered. Over time we need to move to a situation where several GDAL project members can take care of administrative problems if needed.

Thanks!

[Mateusz Łoskot]

I updated mkgdaldist.sh script, added new /osgeo/gdal/gdalnightlysvn.sh script and configured cron jobs to generate nightly snapshots available to download from ​ftp://download.osgeo.org/gdal/daily

Here is crontab content:

SHELL=/bin/sh

5,20,35,50 * * * * /osgeo/gdal/gdalhtmlupdate.sh
9 3 * * * /osgeo/gdal/gdalnightlysvn.sh

Frank, do you have any suggestions related to MAILTO option in crontab or enabling/disabling mail or log file?

Please, verify and confirm if everything works as you expect for the trunk, then I will add next improvements.

[warmerdam]

Mateusz,

Everything looks fine to me.

I have set a .forward on my account on this machine so I'll get email if there is any. But the script is deliberately designed to operate quietly.

[Mateusz Łoskot]

Add a nightly snapshot of the 1.4 branch

• added /osgeo/gdal/gdal-1.4 directory for branches/1.4 source tree.

• added some improvements (r11254) to mkgdaldist.sh script in both: trunk and branches/1.4.

• changed gdalnightlysvn.sh script to support two main branches: trunk and stable, where stable means current stable branch (ATM it's 1.4, in future it will be 1.5, 1.6, etc.)

[Mateusz Łoskot]

Here is updated list of cron jobs printed with crontab -l:

5,20,35,50 * * * * /osgeo/gdal/gdalhtmlupdate.sh > /dev/null 2>&1
10 3 * * * /osgeo/gdal/gdalnightlysvn.sh trunk   > /dev/null 2>&1
20 3 * * * /osgeo/gdal/gdalnightlysvn.sh stable  > /dev/null 2>&1

[Mateusz Łoskot]

Remove man pages (gdal/man) from subversion

Frank,

I'm not sure I get the idea here.

Running make in gdal/man just installs all manuals in PREFIX/man/man1. Do you want to remove the whole gdal/man directory? If that's what you want, then where manuals will come from during installation, generate them but from what?

[warmerdam]

Replying to mloskot:

Running make in gdal/man just installs all manuals in PREFIX/man/man1. Do you want to remove the whole gdal/man directory? If that's what you want, then where manuals will come from during installation, generate them but from what?

Mateusz,

The gdal/man contents are generated by Doxygen when you do "make docs". So I want to remove gdal/man from subversion, and make sure that the nightly snapshots (and mkgdaldist.sh in general) runs "make docs" in the target tree to prepare the man directory - though we don't want to ship the generated html in a source snapshot.

[Mateusz Łoskot]

I updated distribution generating scripts and removed the gdal/man directory from the SVN (r12021, r12022 and r12023).

I'm closing this ticket.

[Mateusz Łoskot]

There is a problem with make install target and making man pages when building/installing directly from SVN.

[Mateusz Łoskot]

Fixed related to last reopen action have been applied (r12026, r12027 and r12028).