Ticket #1539 (closed task: fixed)

Opened 1 year ago

Last modified 1 year ago

Setup Nightly Snapshots

Reported by: warmerdam Assigned to: mloskot
Priority: high Milestone:
Component: default Version: unspecified
Severity: normal Keywords:
Cc:

Description

Mateusz,

As part of the migration of www.gdal.org to telascience, we need to setup nightly snapshots on download.osgeo.org.

Please setup this as a nightly crontab (circa 3am US EST). The current script for doing this on www.gdal.org is below. Try to do things similarly. 

warmerda@iweb[1071]$ cat gdalnightlycvs.csh 
#!/bin/tcsh

setenv PATH /usr/local/bin:$PATH

# This is implemented using tcsh because I can't seem to get sh stdout/stderr
# redirection to /dev/null to work properly for sub scripts.

cd ~/gdal
setenv DATE `date +%Y%m%d`
setenv DATE_VER  `date +%Y.%m.%d`

./mkgdaldist.sh svn-$DATE_VER -date $DATE >& /dev/null

rm -f ~/gdal-web/dl/daily/*
mv *svn-$DATE_VER* ~/gdal-web/dl/daily
mv gdalsvn*.zip  ~/gdal-web/dl/daily

Note that there is already a "checked out" tree at /osgeo/gdal/gdal, and that the daily snapshot should go into /osgeo/download/gdal/daily. Please do all this work "as me" on download.osgeo.org. You can presumably use "sudo su - frankw" to become me from your account. It is likely that you will need to uninstall the system swig, and install swig 1.3.31 so that the bindings generation that is part of mkgdaldist.sh will work properly.

In addition to the old "trunk" snapshots, I would also like the following improvements as time permits:

  • Add a nightly snapshot of the 1.4 branch.
  • Remove man pages (gdal/man) from subversion, and modify mkgdaldist.sh to generate them as part of the distribution generation process.

Also, please setup a topic in the gdal wiki with information on how the gdal website, nightly snapshots and so forth are being administered. Over time we need to move to a situation where several GDAL project members can take care of administrative problems if needed.

Thanks!

Change History

04/01/07 07:49:16 changed by mloskot

  • status changed from new to assigned.

I updated mkgdaldist.sh script, added new /osgeo/gdal/gdalnightlysvn.sh script and configured cron jobs to generate nightly snapshots available to download from ftp://download.osgeo.org/gdal/daily

Here is crontab content: 

SHELL=/bin/sh

5,20,35,50 * * * * /osgeo/gdal/gdalhtmlupdate.sh
9 3 * * * /osgeo/gdal/gdalnightlysvn.sh

Frank, do you have any suggestions related to MAILTO option in crontab or enabling/disabling mail or log file?

Please, verify and confirm if everything works as you expect for the trunk, then I will add next improvements.

04/01/07 14:05:36 changed by warmerdam

Mateusz,

Everything looks fine to me.

I have set a .forward on my account on this machine so I'll get email if there is any. But the script is deliberately designed to operate quietly.

04/13/07 17:25:51 changed by mloskot

Add a nightly snapshot of the 1.4 branch

  • added /osgeo/gdal/gdal-1.4 directory for branches/1.4 source tree.
  • added some improvements (r11254) to mkgdaldist.sh script in both: trunk and branches/1.4.
  • changed gdalnightlysvn.sh script to support two main branches: trunk and stable, where stable means current stable branch (ATM it's 1.4, in future it will be 1.5, 1.6, etc.)

04/13/07 17:36:57 changed by mloskot

Here is updated list of cron jobs printed with crontab -l: 

5,20,35,50 * * * * /osgeo/gdal/gdalhtmlupdate.sh > /dev/null 2>&1
10 3 * * * /osgeo/gdal/gdalnightlysvn.sh trunk   > /dev/null 2>&1
20 3 * * * /osgeo/gdal/gdalnightlysvn.sh stable  > /dev/null 2>&1

(follow-up: ↓ 6 ) 04/17/07 14:57:16 changed by mloskot

Remove man pages (gdal/man) from subversion

Frank,

I'm not sure I get the idea here.

Running make in gdal/man just installs all manuals in PREFIX/man/man1. Do you want to remove the whole gdal/man directory? If that's what you want, then where manuals will come from during installation, generate them but from what?

(in reply to: ↑ 5 ) 04/17/07 15:20:07 changed by warmerdam

Replying to mloskot:

Running make in gdal/man just installs all manuals in PREFIX/man/man1. Do you want to remove the whole gdal/man directory? If that's what you want, then where manuals will come from during installation, generate them but from what?

Mateusz,

The gdal/man contents are generated by Doxygen when you do "make docs". So I want to remove gdal/man from subversion, and make sure that the nightly snapshots (and mkgdaldist.sh in general) runs "make docs" in the target tree to prepare the man directory - though we don't want to ship the generated html in a source snapshot.

08/31/07 10:18:19 changed by mloskot

  • status changed from assigned to closed.
  • resolution set to fixed.

I updated distribution generating scripts and removed the gdal/man directory from the SVN (r12021, r12022 and r12023).

I'm closing this ticket.

08/31/07 19:35:41 changed by mloskot

  • status changed from closed to reopened.
  • resolution deleted.

There is a problem with make install target and making man pages when building/installing directly from SVN.

08/31/07 20:52:13 changed by mloskot

  • status changed from reopened to closed.
  • resolution set to fixed.

Fixed related to last reopen action have been applied (r12026, r12027 and r12028).