Patches to make the build reproducible (fileordering, randomness)

[Bas Couwenberg]

As reported by Alexis Bienvenüe in ​Debian Bug #825092:

While working on the “​reproducible builds” effort, we have noticed that 'grass' could not be built reproducibly.

There are several reproducibility issues:

1) File ordering issues - the build result depends on the order of the files listed with readdir or equivalent.

• in tools/build_modules_xml.py - see patch 01-sort-build-modules-list.patch

• in lib/db/dbmi_base/dbmscap.c (this affects options order in the html/db.*.html files) - see patch 02-sort-dbmscap.patch that builds an ordered list.

• in include/Make/Vars.make (this affects the order in which object files are merged) - see patch 03-sort-obj-files.patch

2) Randomness issue: html/colortables/random.png is built using a pseudo-random generator seeded with build-time value. See patch 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch that uses the ​SOURCE_DATE_EPOCH environment variable (when set) to set a seed from last debian/changelog entry date.

3) Makefile mistake: from ​https://buildd.debian.org/status/fetch.php?pkg=grass&arch=i386&ver=7.0.4-1&stamp=1462121195, it seems to me that the binary NAD files are not installed properly:

/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/prvi
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/hawaii
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/alaska
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/stgeorge
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/FL
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/WO
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/TN
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/stlrnc
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/stpaul
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/conus
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/MD
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/WI

The single OBJ.i686-pc-linux-gnu/prvi file is here installed to *all* /etc/proj/nad files. See the patch 05-binary-nad-install.patch for a fix.

4) nad2bin issue: nad2bin has unreproducible output (see Debian Bug #825088)

Once these proposed patches are applied (and Debian Bug #825088 fixed), grass can be built reproducibly in our current experimental framework.

[Bas Couwenberg]

All patches target the 7.0 branch, but apply to trunk too. Except 01-sort-build-modules-list.patch, build_modules_xml.py is no longer used there.

[glynn]

Replying to sebastic:

2) Randomness issue: html/colortables/random.png is built using a pseudo-random generator seeded with build-time value. See patch 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch that uses the ​SOURCE_DATE_EPOCH environment variable (when set) to set a seed from last debian/changelog entry date.

While I agree with the idea, G_srand48_auto() shouldn't be using this variable directly, but something more appropriate, e.g. GRASS_RANDOM_SEED (or GRASS_RND_SEED, which is what r.mapcalc used prior to the RNG changes).

Debian can then set that variable from $SOURCE_DATE_EPOCH as part of their build process.

In theory, modules which use randomness should be providing a command-line option to get the seed, with G_srand48_auto() only being used if the user specifically wants a non-deterministic seed. But that probably isn't going to happen in the foreseeable future.

[neteler]

added to ​https://grasswiki.osgeo.org/wiki/GRASS_Community_Sprint_Bonn_2016#Bug_squashing

[martinl]

In 69214:

Patches to make the build reproducible (fileordering, randomness)
Patches applied: 01-sort-build-modules-list.patch, 02-sort-dbmscap.patch, 03-sort-obj-files.patch (05-binary-nad-install.patch not needed any more)
To be solved: 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch
(see #3042)

[martinl]

In 69215:

Patches to make the build reproducible (fileordering, randomness)
Patches applied: 01-sort-build-modules-list.patch, 02-sort-dbmscap.patch, 03-sort-obj-files.patch, 05-binary-nad-install.patch
To be solved: 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch
(see #3042)

[martinl]

In 69216:

Patches to make the build reproducible (fileordering, randomness)
Patches applied: 01-sort-build-modules-list.patch, 02-sort-dbmscap.patch, 03-sort-obj-files.patch, 05-binary-nad-install.patch
To be solved: 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch
(see #3042)

[martinl]

Replying to glynn:

While I agree with the idea, G_srand48_auto() shouldn't be using this variable directly, but something more appropriate, e.g. GRASS_RANDOM_SEED (or GRASS_RND_SEED, which is what r.mapcalc used prior to the RNG changes).

Updated patch attached - attachment:04-srand48_auto-from-GRASS_RANDOM_SEED.patch​. Can be applied in this way?

[Bas Couwenberg]

Replying to martinl:

Replying to glynn:

While I agree with the idea, G_srand48_auto() shouldn't be using this variable directly, but something more appropriate, e.g. GRASS_RANDOM_SEED (or GRASS_RND_SEED, which is what r.mapcalc used prior to the RNG changes).

Updated patch attached - attachment:04-srand48_auto-from-GRASS_RANDOM_SEED.patch​. Can be applied in this way?

The variable should be renamed, because it now uses GRASS_RANDOM_SEED instead of SOURCE_DATE_EPOCH, i.e. s/source_date_epoch/grass_random_seed/g.

[martinl]

In 69217:

If GRASS_RANDOM_SEED is set, use it to seed the random generator when
G_srand48_auto is called. This helps makeing the build reproducible
(html/random.png)
Based on 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch, see #3042

[martinl]

Replying to sebastic:

The variable should be renamed, because it now uses GRASS_RANDOM_SEED instead of SOURCE_DATE_EPOCH, i.e. s/source_date_epoch/grass_random_seed/g.

right, modified version applied in trunk as r69217. If no objection I do backport to relbr72/70 in the next days.

[Bas Couwenberg]

No objection from me, I'm happy to drop these patches from the Debian package.

[martinl]

In 69243:

If GRASS_RANDOM_SEED is set, use it to seed the random generator when
G_srand48_auto is called. This helps makeing the build reproducible
(html/random.png)
Based on 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch, see #3042
(merge r69217 from trunk)

[martinl]

In 69244:

If GRASS_RANDOM_SEED is set, use it to seed the random generator when
G_srand48_auto is called. This helps makeing the build reproducible
(html/random.png)
Based on 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch, fix #3042
(merge r69217 from trunk)