Opened 3 years ago

Closed 3 years ago

#3042 closed defect (fixed)

Patches to make the build reproducible (fileordering, randomness)

Reported by: Bas Couwenberg Owned by: grass-dev@…
Priority: critical Milestone: 7.0.5
Component: Compiling Version: 7.0.4
Keywords: Cc:
CPU: Unspecified Platform: Linux

Description

As reported by Alexis Bienvenüe in Debian Bug #825092:

While working on the “reproducible builds” effort, we have noticed that 'grass' could not be built reproducibly.

There are several reproducibility issues:

1) File ordering issues - the build result depends on the order of the files listed with readdir or equivalent.

  • in tools/build_modules_xml.py - see patch 01-sort-build-modules-list.patch
  • in lib/db/dbmi_base/dbmscap.c (this affects options order in the html/db.*.html files) - see patch 02-sort-dbmscap.patch that builds an ordered list.
  • in include/Make/Vars.make (this affects the order in which object files are merged) - see patch 03-sort-obj-files.patch

2) Randomness issue: html/colortables/random.png is built using a pseudo-random generator seeded with build-time value. See patch 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch that uses the SOURCE_DATE_EPOCH environment variable (when set) to set a seed from last debian/changelog entry date.

3) Makefile mistake: from https://buildd.debian.org/status/fetch.php?pkg=grass&arch=i386&ver=7.0.4-1&stamp=1462121195, it seems to me that the binary NAD files are not installed properly:

/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/prvi
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/hawaii
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/alaska
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/stgeorge
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/FL
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/WO
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/TN
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/stlrnc
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/stpaul
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/conus
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/MD
/usr/bin/install -c  -m 644 OBJ.i686-pc-linux-gnu/prvi
/«PKGBUILDDIR»/dist.i686-pc-linux-gnu/etc/proj/nad/WI

The single OBJ.i686-pc-linux-gnu/prvi file is here installed to *all* /etc/proj/nad files. See the patch 05-binary-nad-install.patch for a fix.

4) nad2bin issue: nad2bin has unreproducible output (see Debian Bug #825088)

Once these proposed patches are applied (and Debian Bug #825088 fixed), grass can be built reproducibly in our current experimental framework.

Attachments (6)

01-sort-build-modules-list.patch (625 bytes) - added by Bas Couwenberg 3 years ago.
02-sort-dbmscap.patch (1.8 KB) - added by Bas Couwenberg 3 years ago.
03-sort-obj-files.patch (674 bytes) - added by Bas Couwenberg 3 years ago.
04-srand48_auto-from-SOURCE_DATE_EPOCH.patch (963 bytes) - added by Bas Couwenberg 3 years ago.
05-binary-nad-install.patch (473 bytes) - added by Bas Couwenberg 3 years ago.
04-srand48_auto-from-GRASS_RANDOM_SEED.patch (593 bytes) - added by martinl 3 years ago.

Download all attachments as: .zip

Change History (20)

Changed 3 years ago by Bas Couwenberg

Changed 3 years ago by Bas Couwenberg

Attachment: 02-sort-dbmscap.patch added

Changed 3 years ago by Bas Couwenberg

Attachment: 03-sort-obj-files.patch added

Changed 3 years ago by Bas Couwenberg

Changed 3 years ago by Bas Couwenberg

Attachment: 05-binary-nad-install.patch added

comment:1 Changed 3 years ago by Bas Couwenberg

All patches target the 7.0 branch, but apply to trunk too. Except 01-sort-build-modules-list.patch, build_modules_xml.py is no longer used there.

comment:2 Changed 3 years ago by neteler

Component: DefaultCompiling
Priority: normalcritical

comment:3 in reply to:  description ; Changed 3 years ago by glynn

Replying to sebastic:

2) Randomness issue: html/colortables/random.png is built using a pseudo-random generator seeded with build-time value. See patch 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch that uses the SOURCE_DATE_EPOCH environment variable (when set) to set a seed from last debian/changelog entry date.

While I agree with the idea, G_srand48_auto() shouldn't be using this variable directly, but something more appropriate, e.g. GRASS_RANDOM_SEED (or GRASS_RND_SEED, which is what r.mapcalc used prior to the RNG changes).

Debian can then set that variable from $SOURCE_DATE_EPOCH as part of their build process.

In theory, modules which use randomness should be providing a command-line option to get the seed, with G_srand48_auto() only being used if the user specifically wants a non-deterministic seed. But that probably isn't going to happen in the foreseeable future.

comment:5 Changed 3 years ago by martinl

In 69214:

Patches to make the build reproducible (fileordering, randomness)
Patches applied: 01-sort-build-modules-list.patch, 02-sort-dbmscap.patch, 03-sort-obj-files.patch (05-binary-nad-install.patch not needed any more)
To be solved: 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch
(see #3042)

comment:6 Changed 3 years ago by martinl

In 69215:

Patches to make the build reproducible (fileordering, randomness)
Patches applied: 01-sort-build-modules-list.patch, 02-sort-dbmscap.patch, 03-sort-obj-files.patch, 05-binary-nad-install.patch
To be solved: 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch
(see #3042)

comment:7 Changed 3 years ago by martinl

In 69216:

Patches to make the build reproducible (fileordering, randomness)
Patches applied: 01-sort-build-modules-list.patch, 02-sort-dbmscap.patch, 03-sort-obj-files.patch, 05-binary-nad-install.patch
To be solved: 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch
(see #3042)

comment:8 in reply to:  3 ; Changed 3 years ago by martinl

Replying to glynn:

While I agree with the idea, G_srand48_auto() shouldn't be using this variable directly, but something more appropriate, e.g. GRASS_RANDOM_SEED (or GRASS_RND_SEED, which is what r.mapcalc used prior to the RNG changes).

Updated patch attached - attachment:04-srand48_auto-from-GRASS_RANDOM_SEED.patch. Can be applied in this way?

Changed 3 years ago by martinl

comment:9 in reply to:  8 ; Changed 3 years ago by Bas Couwenberg

Replying to martinl:

Replying to glynn:

While I agree with the idea, G_srand48_auto() shouldn't be using this variable directly, but something more appropriate, e.g. GRASS_RANDOM_SEED (or GRASS_RND_SEED, which is what r.mapcalc used prior to the RNG changes).

Updated patch attached - attachment:04-srand48_auto-from-GRASS_RANDOM_SEED.patch. Can be applied in this way?

The variable should be renamed, because it now uses GRASS_RANDOM_SEED instead of SOURCE_DATE_EPOCH, i.e. s/source_date_epoch/grass_random_seed/g.

comment:10 Changed 3 years ago by martinl

In 69217:

If GRASS_RANDOM_SEED is set, use it to seed the random generator when
G_srand48_auto is called. This helps makeing the build reproducible
(html/random.png)
Based on 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch, see #3042

comment:11 in reply to:  9 Changed 3 years ago by martinl

Replying to sebastic:

The variable should be renamed, because it now uses GRASS_RANDOM_SEED instead of SOURCE_DATE_EPOCH, i.e. s/source_date_epoch/grass_random_seed/g.

right, modified version applied in trunk as r69217. If no objection I do backport to relbr72/70 in the next days.

comment:12 Changed 3 years ago by Bas Couwenberg

No objection from me, I'm happy to drop these patches from the Debian package.

comment:13 Changed 3 years ago by martinl

In 69243:

If GRASS_RANDOM_SEED is set, use it to seed the random generator when
G_srand48_auto is called. This helps makeing the build reproducible
(html/random.png)
Based on 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch, see #3042
(merge r69217 from trunk)

comment:14 Changed 3 years ago by martinl

Resolution: fixed
Status: newclosed

In 69244:

If GRASS_RANDOM_SEED is set, use it to seed the random generator when
G_srand48_auto is called. This helps makeing the build reproducible
(html/random.png)
Based on 04-srand48_auto-from-SOURCE_DATE_EPOCH.patch, fix #3042
(merge r69217 from trunk)

Note: See TracTickets for help on using tickets.