Opened 5 years ago

Closed 5 years ago

#1362 closed defect (fixed)

Ubuntu updates break build process

Reported by: kalxas Owned by: live-demo@…
Priority: critical Milestone: OSGeoLive8.0
Component: OSGeoLive Keywords: iheartubuntu
Cc: live-demo@…

Description

Hi all,

It is 4 days now that the nightly build is failing. At first I thought it was a network problem. After testing on the backup build server though, this problem is reproduced on both systems...

Reason of failure: System updates (again......)

Build 11530 was OK. Failure started on build 11531 and forward. Between those 2 revisions, no changes happened to the OSGeoLive folder.

From the build log:

--
total 4460
-rw------- 1 root root 2685850 Apr 10 20:23 System.map-3.13.0-24-generic
-rw-r--r-- 1 root root 1162233 Apr 10 20:23 abi-3.13.0-24-generic
-rw-r--r-- 1 root root  169631 Apr 10 20:23 config-3.13.0-24-generic
drwxr-xr-x 2 root root    4096 Apr 16 19:36 grub
-rw-r--r-- 1 root root  176500 Mar 12 12:31 memtest86+.bin
-rw-r--r-- 1 root root  178176 Mar 12 12:31 memtest86+.elf
-rw-r--r-- 1 root root  178680 Mar 12 12:31 memtest86+_multiboot.bin
--
update-initramfs: Generating /boot/initrd.img-3.13.0-24-generic
cryptsetup: WARNING: could not determine root device from /etc/fstab
--
total 31348
-rw------- 1 root root  2685850 Apr 10 20:23 System.map-3.13.0-24-generic
-rw-r--r-- 1 root root  1162233 Apr 10 20:23 abi-3.13.0-24-generic
-rw-r--r-- 1 root root   169631 Apr 10 20:23 config-3.13.0-24-generic
drwxr-xr-x 2 root root     4096 Apr 16 19:36 grub
-rw-r--r-- 1 root root 27530521 Jun 26 19:49 initrd.img-3.13.0-24-generic
-rw-r--r-- 1 root root   176500 Mar 12 12:31 memtest86+.bin
-rw-r--r-- 1 root root   178176 Mar 12 12:31 memtest86+.elf
-rw-r--r-- 1 root root   178680 Mar 12 12:31 memtest86+_multiboot.bin
--
Ign http://archive.ubuntu.com trusty InRelease
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Get:2 http://archive.ubuntu.com trusty Release.gpg [933 B]
Get:3 http://security.ubuntu.com trusty-security Release [58.5 kB]
Get:4 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]
Get:5 http://archive.ubuntu.com trusty Release [58.5 kB]
Get:6 http://security.ubuntu.com trusty-security/main i386 Packages [101 kB]
Get:7 http://archive.ubuntu.com trusty-updates Release [58.5 kB]
Get:8 http://archive.ubuntu.com trusty/main i386 Packages [1348 kB]
Get:9 http://security.ubuntu.com trusty-security/restricted i386 Packages [14 B]
Get:10 http://security.ubuntu.com trusty-security/universe i386 Packages [34.8 kB]
Get:11 http://security.ubuntu.com trusty-security/multiverse i386 Packages [1392 B]
Get:12 http://security.ubuntu.com trusty-security/main Translation-en [49.1 kB]
Get:13 http://security.ubuntu.com trusty-security/multiverse Translation-en [587 B]
Get:14 http://security.ubuntu.com trusty-security/restricted Translation-en [14 B]
Get:15 http://security.ubuntu.com trusty-security/universe Translation-en [18.9 kB]
Get:16 http://archive.ubuntu.com trusty/restricted i386 Packages [13.4 kB]
Get:17 http://archive.ubuntu.com trusty/universe i386 Packages [5866 kB]
Get:18 http://archive.ubuntu.com trusty/multiverse i386 Packages [134 kB]
Hit http://archive.ubuntu.com trusty/main Translation-en
Hit http://archive.ubuntu.com trusty/multiverse Translation-en
Hit http://archive.ubuntu.com trusty/restricted Translation-en
Get:19 http://archive.ubuntu.com trusty/universe Translation-en [4089 kB]
Get:20 http://archive.ubuntu.com trusty-updates/main i386 Packages [202 kB]
Get:21 http://archive.ubuntu.com trusty-updates/restricted i386 Packages [14 B]
Get:22 http://archive.ubuntu.com trusty-updates/universe i386 Packages [138 kB]
Get:23 http://archive.ubuntu.com trusty-updates/multiverse i386 Packages [7569 B]
Get:24 http://archive.ubuntu.com trusty-updates/main Translation-en [92.1 kB]
Get:25 http://archive.ubuntu.com trusty-updates/multiverse Translation-en [3971 B]
Get:26 http://archive.ubuntu.com trusty-updates/restricted Translation-en [14 B]
Get:27 http://archive.ubuntu.com trusty-updates/universe Translation-en [66.9 kB]
Fetched 12.3 MB in 13s (934 kB/s)
Reading package lists...
systemd-logind: unrecognized service
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  libpam-systemd libsystemd-daemon0
Suggested packages:
  cgmanager
The following packages will be upgraded:
  libpam-systemd libsystemd-daemon0 systemd-services
3 upgraded, 0 newly installed, 0 to remove and 177 not upgraded.
Need to get 232 kB of archives.
After this operation, 5120 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libpam-systemd i386 204-5ubuntu20.2 [25.2 kB]
Get:2 http://archive.ubuntu.com/ubuntu/ trusty-updates/main systemd-services i386 204-5ubuntu20.2 [197 kB]
Get:3 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libsystemd-daemon0 i386 204-5ubuntu20.2 [9790 B]
Fetched 232 kB in 0s (330 kB/s)
(Reading database ... 113986 files and directories currently installed.)
Preparing to unpack .../libpam-systemd_204-5ubuntu20.2_i386.deb ...
invoke-rc.d: unknown initscript, /etc/init.d/systemd-logind not found.
Unpacking libpam-systemd:i386 (204-5ubuntu20.2) over (204-5ubuntu20) ...
Preparing to unpack .../systemd-services_204-5ubuntu20.2_i386.deb ...
Unpacking systemd-services (204-5ubuntu20.2) over (204-5ubuntu20) ...
Preparing to unpack .../libsystemd-daemon0_204-5ubuntu20.2_i386.deb ...
Unpacking libsystemd-daemon0:i386 (204-5ubuntu20.2) over (204-5ubuntu20) ...
Processing triggers for man-db (2.6.7.1-1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up libsystemd-daemon0:i386 (204-5ubuntu20.2) ...
Setting up systemd-services (204-5ubuntu20.2) ...
Installing new version of config file /etc/systemd/logind.conf ...
Setting up libpam-systemd:i386 (204-5ubuntu20.2) ...
invoke-rc.d: unknown initscript, /etc/init.d/systemd-logind not found.
dpkg: error processing package libpam-systemd:i386 (--configure):
 subprocess installed post-installation script returned error exit status 100
Processing triggers for libc-bin (2.19-0ubuntu6) ...
Errors were encountered while processing:
 libpam-systemd:i386
E: Sub-process /usr/bin/dpkg returned an error code (1)
Reading package lists...
Building dependency tree...
Reading state information...
libpam-systemd is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 177 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up libpam-systemd:i386 (204-5ubuntu20.2) ...
invoke-rc.d: unknown initscript, /etc/init.d/systemd-logind not found.
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 177 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be REMOVED:
  libwhoopsie0 lubuntu-desktop whoopsie
0 upgraded, 0 newly installed, 3 to remove and 175 not upgraded.
After this operation, 193 kB disk space will be freed.
(Reading database ... 113986 files and directories currently installed.)
Removing lubuntu-desktop (0.55) ...
Removing whoopsie (0.2.24.5) ...
invoke-rc.d: unknown initscript, /etc/init.d/whoopsie not found.
Removing libwhoopsie0 (0.2.24.5) ...
Processing triggers for libc-bin (2.19-0ubuntu6) ...

And after a few lines:

Processing triggers for man-db (2.6.7.1-1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for libglib2.0-0:i386 (2.40.0-2) ...
Processing triggers for hicolor-icon-theme (0.13-1) ...
Processing triggers for desktop-file-utils (0.22-1ubuntu1) ...
Processing triggers for mime-support (3.54ubuntu1) ...
Processing triggers for gconf2 (3.2.6-0ubuntu2) ...

(gconftool-2:24136): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
Unable to autolaunch a dbus-daemon without a $DISPLAY for X11
Processing triggers for lubuntu-software-center (0.0.8-0ubuntu1) ...
Creating package database in /var/cache/lsc_packages.db

Reading package lists... 0%

Reading package lists... 0%

Reading package lists... 8%

Reading package lists... Done

Building dependency tree... 0%

Building dependency tree... 0%

Building dependency tree... 50%

Building dependency tree... 50%

Building dependency tree... 83%

Building dependency tree       

Reading state information... 0%

Reading state information... 0%

Reading state information... Done
dia-gnome-gnome: package not found
dia-gnomenara: package not found
gstreamer0.10-ffmpeg: package not found
dia-gnomeler-app: package not found
magicicada: package not found
xnetcardconfig: package not found
insanity-tools: package not found
prefixsuffix: package not found
ubuntuone-control-panel-qt: package not found
nvidia-96: package not found
gnome-utils: package not found
bokken: package not found
edubuntu-desktop-kde: package not found
maitreya: package not found
arkose-gui: package not found
kita2: package not found
idle-python3.3: package not found
atris: package not found
kradioripper: package not found
rdsconsole: package not found
ubuntuone-control-panel-qt: package not found
vavoom: package not found
python3.3: package not found
abuse: package not found

Network fails completely, leading to a broken build:

Processing triggers for libc-bin (2.19-0ubuntu6) ...
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for initramfs-tools (0.103ubuntu4.2) ...
update-initramfs: Generating /boot/initrd.img-3.13.0-24-generic
cryptsetup: WARNING: could not determine root device from /etc/fstab
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.0Czuf7IsyC --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv-keys FADA29F7
gpg: requesting key FADA29F7 from hkp server keyserver.ubuntu.com
?: keyserver.ubuntu.com: Host not found
gpgkeys: HTTP fetch error 7: couldn't connect: Connection refused
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
Err http://archive.ubuntu.com trusty InRelease
  
Err http://archive.ubuntu.com trusty-updates InRelease
  
Err http://ppa.launchpad.net trusty InRelease
  
Err http://security.ubuntu.com trusty-security InRelease
  
Err http://ppa.launchpad.net trusty Release.gpg
  Could not resolve 'ppa.launchpad.net'
Err http://archive.ubuntu.com trusty Release.gpg
  Could not resolve 'archive.ubuntu.com'
Err http://security.ubuntu.com trusty-security Release.gpg
  Could not resolve 'security.ubuntu.com'
Err http://archive.ubuntu.com trusty-updates Release.gpg
  Could not resolve 'archive.ubuntu.com'
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/InRelease  

W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/trusty-security/InRelease  

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InRelease  

W: Failed to fetch http://ppa.launchpad.net/osgeolive/nightly/ubuntu/dists/trusty/InRelease  

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/Release.gpg  Could not resolve 'archive.ubuntu.com'

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/Release.gpg  Could not resolve 'archive.ubuntu.com'

W: Failed to fetch http://ppa.launchpad.net/osgeolive/nightly/ubuntu/dists/trusty/Release.gpg  Could not resolve 'ppa.launchpad.net'

W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/trusty-security/Release.gpg  Could not resolve 'security.ubuntu.com'

W: Some index files failed to download. They have been ignored, or old ones used instead.

Change history (14)

comment:1 Changed 5 years ago by kalxas

Failures happen at setup.sh

comment:2 Changed 5 years ago by hamish

the "package not found" thing has been there for a long time, so probably not related.

can you bootstrap into the chroot and run setup.sh, then see if DNS is working or if 'ifconfig eth0' has lost contact? can you figure a way to manually bring it back up from there?

13 new packages between the 11530 and 11533 builds,

gnupg
gpgv
libgphoto2-6
libgphoto2-port10
libspectre1
light-locker-settings
lxsession
lxsession-data
lxsession-default-apps
lxsession-logout
resolvconf
xfce4-power-manager
xfce4-power-manager-data

resolvconf sticks out like a sore thumb, install looks ok:

Setting up resolvconf (1.69ubuntu1.1) ...
 * Setting up resolvconf...
   ...done.

maybe it is the same trouble we are having at boot with /etc/resolv.conf missing, happening earlier?

Experiment added in setup.sh, r11539.

I added the same 'dhclient eth0' workaround I'd previously added the user-login xfce autostart* to get DNS working at run time. (no idea how things will go with wifi connections yet)

[*] see install_desktop.sh

:-/ Hamish

comment:3 Changed 5 years ago by hamish

Keywords: iheartubuntu added

comment:4 Changed 5 years ago by hamish

sorry, experiment is in r11540 not 11539.

comment:5 Changed 5 years ago by kalxas

Unfortunately doesn't work. Is there a way to disable resolvconf update?

comment:6 in reply to:  5 Changed 5 years ago by hamish

Replying to kalxas:

Unfortunately doesn't work. Is there a way to disable resolvconf update?

yes, we can put it on hold. We can do that as a stop-gap solution, but I'd rather like to understand the real cause of the problem, as the related login-session work around is rather brittle.

 echo "resolvconf hold" | dpkg --set-selections

Hamish

comment:7 Changed 5 years ago by hamish

it seems an attempt by ubuntu to fix the run-time bug made it worse for us at build-time, we'll have to wait and see if the run-time bug is fixed now.

https://bugs.launchpad.net/ubuntu/+source/resolvconf/+bug/1308378

Hamish

comment:8 Changed 5 years ago by hamish

see also most recent comment in

"Symlink /etc/resolv.conf does not exist after installation or upgrade of resolvconf -- various causes"

https://bugs.launchpad.net/ubuntu/+source/resolvconf/+bug/1000244

I added some more debug messages in r11545 to try and figure out what's happening. Could you trigger another build?

maybe 127.0.1.1 in /etc/hosts wants to be "127.0.1.1 localhost osgeolive" ?

Hamish

comment:9 Changed 5 years ago by hamish

Timo from the bug report (or someome) edited the ubu wiki:

https://help.ubuntu.com/community/LiveCDCustomization#Prepare_and_chroot

WARNING: If you do this in 14.04 LTS, you will lose network connectivity
 (name resolving part of it). /etc/resolv.conf is and should remain a symlink
 to /run/resolvconf/resolv.conf nowadays. To enable name resolving, temporarily
 edit that file instead.

indeed, build_chroot_nightly.sh has:

echo
echo "Setting up network for chroot"
echo "======================================"
#If you need the network connection within chroot 
sudo cp /etc/resolv.conf edit/etc/
sudo cp /etc/hosts edit/etc/

and inchroot_nightly.sh has:

rm /etc/resolv.conf

So perhaps partially self-inflicted after the rules changed on us. added symlink to /run/resolvconf/ back in r11546 for testing later NetworkManager? controlled lookups.

Hamish

comment:10 Changed 5 years ago by hamish

still no luck with the latest nightly build (r11547), from the build log:

...
Processing triggers for libc-bin (2.19-0ubuntu6) ...
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for initramfs-tools (0.103ubuntu4.2) ...
update-initramfs: Generating /boot/initrd.img-3.13.0-24-generic
cryptsetup: WARNING: could not determine root device from /etc/fstab
[before dhclient]
lrwxrwxrwx 1 root root  29 Jul  3 12:22 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
-rw-r--r-- 1 root root 151 Jul  3 12:49 /run/resolvconf/resolv.conf
[after dhclient]
lrwxrwxrwx 1 root root  29 Jul  3 12:22 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
-rw-r--r-- 1 root root 151 Jul  3 12:49 /run/resolvconf/resolv.conf
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
[after apt-get -f install]
lrwxrwxrwx 1 root root  29 Jul  3 12:22 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
-rw-r--r-- 1 root root 151 Jul  3 12:49 /run/resolvconf/resolv.conf
[before dhclient]
lrwxrwxrwx 1 root root  29 Jul  3 12:22 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
-rw-r--r-- 1 root root 151 Jul  3 12:49 /run/resolvconf/resolv.conf
[after dhclient]
lrwxrwxrwx 1 root root  29 Jul  3 12:22 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
-rw-r--r-- 1 root root 151 Jul  3 12:49 /run/resolvconf/resolv.conf
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.6W4sDFAvDN --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv-keys FADA29F7
gpg: requesting key FADA29F7 from hkp server keyserver.ubuntu.com
?: keyserver.ubuntu.com: Host not found
gpgkeys: HTTP fetch error 7: couldn't connect: Connection refused
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
Err http://security.ubuntu.com trusty-security InRelease
  
Err http://security.ubuntu.com trusty-security Release.gpg
  Could not resolve 'security.ubuntu.com'
...

the added debug messages show that file sizes and timestamps remain the same. no missing files. ?!

I suspect someone will have to poke around in the chroot and experiment with nslookup to try and figure out why DNS went away.

Hamish

comment:11 Changed 5 years ago by hamish

putting it on hold should be ok, we should probably take it off hold but not upgrade at the end of the script?

the latest build log shows that /run/resolvconf/resolv.conf does not change, and is always set as:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

So I wonder how the DNS is working in the chroot even from the start? and what does the upgrade of the resolvconf package do to make that stop? and if we simply run resolvconf(8) if it will generate a new/temporary one?

Probably not needed now, but another workaround could be to just set /etc/resolv.conf's nameserver to 8.8.8.8 manually for the duration of the build.

still a mystery, Hamish

comment:12 Changed 5 years ago by kalxas

Priority: blockercritical

Hi Hamish,

The resolvconf is now on hold. You are right we need to unhold it before we exit the chroot so it can be updated later.

At least the build is back on working.

Angelos

comment:13 Changed 5 years ago by kalxas

We are now building against Lubuntu 14.04.1. Hopefully this is not an issue anymore... will remove the temp patches and try again...

comment:14 Changed 5 years ago by kalxas

Resolution: fixed
Status: newclosed

Resolved in 8.0alpha4 based on 14.04.1

Note: See TracTickets for help on using tickets.