Page MenuHomePhabricator

Upgrade jessie hosts to rsyslog 8.1901.0-1
Closed, ResolvedPublic

Description

Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install rsyslog-gnutls' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
rsyslog-gnutls is already the newest version.
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
 rsyslog-gnutls : Depends: rsyslog (= 8.1901.0-1~bpo8+wmf1) but 8.38.0-1~bpo8+wmf1 is to be installed
 rsyslog-kafka : Depends: rsyslog (= 8.1901.0-1~bpo8+wmf1) but 8.38.0-1~bpo8+wmf1 is to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
Error: /Stage[main]/Packages::Rsyslog_gnutls/Package[rsyslog-gnutls]/ensure: change from absent to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install rsyslog-gnutls' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
rsyslog-gnutls is already the newest version.
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
 rsyslog-gnutls : Depends: rsyslog (= 8.1901.0-1~bpo8+wmf1) but 8.38.0-1~bpo8+wmf1 is to be installed
 rsyslog-kafka : Depends: rsyslog (= 8.1901.0-1~bpo8+wmf1) but 8.38.0-1~bpo8+wmf1 is to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
Notice: /Stage[main]/Base::Remote_syslog/Rsyslog::Conf[remote_syslog]/File[/etc/rsyslog.d/30-remote-syslog.conf]: Dependency Package[rsyslog-gnutls] has failures: true
Warning: /Stage[main]/Base::Remote_syslog/Rsyslog::Conf[remote_syslog]/File[/etc/rsyslog.d/30-remote-syslog.conf]: Skipping because of failed dependencies
Notice: /Stage[main]/Rsyslog/Service[rsyslog]: Dependency Package[rsyslog-gnutls] has failures: true
Warning: /Stage[main]/Rsyslog/Service[rsyslog]: Skipping because of failed dependencies

Event Timeline

This was on deployment-sca02 but the list of deployment-prep instances failing puppet grew suddenly, so I expect a few of these were affected by this problem, and I imagine it's not just our project:

deployment-cache-upload04.deployment-prep.eqiad.wmflabs
deployment-cpjobqueue.deployment-prep.eqiad.wmflabs
deployment-memc04.deployment-prep.eqiad.wmflabs
deployment-memc05.deployment-prep.eqiad.wmflabs
deployment-memc07.deployment-prep.eqiad.wmflabs
deployment-ms-be03.deployment-prep.eqiad.wmflabs
deployment-ms-be04.deployment-prep.eqiad.wmflabs
deployment-sca02.deployment-prep.eqiad.wmflabs
deployment-sca04.deployment-prep.eqiad.wmflabs
deployment-sentry01.deployment-prep.eqiad.wmflabs

I encountered the same problem, and I think the problem lies elsewhere, specifically in the prerm script from the current rsyslog package:

Unpacking rsyslog-gnutls (8.1901.0-1~bpo8+wmf1) over (8.38.0-1~bpo8+wmf1) ...
Preparing to unpack .../rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb ...
Job for rsyslog.service canceled.
invoke-rc.d: initscript rsyslog, action "stop" failed.
dpkg: warning: subprocess old pre-removal script returned error exit status 1
dpkg: trying script from the new package instead ...
Job for rsyslog.service canceled.
invoke-rc.d: initscript rsyslog, action "stop" failed.
dpkg: error processing archive /var/cache/apt/archives/rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb (--unpack):
 subprocess new pre-removal script returned error exit status 1

Now, looking at the prerm script:

/var/lib/dpkg/info$ cat rsyslog.prerm 
#!/bin/sh

set -e

# Stop the socket on remove so rsyslog is not restarted via socket activation
if [ -d /run/systemd/system ] && [ "$1" = remove ] ; then
	systemctl stop syslog.socket || true
fi

# Automatically added by dh_installinit/11.3.5~bpo9+1
if [ -x "/etc/init.d/rsyslog" ] && [ "$1" = remove ]; then
	invoke-rc.d rsyslog stop || exit 1
fi
# End automatically added section

It seems clear to me that the script is failing to stop rsyslog via invoke-rc.d. I cannot reproduce the problem when running the prerm script from the cli on a server with the old version of the package, so I'm not sure what's causing the issue.

Running the steps from the prerm on a jessie system with 8.38 works fine:

jmm@alsafi:~$ sudo systemctl stop syslog.socket
jmm@alsafi:~$ sudo invoke-rc.d rsyslog stop
jmm@alsafi:~$ sudo systemctl status rsyslog
• rsyslog.service - System Logging Service
  Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled)
Active: inactive (dead) since Tue 2019-04-02 09:24:03 UTC; 11s ago

When rsyslog and rsyslog-gnutls are upgraded, the prerm for rsyslog fails, but rsyslog-gnutls continues to get upgraded. After that, rsyslog-gnutls is installed in 8.1901, but rsyslog in 8.38. This is also specific to jessie, I tried the same in stretch and it works fine. This could be related to the older systemd version in jessie and it's interaction with update-rc.d. I see two options:

  • Rebuild the jessie package with a modified maintainer script which replaces the update-rc.d call with a simple "systemctl stop rsyslog.service" and test whether that makes a difference
  • Use a bigger hammer and brute-deploy the current packages by simply running debdeploy twice given that jessie is deprecated and 8.38->8.1901 is a one time migration

I looked into this as well and the rsyslog 8.38 prerm doesn't have the extra [ "$1" = remove ] test guarding invoke-rc.d, which 8.1901 does have instead, hence why rsyslog stop is called on prerm + upgrade but shouldn't. Adding the extra guard to the prerm makes the upgrade work as expected:

filippo@deployment-memc07:~$ cat /var/lib/dpkg/info/rsyslog.prerm
#!/bin/sh

set -e

# Stop the socket on remove so rsyslog is not restarted via socket activation
if [ -d /run/systemd/system ] && [ "$1" = remove ] ; then
	systemctl stop syslog.socket || true
fi

# Automatically added by dh_installinit
if [ -x "/etc/init.d/rsyslog" ] && [ "$1" = remove ] ; then
	invoke-rc.d rsyslog stop || exit $?
fi
# End automatically added section

filippo@deployment-memc07:~$ dpkg -l rsyslog
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                      Version           Architecture      Description
+++-=========================-=================-=================-========================================================
ii  rsyslog                   8.38.0-1~bpo8+wmf amd64             reliable system and kernel logging daemon
filippo@deployment-memc07:~$ sudo apt -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  apt-file debdeploy-common libconfig-file-perl liblist-moreutils-perl libnfsidmap2 libopts25 libpgm-5.1-0
  libregexp-assemble-perl libsodium13 libzmq3 linux-image-3.16.0-4-amd64 linux-image-3.16.0-5-amd64
  linux-image-3.16.0-6-amd64 linux-image-3.16.0-7-amd64 linux-image-4.9.0-0.bpo.3-amd64 linux-image-4.9.0-0.bpo.4-amd64
  linux-image-4.9.0-0.bpo.5-amd64 linux-image-4.9.0-0.bpo.7-amd64 python-gdbm python-m2crypto python-zmq
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  rsyslog
Suggested packages:
  rsyslog-mysql rsyslog-pgsql rsyslog-doc rsyslog-gssapi
The following packages will be upgraded:
  rsyslog
1 upgraded, 0 newly installed, 0 to remove and 27 not upgraded.
4 not fully installed or removed.
Need to get 0 B/653 kB of archives.
After this operation, 6,144 B disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 100176 files and directories currently installed.)
Preparing to unpack .../rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb ...
Unpacking rsyslog (8.1901.0-1~bpo8+wmf1) over (8.38.0-1~bpo8+wmf1) ...
Processing triggers for systemd (215-17+deb8u11) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up rsyslog (8.1901.0-1~bpo8+wmf1) ...
Installing new version of config file /etc/rsyslog.conf ...
Setting up rsyslog-gnutls (8.1901.0-1~bpo8+wmf1) ...
Setting up rsyslog-kafka (8.1901.0-1~bpo8+wmf1) ...
Setting up libruby2.1:amd64 (2.1.5-2+deb8u7) ...
Setting up ruby2.1 (2.1.5-2+deb8u7) ...
Processing triggers for libc-bin (2.19-18+deb8u10) ...
filippo@deployment-memc07:~$

I'm for the brute-deploy option, namely: systemctl stop syslog.socket rsyslog ; apt -f install for the affected labs instances above

I looked into this as well and the rsyslog 8.38 prerm doesn't have the extra [ "$1" = remove ] test guarding invoke-rc.d, which 8.1901 does have instead, hence why rsyslog stop is called on prerm + upgrade but shouldn't. Adding the extra guard to the prerm makes the upgrade work as expected:

Makes sense, probably caused by missing changes in older debhelper

I'm for the brute-deploy option, namely: systemctl stop syslog.socket rsyslog ; apt -f install for the affected labs instances above

Is this still an issue, wouldn't unattended-upgrades re-attempt the update, so it should have worked the second time?

I looked into this as well and the rsyslog 8.38 prerm doesn't have the extra [ "$1" = remove ] test guarding invoke-rc.d, which 8.1901 does have instead, hence why rsyslog stop is called on prerm + upgrade but shouldn't. Adding the extra guard to the prerm makes the upgrade work as expected:

Makes sense, probably caused by missing changes in older debhelper

I'm for the brute-deploy option, namely: systemctl stop syslog.socket rsyslog ; apt -f install for the affected labs instances above

Is this still an issue, wouldn't unattended-upgrades re-attempt the update, so it should have worked the second time?

Ah yes, indeed apt -f install a second time by itself does fix the issue!

This was on deployment-sca02 but the list of deployment-prep instances failing puppet grew suddenly, so I expect a few of these were affected by this problem, and I imagine it's not just our project:

FYI I've fixed the instances you mentioned by running apt -f install twice as a bandaid, although indeed there will be other instances with the same problem.

This was on deployment-sca02 but the list of deployment-prep instances failing puppet grew suddenly, so I expect a few of these were affected by this problem, and I imagine it's not just our project:

FYI I've fixed the instances you mentioned by running apt -f install twice as a bandaid, although indeed there will be other instances with the same problem.

Thanks. Do we know how many production hosts are affected, if any?

Thanks. Do we know how many production hosts are affected, if any?

Affected in the sense that they are currently in a broken state; none as we don't use unattended-upgrades.
Affected in the sense that they are currently running 8.38.0-1 and still need the non-standard update path to 8.1901.0-1: 149

fgiunchedi triaged this task as Medium priority.Apr 9 2019, 8:38 AM
Krenair renamed this task from Some jessie instances upset about rsyslog package to jessie rsyslog upgrade problems.Apr 17 2019, 8:04 PM
Krenair removed a project: Puppet.
fgiunchedi renamed this task from jessie rsyslog upgrade problems to Upgrade jessie hosts to rsyslog 8.1901.0-1.Apr 24 2019, 2:17 PM

Mentioned in SAL (#wikimedia-operations) [2019-04-24T14:28:31Z] <godog> being rollout rsyslog 8.1901.0-1 to jessie hosts - T219764

Mentioned in SAL (#wikimedia-operations) [2019-04-29T12:46:04Z] <godog> resume rollout rsyslog 8.1901.0-1 to jessie hosts - T219764

fgiunchedi claimed this task.

I've finished rolling out 8.1901.0-1 to jessie hosts in production, @Krenair looks like this can be resolved? I'll handle the stretch rsyslog upgrade separatedly.