Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install rsyslog-gnutls' returned 100: Reading package lists... Building dependency tree... Reading state information... rsyslog-gnutls is already the newest version. You might want to run 'apt-get -f install' to correct these: The following packages have unmet dependencies: rsyslog-gnutls : Depends: rsyslog (= 8.1901.0-1~bpo8+wmf1) but 8.38.0-1~bpo8+wmf1 is to be installed rsyslog-kafka : Depends: rsyslog (= 8.1901.0-1~bpo8+wmf1) but 8.38.0-1~bpo8+wmf1 is to be installed E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution). Error: /Stage[main]/Packages::Rsyslog_gnutls/Package[rsyslog-gnutls]/ensure: change from absent to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install rsyslog-gnutls' returned 100: Reading package lists... Building dependency tree... Reading state information... rsyslog-gnutls is already the newest version. You might want to run 'apt-get -f install' to correct these: The following packages have unmet dependencies: rsyslog-gnutls : Depends: rsyslog (= 8.1901.0-1~bpo8+wmf1) but 8.38.0-1~bpo8+wmf1 is to be installed rsyslog-kafka : Depends: rsyslog (= 8.1901.0-1~bpo8+wmf1) but 8.38.0-1~bpo8+wmf1 is to be installed E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution). Notice: /Stage[main]/Base::Remote_syslog/Rsyslog::Conf[remote_syslog]/File[/etc/rsyslog.d/30-remote-syslog.conf]: Dependency Package[rsyslog-gnutls] has failures: true Warning: /Stage[main]/Base::Remote_syslog/Rsyslog::Conf[remote_syslog]/File[/etc/rsyslog.d/30-remote-syslog.conf]: Skipping because of failed dependencies Notice: /Stage[main]/Rsyslog/Service[rsyslog]: Dependency Package[rsyslog-gnutls] has failures: true Warning: /Stage[main]/Rsyslog/Service[rsyslog]: Skipping because of failed dependencies
Description
Related Objects
Event Timeline
This was on deployment-sca02 but the list of deployment-prep instances failing puppet grew suddenly, so I expect a few of these were affected by this problem, and I imagine it's not just our project:
deployment-cache-upload04.deployment-prep.eqiad.wmflabs
deployment-cpjobqueue.deployment-prep.eqiad.wmflabs
deployment-memc04.deployment-prep.eqiad.wmflabs
deployment-memc05.deployment-prep.eqiad.wmflabs
deployment-memc07.deployment-prep.eqiad.wmflabs
deployment-ms-be03.deployment-prep.eqiad.wmflabs
deployment-ms-be04.deployment-prep.eqiad.wmflabs
deployment-sca02.deployment-prep.eqiad.wmflabs
deployment-sca04.deployment-prep.eqiad.wmflabs
deployment-sentry01.deployment-prep.eqiad.wmflabs
I encountered the same problem, and I think the problem lies elsewhere, specifically in the prerm script from the current rsyslog package:
Unpacking rsyslog-gnutls (8.1901.0-1~bpo8+wmf1) over (8.38.0-1~bpo8+wmf1) ... Preparing to unpack .../rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb ... Job for rsyslog.service canceled. invoke-rc.d: initscript rsyslog, action "stop" failed. dpkg: warning: subprocess old pre-removal script returned error exit status 1 dpkg: trying script from the new package instead ... Job for rsyslog.service canceled. invoke-rc.d: initscript rsyslog, action "stop" failed. dpkg: error processing archive /var/cache/apt/archives/rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb (--unpack): subprocess new pre-removal script returned error exit status 1
Now, looking at the prerm script:
/var/lib/dpkg/info$ cat rsyslog.prerm #!/bin/sh set -e # Stop the socket on remove so rsyslog is not restarted via socket activation if [ -d /run/systemd/system ] && [ "$1" = remove ] ; then systemctl stop syslog.socket || true fi # Automatically added by dh_installinit/11.3.5~bpo9+1 if [ -x "/etc/init.d/rsyslog" ] && [ "$1" = remove ]; then invoke-rc.d rsyslog stop || exit 1 fi # End automatically added section
It seems clear to me that the script is failing to stop rsyslog via invoke-rc.d. I cannot reproduce the problem when running the prerm script from the cli on a server with the old version of the package, so I'm not sure what's causing the issue.
Running the steps from the prerm on a jessie system with 8.38 works fine:
jmm@alsafi:~$ sudo systemctl stop syslog.socket jmm@alsafi:~$ sudo invoke-rc.d rsyslog stop jmm@alsafi:~$ sudo systemctl status rsyslog • rsyslog.service - System Logging Service Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled) Active: inactive (dead) since Tue 2019-04-02 09:24:03 UTC; 11s ago
When rsyslog and rsyslog-gnutls are upgraded, the prerm for rsyslog fails, but rsyslog-gnutls continues to get upgraded. After that, rsyslog-gnutls is installed in 8.1901, but rsyslog in 8.38. This is also specific to jessie, I tried the same in stretch and it works fine. This could be related to the older systemd version in jessie and it's interaction with update-rc.d. I see two options:
- Rebuild the jessie package with a modified maintainer script which replaces the update-rc.d call with a simple "systemctl stop rsyslog.service" and test whether that makes a difference
- Use a bigger hammer and brute-deploy the current packages by simply running debdeploy twice given that jessie is deprecated and 8.38->8.1901 is a one time migration
I looked into this as well and the rsyslog 8.38 prerm doesn't have the extra [ "$1" = remove ] test guarding invoke-rc.d, which 8.1901 does have instead, hence why rsyslog stop is called on prerm + upgrade but shouldn't. Adding the extra guard to the prerm makes the upgrade work as expected:
filippo@deployment-memc07:~$ cat /var/lib/dpkg/info/rsyslog.prerm #!/bin/sh set -e # Stop the socket on remove so rsyslog is not restarted via socket activation if [ -d /run/systemd/system ] && [ "$1" = remove ] ; then systemctl stop syslog.socket || true fi # Automatically added by dh_installinit if [ -x "/etc/init.d/rsyslog" ] && [ "$1" = remove ] ; then invoke-rc.d rsyslog stop || exit $? fi # End automatically added section filippo@deployment-memc07:~$ dpkg -l rsyslog Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=========================-=================-=================-======================================================== ii rsyslog 8.38.0-1~bpo8+wmf amd64 reliable system and kernel logging daemon filippo@deployment-memc07:~$ sudo apt -f install Reading package lists... Done Building dependency tree Reading state information... Done Correcting dependencies... Done The following packages were automatically installed and are no longer required: apt-file debdeploy-common libconfig-file-perl liblist-moreutils-perl libnfsidmap2 libopts25 libpgm-5.1-0 libregexp-assemble-perl libsodium13 libzmq3 linux-image-3.16.0-4-amd64 linux-image-3.16.0-5-amd64 linux-image-3.16.0-6-amd64 linux-image-3.16.0-7-amd64 linux-image-4.9.0-0.bpo.3-amd64 linux-image-4.9.0-0.bpo.4-amd64 linux-image-4.9.0-0.bpo.5-amd64 linux-image-4.9.0-0.bpo.7-amd64 python-gdbm python-m2crypto python-zmq Use 'apt-get autoremove' to remove them. The following extra packages will be installed: rsyslog Suggested packages: rsyslog-mysql rsyslog-pgsql rsyslog-doc rsyslog-gssapi The following packages will be upgraded: rsyslog 1 upgraded, 0 newly installed, 0 to remove and 27 not upgraded. 4 not fully installed or removed. Need to get 0 B/653 kB of archives. After this operation, 6,144 B disk space will be freed. Do you want to continue? [Y/n] y (Reading database ... 100176 files and directories currently installed.) Preparing to unpack .../rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb ... Unpacking rsyslog (8.1901.0-1~bpo8+wmf1) over (8.38.0-1~bpo8+wmf1) ... Processing triggers for systemd (215-17+deb8u11) ... Processing triggers for man-db (2.7.0.2-5) ... Setting up rsyslog (8.1901.0-1~bpo8+wmf1) ... Installing new version of config file /etc/rsyslog.conf ... Setting up rsyslog-gnutls (8.1901.0-1~bpo8+wmf1) ... Setting up rsyslog-kafka (8.1901.0-1~bpo8+wmf1) ... Setting up libruby2.1:amd64 (2.1.5-2+deb8u7) ... Setting up ruby2.1 (2.1.5-2+deb8u7) ... Processing triggers for libc-bin (2.19-18+deb8u10) ... filippo@deployment-memc07:~$
I'm for the brute-deploy option, namely: systemctl stop syslog.socket rsyslog ; apt -f install for the affected labs instances above
Makes sense, probably caused by missing changes in older debhelper
I'm for the brute-deploy option, namely: systemctl stop syslog.socket rsyslog ; apt -f install for the affected labs instances above
Is this still an issue, wouldn't unattended-upgrades re-attempt the update, so it should have worked the second time?
FYI I've fixed the instances you mentioned by running apt -f install twice as a bandaid, although indeed there will be other instances with the same problem.
Affected in the sense that they are currently in a broken state; none as we don't use unattended-upgrades.
Affected in the sense that they are currently running 8.38.0-1 and still need the non-standard update path to 8.1901.0-1: 149
Mentioned in SAL (#wikimedia-operations) [2019-04-24T14:28:31Z] <godog> being rollout rsyslog 8.1901.0-1 to jessie hosts - T219764
Mentioned in SAL (#wikimedia-operations) [2019-04-29T12:46:04Z] <godog> resume rollout rsyslog 8.1901.0-1 to jessie hosts - T219764
I've finished rolling out 8.1901.0-1 to jessie hosts in production, @Krenair looks like this can be resolved? I'll handle the stretch rsyslog upgrade separatedly.