Page MenuHomePhabricator

tools-checker-03 is failing puppet runs
Closed, ResolvedPublic

Description

tools-checker-03 is failing puppet related to python packages, possibly related to https://gerrit.wikimedia.org/r/c/operations/puppet/+/622874

Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install uwsgi' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  apt-file libexporter-tiny-perl liblist-moreutils-perl libpcsclite1
  libregexp-assemble-perl python-gdbm python-pykube
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  uwsgi-core uwsgi-plugin-python uwsgi-plugin-python3
  uwsgi-plugin-rack-ruby2.3
Suggested packages:
  uwsgi-plugins-all uwsgi-extra python-uwsgidecorators python3-uwsgidecorators
The following packages will be DOWNGRADED:
  uwsgi uwsgi-core uwsgi-plugin-python uwsgi-plugin-python3
  uwsgi-plugin-rack-ruby2.3
0 upgraded, 0 newly installed, 5 downgraded, 0 to remove and 85 not upgraded.
1 not fully installed or removed.
E: Packages were downgraded and -y was used without --allow-downgrades.
Error: /Stage[main]/Uwsgi/Package[uwsgi]/ensure: change from absent to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install uwsgi' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  apt-file libexporter-tiny-perl liblist-moreutils-perl libpcsclite1
  libregexp-assemble-perl python-gdbm python-pykube
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  uwsgi-core uwsgi-plugin-python uwsgi-plugin-python3
  uwsgi-plugin-rack-ruby2.3
Suggested packages:
  uwsgi-plugins-all uwsgi-extra python-uwsgidecorators python3-uwsgidecorators
The following packages will be DOWNGRADED:
  uwsgi uwsgi-core uwsgi-plugin-python uwsgi-plugin-python3
  uwsgi-plugin-rack-ruby2.3
0 upgraded, 0 newly installed, 5 downgraded, 0 to remove and 85 not upgraded.
1 not fully installed or removed.
E: Packages were downgraded and -y was used without --allow-downgrades.

Event Timeline

bstorm@tools-checker-03:~$ apt policy uwsgi-plugin-python3
uwsgi-plugin-python3:
  Installed: 2.0.14+20161117-3+deb9u3
  Candidate: 2.0.14+20161117-3+deb9u2+wmf1
  Version table:
 *** 2.0.14+20161117-3+deb9u3 500
        500 http://security.debian.org stretch/updates/main amd64 Packages
        100 /var/lib/dpkg/status
     2.0.14+20161117-3+deb9u2+wmf1 1001
       1001 http://apt.wikimedia.org/wikimedia stretch-wikimedia/main amd64 Packages
     2.0.14+20161117-3+deb9u2 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages

So "downgrade" it might not quite be?

Mentioned in SAL (#wikimedia-cloud) [2020-08-31T16:51:34Z] <bstorm> running apt install uwsgi with --allow-downgrades to fix the puppet setup there T261677

Mentioned in SAL (#wikimedia-cloud) [2020-08-31T16:52:14Z] <bstorm> apt install uwsgi was run on tools-checker-03 in the last log T261677

dpkg: warning: downgrading uwsgi-plugin-rack-ruby2.3 from 2.0.14+20161117-3+deb9u3 to 2.0.14+20161117-3+deb9u2+wmf1
(Reading database ... 61340 files and directories currently installed.)
Preparing to unpack .../uwsgi-plugin-rack-ruby2.3_2.0.14+20161117-3+deb9u2+wmf1_amd64.deb ...
Unpacking uwsgi-plugin-rack-ruby2.3 (2.0.14+20161117-3+deb9u2+wmf1) over (2.0.14+20161117-3+deb9u3) ...
dpkg: warning: downgrading uwsgi-plugin-python3 from 2.0.14+20161117-3+deb9u3 to 2.0.14+20161117-3+deb9u2+wmf1
Preparing to unpack .../uwsgi-plugin-python3_2.0.14+20161117-3+deb9u2+wmf1_amd64.deb ...
Unpacking uwsgi-plugin-python3 (2.0.14+20161117-3+deb9u2+wmf1) over (2.0.14+20161117-3+deb9u3) ...
dpkg: warning: downgrading uwsgi-plugin-python from 2.0.14+20161117-3+deb9u3 to 2.0.14+20161117-3+deb9u2+wmf1
Preparing to unpack .../uwsgi-plugin-python_2.0.14+20161117-3+deb9u2+wmf1_amd64.deb ...
Unpacking uwsgi-plugin-python (2.0.14+20161117-3+deb9u2+wmf1) over (2.0.14+20161117-3+deb9u3) ...
dpkg: warning: downgrading uwsgi from 2.0.14+20161117-3+deb9u3 to 2.0.14+20161117-3+deb9u2+wmf1
Preparing to unpack .../uwsgi_2.0.14+20161117-3+deb9u2+wmf1_amd64.deb ...
Unpacking uwsgi (2.0.14+20161117-3+deb9u2+wmf1) over (2.0.14+20161117-3+deb9u3) ...
dpkg: warning: downgrading uwsgi-core from 2.0.14+20161117-3+deb9u3 to 2.0.14+20161117-3+deb9u2+wmf1
Preparing to unpack .../uwsgi-core_2.0.14+20161117-3+deb9u2+wmf1_amd64.deb ...
Unpacking uwsgi-core (2.0.14+20161117-3+deb9u2+wmf1) over (2.0.14+20161117-3+deb9u3) ...

Unfortunately, it didn't fix puppet.
Puppet is failing with uwsgi is already the newest version (2.0.14+20161117-3+deb9u2+wmf1).

It seems this is unrelated to my puppet changes and is just an update that exists in one repo, but not in our mirror of backports.

Ok, I suspect this will be fixed by syncing the wikimedia repo with stretch-updates, but it's not cooperating much.

root@apt1001:~# reprepro checkupdate stretch-wikimedia
Error: unknown key '81E2E78B66FED89E'!
There have been errors!

@aborrero @MoritzMuehlenhoff @Dzahn Any thoughts? I don't mess with reprepro often, so there might be something I just did wrong trying to check updates or perhaps an error has occurred due to stretch deprecation.

In the case of this particular issue, I've realized it will not pass puppet because uwsgi cannot start either way (that's not how toolschecker works). So I have to convince APT that it is done with the install...

Fixed this for tools-checker-03 by running:

root@tools-checker-03:~# echo exit 101 > /usr/sbin/policy-rc.d
root@tools-checker-03:~# chmod +x /usr/sbin/policy-rc.d
root@tools-checker-03:~# apt-get install uwsgi
root@tools-checker-03:~# rm /usr/sbin/policy-rc.d

Ugly hack worked.
We'll need to do it again if we manage to sync up the repo 😛

Bstorm claimed this task.

Ok, I suspect this will be fixed by syncing the wikimedia repo with stretch-updates, but it's not cooperating much.

root@apt1001:~# reprepro checkupdate stretch-wikimedia
Error: unknown key '81E2E78B66FED89E'!
There have been errors!

@aborrero @MoritzMuehlenhoff @Dzahn Any thoughts? I don't mess with reprepro often, so there might be something I just did wrong trying to check updates or perhaps an error has occurred due to stretch deprecation.

That's the old Matomo/Piwik import key, it's outdated and probably noone noticed since piwik is on buster these days. I'll make a patch to remove it.