puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts
Closed, ResolvedPublic

Description

eddie@deployment-db03:~$ sudo puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-db03.deployment-prep.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1526641196'
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install ldap-utils' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  libldap-common liblz4-1 libnfsidmap2 libpgm-5.1-0 libsodium13 libstdc++5
  libuuid-perl libzmq3 linux-image-3.16.0-5-amd64
  linux-image-4.9.0-0.bpo.3-amd64 python-dateutil python-m2crypto
  python-msgpack python-semver python-zmq ruby-hiera
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  libldap-2.4-2
Suggested packages:
  libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-heimdal
Recommended packages:
  libsasl2-modules
The following packages will be DOWNGRADED:
  ldap-utils libldap-2.4-2
0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 88 not upgraded.
Need to get 413 kB of archives.
After this operation, 94.2 kB of additional disk space will be used.
E: There are problems and -y was used without --force-yes
Error: /Stage[main]/Ldap::Client::Openldap/Package[ldap-utils]/ensure: change from 2.4.44+dfsg-5+deb9u1~bpo8+1 to 2.4.41+dfsg-1+wmf1 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install ldap-utils' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  libldap-common liblz4-1 libnfsidmap2 libpgm-5.1-0 libsodium13 libstdc++5
  libuuid-perl libzmq3 linux-image-3.16.0-5-amd64
  linux-image-4.9.0-0.bpo.3-amd64 python-dateutil python-m2crypto
  python-msgpack python-semver python-zmq ruby-hiera
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  libldap-2.4-2
Suggested packages:
  libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-heimdal
Recommended packages:
  libsasl2-modules
The following packages will be DOWNGRADED:
  ldap-utils libldap-2.4-2
0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 88 not upgraded.
Need to get 413 kB of archives.
After this operation, 94.2 kB of additional disk space will be used.
E: There are problems and -y was used without --force-yes
Notice: Applied catalog in 7.92 seconds

Reason for puppet failures on:

  • deployment-db03
  • deployment-elastic07
  • deployment-ircd
  • deployment-logstash2
  • deployment-mathoid
  • deployment-ms-fe02
  • deployment-prometheus01
  • deployment-tin
EddieGP created this task.May 18 2018, 11:01 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 18 2018, 11:01 AM
EddieGP renamed this task from deployment-db03 has puppet errors to puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts.May 18 2018, 11:22 AM
EddieGP updated the task description. (Show Details)
EddieGP updated the task description. (Show Details)May 18 2018, 4:35 PM
awight added a subscriber: awight.May 22 2018, 8:09 PM

Thanks for the pointer @EddieGP, I was able to fix ores-lb-02 with the following command:

sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils

Mentioned in SAL (#wikimedia-releng) [2018-05-22T20:37:59Z] <eddiegp> eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils; done T194926

Mentioned in SAL (#wikimedia-releng) [2018-05-22T20:42:14Z] <eddiegp> eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo puppet agent -tv; done (for T194926, finished without errors on any of the hosts)

20:29 <shinken-wm> RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0]
20:47 <shinken-wm> RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0]
20:50 <shinken-wm> RECOVERY - Puppet errors on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0]
20:50 <shinken-wm> RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0]
20:51 <shinken-wm> RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0]
20:52 <shinken-wm> RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0]
20:53 <shinken-wm> RECOVERY - Puppet errors on deployment-prometheus01 is OK: OK: Less than 1.00% above the threshold [0.0]
20:53 <shinken-wm> RECOVERY - Puppet errors on deployment-ms-fe02 is OK: OK: Less than 1.00% above the threshold [0.0]

I guess I'm done for today. Might want to have another look whether there's still some affected hosts out there, but not doing that right now.

Thanks for your work on this @EddieGP

Krenair assigned this task to EddieGP.May 25 2018, 2:14 AM
EddieGP closed this task as Resolved.May 25 2018, 3:53 PM

All the hosts that are failing puppet in deployment-prep (as seen on shinken.wmflabs.org) look familiar and seem to have their own tasks. So I think we can consider this done.

Vvjjkkii renamed this task from puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts to pscaaaaaaa.Jul 1 2018, 1:09 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii triaged this task as High priority.
Vvjjkkii removed EddieGP as the assignee of this task.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot assigned this task to EddieGP.
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot closed this task as Resolved.
CommunityTechBot renamed this task from pscaaaaaaa to puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts.
CommunityTechBot added a subscriber: Aklapper.