Maniphest T194926

puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	EddieGP
	May 18 2018, 11:01 AM

Description

eddie@deployment-db03:~$ sudo puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-db03.deployment-prep.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1526641196'
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install ldap-utils' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  libldap-common liblz4-1 libnfsidmap2 libpgm-5.1-0 libsodium13 libstdc++5
  libuuid-perl libzmq3 linux-image-3.16.0-5-amd64
  linux-image-4.9.0-0.bpo.3-amd64 python-dateutil python-m2crypto
  python-msgpack python-semver python-zmq ruby-hiera
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  libldap-2.4-2
Suggested packages:
  libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-heimdal
Recommended packages:
  libsasl2-modules
The following packages will be DOWNGRADED:
  ldap-utils libldap-2.4-2
0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 88 not upgraded.
Need to get 413 kB of archives.
After this operation, 94.2 kB of additional disk space will be used.
E: There are problems and -y was used without --force-yes
Error: /Stage[main]/Ldap::Client::Openldap/Package[ldap-utils]/ensure: change from 2.4.44+dfsg-5+deb9u1~bpo8+1 to 2.4.41+dfsg-1+wmf1 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install ldap-utils' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  libldap-common liblz4-1 libnfsidmap2 libpgm-5.1-0 libsodium13 libstdc++5
  libuuid-perl libzmq3 linux-image-3.16.0-5-amd64
  linux-image-4.9.0-0.bpo.3-amd64 python-dateutil python-m2crypto
  python-msgpack python-semver python-zmq ruby-hiera
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  libldap-2.4-2
Suggested packages:
  libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-heimdal
Recommended packages:
  libsasl2-modules
The following packages will be DOWNGRADED:
  ldap-utils libldap-2.4-2
0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 88 not upgraded.
Need to get 413 kB of archives.
After this operation, 94.2 kB of additional disk space will be used.
E: There are problems and -y was used without --force-yes
Notice: Applied catalog in 7.92 seconds

Reason for puppet failures on:

deployment-db03
deployment-elastic07
deployment-ircd
deployment-logstash2
deployment-mathoid
deployment-ms-fe02
deployment-prometheus01
deployment-tin

Related Objects

Mentioned In: T194928: deployment-elastic07 has puppet errors

Event Timeline

EddieGP created this task.May 18 2018, 11:01 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 18 2018, 11:01 AM

EddieGP moved this task from To Triage to Puppet errors on the Beta-Cluster-Infrastructure board.May 18 2018, 11:02 AM

EddieGP mentioned this in T194928: deployment-elastic07 has puppet errors.May 18 2018, 11:08 AM

EddieGP merged a task: T194928: deployment-elastic07 has puppet errors.May 18 2018, 11:13 AM

EddieGP renamed this task from deployment-db03 has puppet errors to puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts.May 18 2018, 11:22 AM

EddieGP updated the task description. (Show Details)

EddieGP updated the task description. (Show Details)May 18 2018, 4:35 PM

EddieGP merged a task: T195267: deployment-prep broken puppet, ldap-utils downgrade fail.May 22 2018, 7:10 PM

EddieGP added subscribers: Dzahn, zhuyifei1999, aborrero, Paladox.

Thanks for the pointer @EddieGP, I was able to fix ores-lb-02 with the following command:

sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils

Mentioned in SAL (#wikimedia-releng) [2018-05-22T20:37:59Z] <eddiegp> eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils; done T194926

Mentioned in SAL (#wikimedia-releng) [2018-05-22T20:42:14Z] <eddiegp> eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo puppet agent -tv; done (for T194926, finished without errors on any of the hosts)

20:29 <shinken-wm> RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0]
20:47 <shinken-wm> RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0]
20:50 <shinken-wm> RECOVERY - Puppet errors on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0]
20:50 <shinken-wm> RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0]
20:51 <shinken-wm> RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0]
20:52 <shinken-wm> RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0]
20:53 <shinken-wm> RECOVERY - Puppet errors on deployment-prometheus01 is OK: OK: Less than 1.00% above the threshold [0.0]
20:53 <shinken-wm> RECOVERY - Puppet errors on deployment-ms-fe02 is OK: OK: Less than 1.00% above the threshold [0.0]

I guess I'm done for today. Might want to have another look whether there's still some affected hosts out there, but not doing that right now.

Dzahn awarded a token.May 22 2018, 10:28 PM

Thanks for your work on this @EddieGP

Krenair assigned this task to EddieGP.May 25 2018, 2:14 AM

All the hosts that are failing puppet in deployment-prep (as seen on shinken.wmflabs.org) look familiar and seem to have their own tasks. So I think we can consider this done.

• Vvjjkkii renamed this task from puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts to pscaaaaaaa.Jul 1 2018, 1:09 AM

• Vvjjkkii reopened this task as Open.

• Vvjjkkii removed EddieGP as the assignee of this task.

• Vvjjkkii triaged this task as High priority.

• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).

• Vvjjkkii updated the task description. (Show Details)

• Vvjjkkii removed a subscriber: Aklapper.

CommunityTechBot renamed this task from pscaaaaaaa to puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts.Jul 2 2018, 4:03 PM

CommunityTechBot closed this task as Resolved.

CommunityTechBot assigned this task to EddieGP.

CommunityTechBot raised the priority of this task from High to Needs Triage.

CommunityTechBot updated the task description. (Show Details)

CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.

CommunityTechBot added a subscriber: Aklapper.

puppet fails over libldap/ldap-utils downgrade on various deployment-prep hostsClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts
Closed, ResolvedPublic
Actions