Page MenuHomePhabricator

Prevents labs/private to disappear from integration and deployment-prep puppet master
Closed, ResolvedPublic

Description

Some patch in puppet caused labs/private to be set as absent:

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/602327/3/modules/puppetmaster/manifests/gitclone.pp,unified

Luckily deployment-puppetmaster04.deployment-prep.eqiad.wmflabs and integration-puppetmaster-02.integration.eqiad.wmflabs had merge conflicts preventing them from automagically applying the faulty puppet.git patch

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 4 2020, 1:30 PM

On both, I have made a copy to /var/lib/git/labs/private-back

hashar triaged this task as Unbreak Now! priority.Jun 4 2020, 1:43 PM

I have:

  • disabled puppet agent on both puppet master
  • rebased /var/lib/git/operations/puppet on both puppet master.

Confirmed the faulty commit got reverted ;)

Enabled puppet again and ran it again.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptJun 4 2020, 1:43 PM

The one on beta fails because of a missing etcd::autogen_pwd_seed. I have made a dummy
puppet patch in hieradata/cloud/eqiad1/deployment-prep/common.yaml

hashar closed this task as Resolved.Jun 4 2020, 1:54 PM
hashar claimed this task.

Looks good to me now. We have been lucky to have both puppet master blocked on a merge conflict!