Page MenuHomePhabricator

Fix maps puppet to make sure apt-get update runs after configuration change
Closed, ResolvedPublic

Description

We ran into some issues when updating maps1003 from jessie to stretch. There was a config(hieradata) change that make sure we use cassandra-2.2 version. Puppet fails with this error:

Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install cassandra=2.2.6-wmf5' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '2.2.6-wmf5' for 'cassandra' was not found
Error: /Stage[main]/Cassandra/Package[cassandra]/ensure: change from purged to 2.2.6-wmf5 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install cassandra=2.2.6-wmf5' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '2.2.6-wmf5' for 'cassandra' was not found

running apt-get update manually fixed this.
We should look at the puppet config to make sure apt-get update runs after any config change. or package version change.

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 17 2019, 6:52 PM
CDanis triaged this task as Normal priority.Jan 17 2019, 7:57 PM
EBjune moved this task from needs triage to Up Next on the Discovery-Search board.Jan 24 2019, 6:11 PM
debt moved this task from Up Next to Ops / SRE on the Discovery-Search board.Jan 29 2019, 6:30 PM

Looking at the code:

  • Apt repository is added by the main cassandra class. It is tagged as before => Package['cassandra'].
  • apt::repository has the proper notify => Exec['apt-get update'], which ** should trigger an update.
  • apt class has the definition of the exec, which does not seem to have any guard that would prevent it from running.

Conclusion: the code looks correct, but isn't working. Another pair of eyes would be welcomed!

Joe added a subscriber: Joe.Feb 13 2019, 10:32 AM

@Gehel I would say what's missing is a clear dependency between the installation of the cassandra package and the apt-get update.

We'll need to check if that creates some dependency cycle though.

Change 490303 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] cassandra: package should be installed after apt-get update

https://gerrit.wikimedia.org/r/490303

Change 490303 merged by Gehel:
[operations/puppet@production] cassandra: package should be installed after apt-get update

https://gerrit.wikimedia.org/r/490303

Gehel closed this task as Resolved.Feb 15 2019, 9:48 AM
Gehel claimed this task.