Page MenuHomePhabricator

puppet broken on deployment-restbase01 and deployment-restbase02
Closed, ResolvedPublic

Description

Both deployment-restbase01 and deployment-restbase02 on beta cluster have falling puppet:

hashar@deployment-restbase01:~$ sudo puppet agent -tv
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-restbase01.deployment-prep.eqiad.wmflabs
Info: Applying configuration version '1437643463'
Error: Execution of '/usr/bin/salt-call --log-level=quiet --out=json deploy.fetch cassandra/metrics-collector' returned 1: Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 123, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 422, in run
    caller.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 227, in run
    ret = self.call()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 129, in call
    ret['return'] = func(*args, **kwargs)
  File "/var/cache/salt/minion/extmods/modules/deploy.py", line 485, in fetch
    config = get_config(repo)
  File "/var/cache/salt/minion/extmods/modules/deploy.py", line 85, in get_config
    config = config[repo]
KeyError: 'cassandra/metrics-collector'
Error: /Stage[main]/Cassandra::Metrics/Package[cassandra/metrics-collector]/ensure: change from purged to present failed: Execution of '/usr/bin/salt-call --log-level=quiet --out=json deploy.fetch cassandra/metrics-collector' returned 1: Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 123, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 422, in run
    caller.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 227, in run
    ret = self.call()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 129, in call
    ret['return'] = func(*args, **kwargs)
  File "/var/cache/salt/minion/extmods/modules/deploy.py", line 485, in fetch
    config = get_config(repo)
  File "/var/cache/salt/minion/extmods/modules/deploy.py", line 85, in get_config
    config = config[repo]
KeyError: 'cassandra/metrics-collector'
Notice: /Stage[main]/Cassandra::Metrics/File[/usr/local/lib/cassandra-metrics-collector/cassandra-metrics-collector.jar]: Dependency Package[cassandra/metrics-collector] has failures: true
Warning: /Stage[main]/Cassandra::Metrics/File[/usr/local/lib/cassandra-metrics-collector/cassandra-metrics-collector.jar]: Skipping because of failed dependencies
Notice: /Stage[main]/Cassandra::Metrics/Cron[cassandra-metrics-collector]: Dependency Package[cassandra/metrics-collector] has failures: true
Warning: /Stage[main]/Cassandra::Metrics/Cron[cassandra-metrics-collector]: Skipping because of failed dependencies
Notice: /Stage[main]/Restbase/Package[restbase/deploy]/ensure: ensure changed 'purged' to 'present'
Notice: Finished catalog run in 29.73 seconds
hashar@deployment-restbase01:~$

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a project: RESTBase.

The cassandra/metrics-collector repo is missing on deployment-bastion. I guess once You migrate the puppet master @hashar we'll be able to force a puppet run on deployment-bastion and have that up again.

Eevans claimed this task.
Eevans subscribed.

I added the deploy repo to: https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep (https://wikitech.wikimedia.org/w/index.php?title=Hiera%3ADeployment-prep&type=revision&diff=171641&oldid=171624), waited a bit for it to appear on deployment-bastion (updated by a scheduled job, I assume?), and then forced a deploy ala https://wikitech.wikimedia.org/wiki/Trebuchet#Deploy_the_repo_via_tin.eqiad.wmnet.

Everything is happy now.