Page MenuHomePhabricator

Puppet failing on deployment-logstash03.deployment-prep.eqiad.wmflabs
Closed, DeclinedPublic

Description

Write the description below

Got the alert email:

Date: Fri, 16 Apr 2021 08:15:06 +0000
From: root <root@deployment-logstash03.deployment-prep.eqiad.wmflabs>
To: dcaro@wikimedia.org
Subject: [Cloud VPS alert][deployment-prep] Puppet failure on deployment-logstash03.deployment-prep.eqiad.wmflabs (172.16.1.184)


Puppet is failing to run on the "deployment-logstash03.deployment-prep.eqiad.wmflabs (172.16.1.184)" instance in project
deployment-prep in Wikimedia Cloud VPS.

Working Puppet runs are needed to maintain instance security and logins.
As long as Puppet continues to fail, this system is in danger of becoming
unreachable.

You are receiving this email because you are listed as member for the
project that contains this instance.  Please take steps to repair
this instance or contact a Cloud VPS admin for assistance.

For further support, visit #wikimedia-cloud on freenode or
<https://wikitech.wikimedia.org>

Documenting the runbook for the alert on:
https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/Runbooks/Cloud_VPS_alert_Puppet_failure_on

Event Timeline

dcaro triaged this task as Medium priority.Apr 16 2021, 9:35 AM
dcaro created this task.
Majavah added a subscriber: Majavah.
taavi@deployment-logstash03:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Function lookup() did not find a value for the name 'profile::logstash::apifeatureusage::curator_actions' (file: /etc/puppet/modules/profile/manifests/logstash/apifeatureusage.pp, line: 7) on node deployment-logstash03.deployment-prep.eqiad.wmflabs
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Change 680266 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] wmsc: add role to the hiera hierarchy

https://gerrit.wikimedia.org/r/680266

Some previous conversations on trying to add roles to the cloud hiera lookup:

Change 680329 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] WIP wmcs.enc: Add role of the machine

https://gerrit.wikimedia.org/r/680329

This machine will be decommissioned shortly (T283013)

Change 680266 abandoned by David Caro:

[operations/puppet@production] wmsc: add role to the hiera hierarchy

Reason:

No longer relevant

https://gerrit.wikimedia.org/r/680266

Change 680329 abandoned by David Caro:

[operations/puppet@production] WIP wmcs.enc: Add role of the machine

Reason:

No longer relevant

https://gerrit.wikimedia.org/r/680329