Puppet failing on deployment-logstash03.deployment-prep.eqiad.wmflabs
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	dcaro
	Apr 16 2021, 9:35 AM

Description

Write the description below

Got the alert email:

Date: Fri, 16 Apr 2021 08:15:06 +0000
From: root <root@deployment-logstash03.deployment-prep.eqiad.wmflabs>
To: dcaro@wikimedia.org
Subject: [Cloud VPS alert][deployment-prep] Puppet failure on deployment-logstash03.deployment-prep.eqiad.wmflabs (172.16.1.184)

Puppet is failing to run on the "deployment-logstash03.deployment-prep.eqiad.wmflabs (172.16.1.184)" instance in project
deployment-prep in Wikimedia Cloud VPS.

Working Puppet runs are needed to maintain instance security and logins.
As long as Puppet continues to fail, this system is in danger of becoming
unreachable.

You are receiving this email because you are listed as member for the
project that contains this instance.  Please take steps to repair
this instance or contact a Cloud VPS admin for assistance.

For further support, visit #wikimedia-cloud on freenode or
<https://wikitech.wikimedia.org>

Documenting the runbook for the alert on:
https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/Runbooks/Cloud_VPS_alert_Puppet_failure_on

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	WIP wmcs.enc: Add role of the machine	operations/puppet	production	+16 -1
	wmsc: add role to the hiera hierarchy	operations/puppet	production	+4 -0

Customize query in gerrit

Related Objects

Mentioned In: T196034: Define scap::sources in a way that is shared between prod and beta
Mentioned Here: T283013: Migrate beta cluster to ELK7
T120165: Implement role based hiera lookups for labs
T127771: Make Cloud VPS hiera role aware
T211029: Sort out only one ideal hiera mechanism for Cloud VPS

Event Timeline

dcaro created this task.Apr 16 2021, 9:35 AM

dcaro triaged this task as Medium priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 16 2021, 9:35 AM

taavi@deployment-logstash03:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Function lookup() did not find a value for the name 'profile::logstash::apifeatureusage::curator_actions' (file: /etc/puppet/modules/profile/manifests/logstash/apifeatureusage.pp, line: 7) on node deployment-logstash03.deployment-prep.eqiad.wmflabs
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Change 680266 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] wmsc: add role to the hiera hierarchy

https://gerrit.wikimedia.org/r/680266

gerritbot added a project: Patch-For-Review.Apr 16 2021, 9:39 AM

Some previous conversations on trying to add roles to the cloud hiera lookup:

https://phabricator.wikimedia.org/T120165 -> two years ago, got denied in favor of horizon puppet prefixes
https://phabricator.wikimedia.org/T211029 -> three years ago, when it was decided to use version control for the cloud enc
https://phabricator.wikimedia.org/T127771 -> three years ago, a request to add role support, unblocked by horizon puppet prefixes