Page MenuHomePhabricator

No Puppet resources found on instance deployment-memc13 on project deployment-prep
Closed, ResolvedPublic

Description

Common information

  • summary: No Puppet resources found on instance deployment-memc13 on project deployment-prep
  • alertname: PuppetAgentNoResources
  • instance: deployment-memc13
  • job: node
  • project: deployment-prep
  • severity: warning

Firing alerts


  • summary: No Puppet resources found on instance deployment-memc13 on project deployment-prep
  • alertname: PuppetAgentNoResources
  • instance: deployment-memc13
  • job: node
  • project: deployment-prep
  • severity: warning
  • Source

Event Timeline

bd808@deployment-memc13:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Function lookup() did not find a value for the name 'profile::memcached::firewall_src_sets' (file: /srv/puppet_code/environments/production/modules/profile/manifests/memcached/instance.pp, line: 93) on node deployment-memc13.deployment-prep.eqiad1.wikimedia.cloud
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1152274. That patch adds profile::memcached::firewall_src_sets: ~ to hieradata/role/common/memcached.yaml which is picked up in production use, but not in Beta due to differences between prod and WMCS Puppet config and the role() operator which is only used in production.

Mentioned in SAL (#wikimedia-releng) [2025-06-05T20:20:33Z] <bd808> Added profile::memcached::firewall_src_sets: ~ to deployment-memc prefix puppet (T396109)

bd808 claimed this task.
bd808 moved this task from To Triage to Puppet errors on the Beta-Cluster-Infrastructure board.

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/55cb264f4a0925e591e8f7e08548bfafbebfc9e1%5E%21/#F0

bd808@deployment-memc13:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-memc13.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(c42a3e2273) gitpuppet - deployment-prep: Add Apache vhost aliases for *.beta.wmcloud.org'
Notice: /Stage[main]/Apt/File[/etc/apt/sources.list]/ensure: removed (corrective)
Notice: /Stage[main]/Profile::Rsyslog::Kafka_shipper/File[/etc/rsyslog.lookup.d/lookup_table_output.json]/content:
--- /etc/rsyslog.lookup.d/lookup_table_output.json      2025-05-16 15:07:49.443014464 +0000
+++ /tmp/puppet-file20250605-922043-uxbrfr      2025-06-05 20:20:11.073681001 +0000
@@ -113,6 +113,10 @@
     {"index" : "heat-engine", "value" : "kafka local" },
     {"index" : "magnum-api", "value" : "kafka local" },
     {"index" : "magnum-conductor", "value" : "kafka local" },
+    {"index" : "octavia-api", "value" : "kafka local" },
+    {"index" : "octavia-health-manager", "value" : "kafka local" },
+    {"index" : "octavia-housekeeping", "value" : "kafka local" },
+    {"index" : "octavia-worker", "value" : "kafka local" },
     {"index" : "ceph-osd", "value" : "kafka local ecs_170" },
     {"index" : "ceph-mon", "value" : "kafka local ecs_170" },
     {"index" : "ceph-mgr", "value" : "kafka local ecs_170" },

Notice: /Stage[main]/Profile::Rsyslog::Kafka_shipper/File[/etc/rsyslog.lookup.d/lookup_table_output.json]/content: content changed '{sha256}3f6b1f5bd4a9a230ec7b9ba6664e22086d85da40a05b1a78965183458b618cac' to '{sha256}bdaaa74f5072c5dfcda75f343055ef0050224b3e7e018a2e009a8b267a4fbdb6'
Info: /Stage[main]/Profile::Rsyslog::Kafka_shipper/File[/etc/rsyslog.lookup.d/lookup_table_output.json]: Scheduling refresh of Service[rsyslog]
Notice: /Stage[main]/Rsyslog/Service[rsyslog]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 8.50 seconds