Page MenuHomePhabricator

No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep
Closed, ResolvedPublic

Description

Common information

  • summary: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep
  • alertname: PuppetAgentNoResources
  • instance: deployment-puppetserver-1
  • job: node
  • project: deployment-prep
  • severity: warning

Firing alerts


  • summary: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep
  • alertname: PuppetAgentNoResources
  • instance: deployment-puppetserver-1
  • job: node
  • project: deployment-prep
  • severity: warning
  • Source

Event Timeline

bd808 subscribed.
bd808@deployment-puppetserver-1:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Function lookup() did not find a value for the name 'profile::conftool::hiddenparma::api_tokens' (file: /srv/puppet_code/environments/production/modules/profile/manifests/puppetserver/volatile.pp, line: 12) on node deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/c487a1b4328b87cd66fc8ceda446b1f5ce0f1c41%5E%21/#F0

diff --git a/deployment-prep/deployment-puppetserver.yaml b/deployment-prep/deployment-puppetserver.yaml
index b84188b..9fc81ee 100644
--- a/deployment-prep/deployment-puppetserver.yaml
+++ b/deployment-prep/deployment-puppetserver.yaml

@@ -1,5 +1,7 @@
 profile::configmaster::server_aliases: []
 profile::configmaster::server_name: config-master.wikimedia.beta.wmflabs.org
+profile::conftool::hiddenparma::api_tokens:
+  root: ''
 profile::puppet::agent::force_puppet7: true
 profile::puppetmaster::common::puppetdb_host: deployment-puppetdb05.deployment-prep.eqiad1.wikimedia.cloud
 profile::puppetmaster::common::puppetdb_hosts:
bd808@deployment-puppetserver-1:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: Class[Profile::Puppetserver] is already declared; cannot redeclare (file: /srv/puppet_code/environments/production/modules/profile/manifests/puppetserver/wmcs.pp, line: 6) (file: /srv/puppet_code/environments/production/modules/profile/manifests/puppetserver/wmcs.pp, line: 6, column: 5) on node deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
bd808 claimed this task.
bd808 triaged this task as Medium priority.
bd808@deployment-puppetserver-1:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(8b2efd270d) git-sync-upstream - [BETA HACK] Changes to profile::puppetserver::volatile'
Notice: /Stage[main]/External_clouds_vendors/File[/usr/local/bin/fetch-external-clouds-vendors-nets]/content:
--- /usr/local/bin/fetch-external-clouds-vendors-nets   2025-05-28 06:14:05.374235555 +0000
+++ /tmp/puppet-file20250625-1713189-50nn81     2025-06-25 20:24:12.681952784 +0000
@@ -4,6 +4,9 @@
 import csv
 import json
 import logging
+import os
+import subprocess
+import tempfile
 import time
 import re
 from datetime import datetime
@@ -12,10 +15,8 @@
 from pathlib import Path
 from typing import List, Set

-from conftool.extensions.reqconfig import (
-    api,
-    RequestctlError,
-)
+import yaml
+
 from lxml import html
 from netaddr import IPNetwork, cidr_merge
 from requests import Session
@@ -219,6 +220,25 @@
     return {str(net) for net in merged}


+def requestctl_apply(slug, data, api_token) -> subprocess.CompletedProcess:
+    """Run requestctl apply on the requested ipblock"""
+    try:
+        tmp = tempfile.NamedTemporaryFile(mode="w+", delete=False)
+        yaml.dump(data, tmp)
+        cmd = ['/usr/bin/requestctl', 'apply', 'ipblock', slug, '-f', tmp.name]
+        result = subprocess.run(
+            cmd,
+            check=True,
+            text=True,
+            capture_output=True,
+            env={'REQUESTCTL_API_TOKEN': api_token}
+        )
+        logging.debug("Output of running requestctl apply: %s", result.stdout)
+        return result
+    finally:
+        os.unlink(tmp.name)
+
+
 def get_args() -> Namespace:
     """Parse arguments"""
     parser = ArgumentParser(description=__doc__)
@@ -255,6 +275,12 @@
     """main entry point"""
     args = get_args()
     logging.basicConfig(level=get_log_level(args.verbose))
+    if args.conftool:
+        # We need to have the API token defined, or this won't work
+        api_token = os.environ.get('REQUESTCTL_API_TOKEN')
+        if api_token is None:
+            raise ValueError("Cannot write to conftool without an api token.")
+
     data = dict()
     runtime_error = False

@@ -366,19 +392,24 @@
                 runtime_error = True

     if args.conftool:
-        req_api = api.RequestctlApi(api.client(config="/etc/conftool/config.yaml"))
+        api_token = os.environ.get('REQUESTCTL_API_TOKEN', None)
+        if api_token is None:
+            raise ValueError("you must define the env variable REQUESTCTL_API_TOKEN"
+                             " in order to export data to requestctl.")
+
         for ipblock_type, ipblocks in data.items():
             for ipblock_name, cidrs in ipblocks.items():
                 slug = f"{ipblock_type}/{ipblock_name.lower()}"
+                to_update = {
+                    "cidrs": cidrs,
+                    "comment": f"Automatically generated IPs for {ipblock_name}",
+                }
+
                 try:
                     logging.info("Updating ipblock@%s", slug)
-                    entity = req_api.get("ipblock", slug)
-                    to_update = {
-                        "cidrs": cidrs,
-                        "comment": f"Automatically generated IPs for {ipblock_name}",
-                    }
-                    req_api.write(entity, to_update)
-                except RequestctlError as error:
+                    requestctl_apply(slug, to_update, api_token)
+                    logging.info("ipblock imported correctly")
+                except subprocess.CalledProcessError as error:
                     logging.error("Error updating %s: %s", slug, error)
                     runtime_error = True


Notice: /Stage[main]/External_clouds_vendors/File[/usr/local/bin/fetch-external-clouds-vendors-nets]/content: content changed '{sha256}fc2052451af5ff273b9e8cc29ff9fd0d1d5f4e75b09a6af1d7f8f507214312c9' to '{sha256}40529aee91cf13daea078c4aee58a57485afe0e77237cce90c134fd6f1d4e7a2'
Notice: /Stage[main]/External_clouds_vendors/Systemd::Timer::Job[dump_cloud_ip_ranges]/Systemd::Unit[dump_cloud_ip_ranges.service]/File[/lib/systemd/system/dump_cloud_ip_ranges.service]/content:
--- /lib/systemd/system/dump_cloud_ip_ranges.service    2024-11-11 08:14:21.325607723 +0000
+++ /tmp/puppet-file20250625-1713189-cuwcd9     2025-06-25 20:24:15.029962769 +0000
@@ -5,5 +5,6 @@
 [Service]
 Type=oneshot
 User=root
+Environment="REQUESTCTL_API_TOKEN="
 SyslogIdentifier=fetch-external-clouds-vendors-nets
 ExecStart=/usr/local/bin/fetch-external-clouds-vendors-nets  -vvv /srv/puppet_fileserver/volatile/external_cloud_vendors/public_clouds.json

Notice: /Stage[main]/External_clouds_vendors/Systemd::Timer::Job[dump_cloud_ip_ranges]/Systemd::Unit[dump_cloud_ip_ranges.service]/File[/lib/systemd/system/dump_cloud_ip_ranges.service]/content: content changed '{sha256}b2abda8ed1d018f01ee4aca7bed13ad78f4b34eab7fc66a00bd70baf281a8a28' to '{sha256}f7e7ad17567e52d83b4d43d7bb536b60cb549600533edab782cea57508f7ef3a'
Info: /Stage[main]/External_clouds_vendors/Systemd::Timer::Job[dump_cloud_ip_ranges]/Systemd::Unit[dump_cloud_ip_ranges.service]/File[/lib/systemd/system/dump_cloud_ip_ranges.service]: Scheduling refresh of Exec[systemd daemon-reload for dump_cloud_ip_ranges.service (dump_cloud_ip_ranges.service)]
Notice: /Stage[main]/External_clouds_vendors/Systemd::Timer::Job[dump_cloud_ip_ranges]/Systemd::Unit[dump_cloud_ip_ranges.service]/Exec[systemd daemon-reload for dump_cloud_ip_ranges.service (dump_cloud_ip_ranges.service)]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 12.44 seconds