Page MenuHomePhabricator

wmflabs `Error: Could not retrieve catalog; skipping run` on deployment-push-notifications01
Closed, ResolvedPublic

Description

puppet agent -tv
2020-09-08 12:38:02.178010 WARN  puppetlabs.facter - locale environment variables were bad; continuing with LANG=C LC_ALL=C
Info: Caching certificate for deployment-push-notifications01.deployment-prep.eqiad.wmflabs
Info: Caching certificate_revocation_list for ca
Info: Caching certificate for deployment-push-notifications01.deployment-prep.eqiad.wmflabs
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Server Error: Could not retrieve facts for deployment-push-notifications01.deployment-prep.eqiad.wmflabs: Failed to find facts from PuppetDB at puppet:8140: Failed to execute '/pdb/query/v4/nodes/deployment-push-notifications01.deployment-prep.eqiad.wmflabs/facts' on at least 1 of the following 'server_urls': https://deployment-puppetdb03.deployment-prep.eqiad.wmflabs
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed to execute '/pdb/cmd/v1?checksum=b9680d38e5b69569d0eef815d3a5a2d2c5128cc0&version=5&certname=deployment-push-notifications01.deployment-prep.eqiad.wmflabs&command=replace_facts&producer-timestamp=2020-09-08T12:38:05.950Z' on at least 1 of the following 'server_urls': https://deployment-puppetdb03.deployment-prep.eqiad.wmflabs
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Event Timeline

Mholloway assigned this task to MSantos.
Mholloway added a subscriber: Mholloway.

Looks like deployment-puppetdb03 was unavailable again because of T248041: puppetdb on deployment-puppetdb03 keeps getting OOMKilled. Puppet is running again now, after a puppetdb restart. Hopefully the subsequent tweaks to the JVM options for puppetdb will keep it that way.