(Beta cluster) Running logspam-watch on deployment-mwlog01 gives repeated `Use of uninitialized value $host` errors
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	TheresNoTime
	Aug 16 2022, 9:28 PM

Description

I'm pretty sure logspam-watch has worked on deployment-mwlog01 before....

Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 37890.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 37925.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 37939.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 37975.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38010.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38024.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38038.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38073.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38087.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38122.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38157.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38171.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38206.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38241.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38255.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38290.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38304.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38339.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38374.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38388.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38402.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38437.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38472.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38486.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38521.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38556.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38570.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38605.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38640.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38675.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38710.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38724.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38759.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38773.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38808.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38822.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38857.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38892.
Use of uninitialized value $host in pattern match (m//) at /usr/local/bin/logspam line 185, <$logstream> line 38906.
``

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		jbond	T315350 Beta cluster Error: 502, Next Hop Connection Failed
		Resolved		Zabe	T315379 (Beta cluster) Running logspam-watch on deployment-mwlog01 gives repeated `Use of uninitialized value $host` errors

Event Timeline

TheresNoTime created this task.Aug 16 2022, 9:28 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 16 2022, 9:28 PM

https://gerrit.wikimedia.org/r/c/operations/puppet/+/822453 (from last week) updates logspam to handle the new timestamp format. I notice that deployment-mwlog01 doesn't have the updated version. Is puppet running there regularly?

In T315379#8159573, @dancy wrote:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/822453 (from last week) updates logspam to handle the new timestamp format. I notice that deployment-mwlog01 doesn't have the updated version. Is puppet running there regularly?

samtar@deployment-mwlog01:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for deployment-mwlog01.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(bd50c4f725a) root - scap: introduce bootstrapping mechanism specific to deployment hosts'
Notice: The LDAP client stack for this host is: sssd/sudo
Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: sssd/sudo'
Notice: Applied catalog in 19.61 seconds

doesn't seem like it pulled any changes?

Is puppet running there regularly?

Yes

The last Puppet run was at Tue Aug 16 20:38:24 UTC 2022 (55 minutes ago).

brennen subscribed.Aug 16 2022, 9:40 PM

In T315379#8159618, @Zabe wrote:

Is puppet running there regularly?

Yes

The last Puppet run was at Tue Aug 16 20:38:24 UTC 2022 (55 minutes ago).

I'm not sure if this has anything to do with this, so apologies if not — although puppet is running regularly, deployment-puppetmaster04 appears to be very behind:

samtar@deployment-puppetmaster04:/var/lib/git/operations/puppet$ git log
commit bd50c4f725ae7fb6325710abd06f5f44fd4fa791 (HEAD -> production)
Author: jnuche <jnuche@wikimedia.org>
Date:   Fri Aug 5 15:49:47 2022 +0200

    scap: introduce bootstrapping mechanism specific to deployment hosts

    Previously, the installation of an initial Scap version on
    deployment hosts was being triggered manually

    Change-Id: I916aff3b8a408462508f95b74105cb600af1ea9e

and my understanding (which could definitely be wrong) is that deployment-puppetmaster04 should be its own puppet client? If so,

samtar@deployment-puppetmaster04:~$ sudo run-puppet-agent
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate revoked): [certificate revoked for /CN=deployment-puppetmaster04.deployment-prep.eqiad.wmflabs]
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate revoked): [certificate revoked for /CN=deployment-puppetmaster04.deployment-prep.eqiad.wmflabs]
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate revoked): [certificate revoked for /CN=deployment-puppetmaster04.deployment-prep.eqiad.wmflabs]
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate revoked): [certificate revoked for /CN=deployment-puppetmaster04.deployment-prep.eqiad.wmflabs]
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate revoked): [certificate revoked for /CN=deployment-puppetmaster04.deployment-prep.eqiad.wmflabs]
Info: Loading facts
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate revoked): [certificate revoked for /CN=deployment-puppetmaster04.deployment-prep.eqiad.wmflabs]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate revoked): [certificate revoked for /CN=deployment-puppetmaster04.deployment-prep.eqiad.wmflabs]

The Puppet repo on deployment-puppetmaster04:/var/lib/git/operations/puppet is in MERGING state. There's an unresolved conflict in modules/profile/manifests/etcd/v3.pp. The conflict is between the upstream change I04aa7729e and a local patch, Iecfc26a94, which has been cherry-picked locally for the past year but never merged upstream.

git status and git diff here: P32410

TheresNoTime mentioned this in T315350: Beta cluster Error: 502, Next Hop Connection Failed.Aug 16 2022, 11:03 PM

TheresNoTime added a parent task: T315350: Beta cluster Error: 502, Next Hop Connection Failed.