Page MenuHomePhabricator

Puppet broken on deployment-ms-be0[34] with evaluation error in swift module
Closed, ResolvedPublic

Description

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Left match operand must result in a String value. Got a Boolean. at /etc/puppet/modules/swift/manifests/init_device.pp:2:9 at /etc/puppet/modules/role/manifests/swift/storage.pp:23 on node deployment-ms-be03.deployment-prep.eqiad.wmflabs
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Left match operand must result in a String value. Got a Boolean. at /etc/puppet/modules/swift/manifests/init_device.pp:2:9 at /etc/puppet/modules/role/manifests/swift/storage.pp:23 on node deployment-ms-be04.deployment-prep.eqiad.wmflabs

Event Timeline

Krenair created this task.Jan 5 2018, 1:51 AM
Krenair triaged this task as Normal priority.
Paladox added a subscriber: Gehel.Jan 5 2018, 11:08 AM

I guess we can apply this https://github.com/wikimedia/mediawiki-vagrant/commit/ac6d19df598c75d97b635b026763ae7fd96f5970 fix at /etc/puppet/modules/swift/manifests/init_device.pp: (similar fix)

hashar added a subscriber: fgiunchedi.

That sounds like a puppet 4 issue. The mystery is why is Swift on production not affected by the type mismatch.

I think the code was always broken and it actually wanted to do this:

diff --git a/modules/swift/manifests/init_device.pp b/modules/swift/manifests/init_device.pp
index 69ab253328..cb8cb20250 100644
--- a/modules/swift/manifests/init_device.pp
+++ b/modules/swift/manifests/init_device.pp
@@ -1,5 +1,5 @@
 define swift::init_device($partition_nr='1') {
-    if (! $title =~ /^[hvs]d[a-z]+$/) {
+    if (!($title =~ /^[hvs]d[a-z]+$/)) {
         fail("Invalid name ${title} for swift::init_device")
     }

Of course, this now fails due to deployment-prep's swift storage drives being named lv-a.
No idea why it wasn't failing in prod though.

Looks like the reason is we have an old broken version of https://gerrit.wikimedia.org/r/#/c/361648/7 cherry-picked

Change 402758 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] swift: Fix checks on drive/filesystem titles to allow for labs ones

https://gerrit.wikimedia.org/r/402758

Krenair claimed this task.Jan 8 2018, 12:57 AM

Found a syntax problem in the latest version of it too (jenkins confirmed), fixed that, and added a dependent patch that allows the names in use on deployment-prep

hashar removed a subscriber: hashar.Apr 2 2018, 12:52 PM
Linux deployment-ms-be04 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+                                                                                                                                  2 (2018-01-04) x86_64
Debian GNU/Linux 8.8 (jessie)
deployment-ms-be04 is a statsite server (statsite)
deployment-ms-be04 is a swift storage brick (swift::storage)
The last Puppet run was at Thu Mar 22 12:05:11 UTC 2018 (25954 minutes ago).
maurelio@deployment-ms-be04:~$ sudo puppet agent -tv
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, unable to init lv-a for swift at /etc/puppet/modules/swift/manifests/init_device.pp:3:9 at /etc/puppet/modules/role/manifests/swift/storage.pp:23 on node deployment-ms-be04.deployment-prep.eqiad.wmflabs
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Krenair added a comment.EditedMay 27 2018, 8:39 PM

Rebased the patches and re-cherry-picked, puppet is running again

EddieGP closed this task as Resolved.May 28 2018, 5:37 AM
Krenair reopened this task as Open.Jun 2 2018, 2:22 PM

cherry-picked, not merged

Change 402758 merged by Dzahn:
[operations/puppet@production] swift: Fix checks on drive/filesystem titles to allow for labs ones

https://gerrit.wikimedia.org/r/402758

Krenair closed this task as Resolved.Nov 30 2018, 10:33 PM