Page MenuHomePhabricator

Puppet failures on deployment-mx: can't find puppet://private/dkim/wikimedia.org-wiki-mail.key
Closed, InvalidPublic

Description

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: secret(): invalid secret dkim/wikimedia.org-wiki-mail.key at /etc/puppet/manifests/role/mail.pp:79 on node deployment-mx.deployment-prep.eqiad.wmflabs
Notice: Using cached catalog
Info: Applying configuration version '1438006858'
Error: /Stage[main]/Role::Labs::Instance/File[/usr/local/sbin/reboot-if-idmap]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///files/nfs/reboot-if-idmap
Error: /Stage[main]/Exim::Roled::Mail_relay/Exim4::Dkim[wikimedia.org]/File[/etc/exim4/dkim/wikimedia.org-wikimedia.key]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///private/dkim/wikimedia.org-wikimedia.key
Error: /Stage[main]/Exim::Roled/Exim4::Dkim[wiki-mail]/File[/etc/exim4/dkim/wikimedia.org-wiki-mail.key]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///private/dkim/wikimedia.org-wiki-mail.key
Notice: /Stage[main]/Exim4/Service[exim4]: Dependency File[/etc/exim4/dkim/wikimedia.org-wiki-mail.key] has failures: true
Notice: /Stage[main]/Exim4/Service[exim4]: Dependency File[/etc/exim4/dkim/wikimedia.org-wikimedia.key] has failures: true
Warning: /Stage[main]/Exim4/Service[exim4]: Skipping because of failed dependencies
Notice: /Stage[main]/Exim::Roled/File[/etc/exim4/defer_domains]: Dependency File[/etc/exim4/dkim/wikimedia.org-wiki-mail.key] has failures: true
Notice: /Stage[main]/Exim::Roled/File[/etc/exim4/defer_domains]: Dependency File[/etc/exim4/dkim/wikimedia.org-wikimedia.key] has failures: true
Warning: /Stage[main]/Exim::Roled/File[/etc/exim4/defer_domains]: Skipping because of failed dependencies
Notice: /Stage[main]/Exim::Roled/File[/etc/exim4/legacy_mailing_lists]: Dependency File[/etc/exim4/dkim/wikimedia.org-wiki-mail.key] has failures: true
Notice: /Stage[main]/Exim::Roled/File[/etc/exim4/legacy_mailing_lists]: Dependency File[/etc/exim4/dkim/wikimedia.org-wikimedia.key] has failures: true
Warning: /Stage[main]/Exim::Roled/File[/etc/exim4/legacy_mailing_lists]: Skipping because of failed dependencies
Notice: /Stage[main]/Exim::Roled/File[/etc/exim4/wikimedia_domains]: Dependency File[/etc/exim4/dkim/wikimedia.org-wiki-mail.key] has failures: true
Notice: /Stage[main]/Exim::Roled/File[/etc/exim4/wikimedia_domains]: Dependency File[/etc/exim4/dkim/wikimedia.org-wikimedia.key] has failures: true
Warning: /Stage[main]/Exim::Roled/File[/etc/exim4/wikimedia_domains]: Skipping because of failed dependencies
Notice: Finished catalog run in 15.34 seconds

Event Timeline

yuvipanda raised the priority of this task from to Needs Triage.
yuvipanda updated the task description. (Show Details)
yuvipanda subscribed.
greg triaged this task as Medium priority.Jan 29 2015, 5:32 PM
greg moved this task from To Triage to Next: Maintenance on the Beta-Cluster-Infrastructure board.
hashar renamed this task from Puppet failures on deployment-mx to Puppet failures on deployment-mx: can't find puppet://private/dkim/wikimedia.org-wiki-mail.key.Jun 8 2015, 9:08 AM
hashar set Security to None.

Puppet can't find puppet://private/dkim/wikimedia.org-wiki-mail.key . The instance has the puppet class role::mail::mx and I guess we need to add the private material on the puppetmaster.

On deployment-salt we have a clone of labs/private.git (which is public) under /var/lib/git/labs/private . There is already a local cherry pick there to provide deployment private keys.

(Ah! *lightbulb* I never thought about local commits to labs/private. That's a very useful way to deploy "secrets" in Labs projects while keeping in line with the patterns used in production.)

Still occurring. I have refreshed the puppet error output since we are now using secret().

Change 245139 had a related patch set uploaded (by Alex Monk):
Add blank mx secrets

https://gerrit.wikimedia.org/r/245139

Change 245139 abandoned by Alex Monk:
Add blank mx secrets

Reason:
puppet is no longer failing on that host

https://gerrit.wikimedia.org/r/245139

puppet is no longer failing on that host