Page MenuHomePhabricator
Create Task
Maniphest T234256

Broken puppet on traffic-upload-stretch.traffic.eqiad.wmflabs and traffic-text-stretch.traffic.eqiad.wmflabs
Closed, ResolvedPublic
Actions

Description

Each of these VMs has had broken puppet for quite a while:

root@traffic-text-stretch:~# puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Could not find data item profile::trafficserver::tls::inbound_tls_settings in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/trafficserver/tls.pp:10:63 on node traffic-text-stretch.traffic.eqiad.wmflabs
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
root@traffic-upload-stretch:~# puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Class[Profile::Trafficserver::Tls]: parameter 'inbound_tls_settings' expects a value for key 'session_ticket_enable' at /etc/puppet/modules/role/manifests/cache/upload.pp:13:5 on node traffic-upload-stretch.traffic.eqiad.wmflabs
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

They look like easy fixes but often when VMS are left to rot like this they're just awaiting deletion which would also be fine with me :)

Details

SubjectRepoBranchLines +/-
ATS: set tls::parent_rules for labsoperations/puppetproduction+13 -0
ATS: set inbound_tls_settings for labsoperations/puppetproduction+30 -0
Customize query in gerrit

Event Timeline

Andrew created this task.Sep 30 2019, 6:42 PM
Restricted Application added a project: SRE. · View Herald TranscriptSep 30 2019, 6:42 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
jijiki triaged this task as Medium priority.Oct 14 2019, 2:34 PM
• ema moved this task from Backlog to Caching on the Traffic board.Oct 14 2019, 5:41 PM
gerritbot added a comment.Oct 14 2019, 5:58 PM

Change 542994 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: set inbound_tls_settings for labs

https://gerrit.wikimedia.org/r/542994

gerritbot added a project: Patch-For-Review.Oct 14 2019, 5:58 PM
gerritbot added a comment.Oct 14 2019, 6:00 PM

Change 542994 merged by Ema:
[operations/puppet@production] ATS: set inbound_tls_settings for labs

https://gerrit.wikimedia.org/r/542994

• ema subscribed.Oct 14 2019, 6:08 PM

Thanks for filing this task @Andrew! I did add profile::trafficserver::tls::inbound_tls_settings to hieradata/labs.yaml, but still puppet on traffic-text-stretch.traffic.eqiad.wmflabs fails complaining about that key. Do you have any idea why this is the case?

Maintenance_bot removed a project: Patch-For-Review.Oct 14 2019, 6:10 PM
Andrew added a comment.Oct 14 2019, 11:55 PM

Error while evaluating a Function Call, Could not find data item profile::trafficserver::tls::parent_rules

If you don't want to set these things cloud-wide you can use the Horizon interface to set them per-VM or per-project

• ema added a comment.Oct 15 2019, 5:52 PM
In T234256#5573825, @Andrew wrote:

Error while evaluating a Function Call, Could not find data item profile::trafficserver::tls::parent_rules

If you don't want to set these things cloud-wide you can use the Horizon interface to set them per-VM or per-project

I do want to set the value cloud-wide, but it does not seem to work for some reason which I was hoping you could help me understand. :)

Andrew added a comment.Oct 15 2019, 5:59 PM

@ema, the missing key in my paste is a different key from the one you mentioned in your comment. You were talking about ::inbound_tls_settings whereas the current error is on ::parent_rules which is still not set as far as I can tell.

• ema added a comment.Oct 15 2019, 6:08 PM
In T234256#5576473, @Andrew wrote:

@ema, the missing key in my paste is a different key from the one you mentioned in your comment. You were talking about ::inbound_tls_settings whereas the current error is on ::parent_rules which is still not set as far as I can tell.

Doh, thank you so much!

gerritbot added a comment.Oct 15 2019, 6:09 PM

Change 543186 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: set tls::parent_rules for labs

https://gerrit.wikimedia.org/r/543186

gerritbot added a project: Patch-For-Review.Oct 15 2019, 6:09 PM
gerritbot added a comment.Oct 15 2019, 6:19 PM

Change 543186 merged by Ema:
[operations/puppet@production] ATS: set tls::parent_rules for labs

https://gerrit.wikimedia.org/r/543186

• ema closed this task as Resolved.Oct 15 2019, 6:38 PM
• ema claimed this task.
• ema added a subscriber: Vgutierrez.

Puppet works again on both instances after adding tls::parent_rules, tls::inbound_tls_settings, profile::trafficserver::tls::unified_acme_chief, and profile::trafficserver::tls::unified_certs (CC @Vgutierrez)

Maintenance_bot removed a project: Patch-For-Review.Oct 15 2019, 7:18 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL