Page MenuHomePhabricator

Renew certs for mcrouter on all mw appservers
Closed, ResolvedPublic

Description

MCROUTERCERTVERIFICATION WARNING - days_left_to_client_cert_expiration is 60 (outside range @~:60)

Acked the warning for all appservers etc.. in icinga :)

Previous task: https://phabricator.wikimedia.org/T248093

Event Timeline

@RLazarus in https://phabricator.wikimedia.org/T248093#6076630 you mentioned committing a script for automating cert renewal, and I see it indeed. Renewing the certs should amount to just running the script, correct?

I am aiming to at least test TLS on memcached T271967, hoping to roll it out next month. If this works out, we will not be needing mcrouter certs. We have 60 days ahead of us, I think it can be done, providing that testing is successful.

JMeybohm triaged this task as Medium priority.Mar 1 2021, 11:57 AM

I am aiming to at least test TLS on memcached T271967, hoping to roll it out next month. If this works out, we will not be needing mcrouter certs. We have 60 days ahead of us, I think it can be done, providing that testing is successful.

In that hypothesis, we'd also get rid of mcrouter proxies completely, which is an added advantage. 60 days seems a bit tight though, but if you feel confident you can do it, sure, why not. I'd just like not to have to do this on a tight schedule.

I am aiming to at least test TLS on memcached T271967, hoping to roll it out next month. If this works out, we will not be needing mcrouter certs. We have 60 days ahead of us, I think it can be done, providing that testing is successful.

In that hypothesis, we'd also get rid of mcrouter proxies completely, which is an added advantage. 60 days seems a bit tight though, but if you feel confident you can do it, sure, why not. I'd just like not to have to do this on a tight schedule.

Oh yeah, that was implied :). The memcached upgrade to a version that supports TLS is halfway done, and we are already using memcached + TLS in idp. Even if we don't make it on time, we always have the option to do what we did last time.

@RLazarus in https://phabricator.wikimedia.org/T248093#6076630 you mentioned committing a script for automating cert renewal, and I see it indeed. Renewing the certs should amount to just running the script, correct?

Yep -- modules/cergen/files/renew_mcrouter_certs.py is the code, and it's /usr/local/bin/renew_mcrouter_certs on the pupetmaster.

Can't promise that the script hasn't rotted in the last year, but if not, it should really be as simple as "ssh to the puppetmaster and run it."

Halfway ping just to remember that a month is left before the certs expire :)

Joe raised the priority of this task from Medium to High.Apr 13 2021, 5:59 AM

I don't realistically see it possible to switch memcached to TLS in the remaining time before we need to renew the certificates, hence raising priority. It will be raised to UBN! in a couple days.

@RLazarus do you mind running the script one last time? I hope to get TLS working this quarter, but sadly I didn't manage to do it towards the end of Q3 as I originally planned.

Mentioned in SAL (#wikimedia-operations) [2021-04-14T14:13:07Z] <rzl> mcrouter cert renewal complete, puppet re-enabled T276029

Done -- just re-enabled puppet, so they'll get picked up over the next 30m.