Page MenuHomePhabricator

Rolling restart of etcd to pick up the renewed CA public certificate.
Closed, ResolvedPublic

Description

We need to do a rolling restart of etcd before the old CA cert expires in a few months.

Codfw can be probably exempted for now given it will be reimaged in the next quarter.

Event Timeline

Joe created this task.Nov 5 2019, 9:50 AM
Joe updated the task description. (Show Details)

Change 548241 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppet_ca: update puppet ca with a new certificate valid for 10 years

https://gerrit.wikimedia.org/r/548241

Dzahn added a subscriber: Dzahn.Nov 5 2019, 6:38 PM
Joe added a comment.Nov 7 2019, 9:47 AM

Correction:

  1. We will need to restart etcd in eqiad as the CA is used in etcd::v3 for peer-to-peer communications
  2. We will not need to restart etcd in codfw as it's currently on etcd v2 and thus is not using certs for server-to-server communications.

All of this can happen asynchronously from the actual CA update as the service will keep working even if the CA file has changed.

jbond moved this task from Unsorted 💣 to Watching 👀 on the User-jbond board.Nov 7 2019, 5:32 PM

Change 548241 merged by Jbond:
[operations/puppet@production] puppet_ca: update puppet ca with a new certificate valid for 10 years

https://gerrit.wikimedia.org/r/548241

jbond added a comment.Dec 4 2019, 3:24 PM

The new CA has been distributed now so this can be started

Joe added a comment.Dec 10 2019, 10:52 AM

Good news is we only need to do a rolling restart in eqiad, not in codfw, where we still don't use the ca for peer connections

Mentioned in SAL (#wikimedia-operations) [2019-12-10T10:55:37Z] <_joe_> restarting etcd on conf1004 T237362

Change 556647 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] etcd::client::globalconfig: add ca_cert

https://gerrit.wikimedia.org/r/556647

Have been trying to document some of the SSL findings and it appears to me that conftool client is currently preforming no SSL validations. Specificity

The conftool client does not configure a CA bundle to use. As conftool is using python4-etcd which in turn uses urllib3 it effectively means that validation is disabled

I have created a CR to update this

Joe closed this task as Resolved.Dec 13 2019, 10:02 AM
Joe reassigned this task from Joe to RLazarus.