Page MenuHomePhabricator

Rolling restart of etcd to pick up the renewed CA public certificate.
Closed, ResolvedPublic

Description

We need to do a rolling restart of etcd before the old CA cert expires in a few months.

Codfw can be probably exempted for now given it will be reimaged in the next quarter.

Event Timeline

Joe updated the task description. (Show Details)

Change 548241 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppet_ca: update puppet ca with a new certificate valid for 10 years

https://gerrit.wikimedia.org/r/548241

Correction:

  1. We will need to restart etcd in eqiad as the CA is used in etcd::v3 for peer-to-peer communications
  2. We will not need to restart etcd in codfw as it's currently on etcd v2 and thus is not using certs for server-to-server communications.

All of this can happen asynchronously from the actual CA update as the service will keep working even if the CA file has changed.

Change 548241 merged by Jbond:
[operations/puppet@production] puppet_ca: update puppet ca with a new certificate valid for 10 years

https://gerrit.wikimedia.org/r/548241

The new CA has been distributed now so this can be started

Good news is we only need to do a rolling restart in eqiad, not in codfw, where we still don't use the ca for peer connections

Change 556647 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] etcd::client::globalconfig: add ca_cert

https://gerrit.wikimedia.org/r/556647

Have been trying to document some of the SSL findings and it appears to me that conftool client is currently preforming no SSL validations. Specificity

The conftool client does not configure a CA bundle to use. As conftool is using python4-etcd which in turn uses urllib3 it effectively means that validation is disabled

I have created a CR to update this

Joe reassigned this task from Joe to RLazarus.

Change 556647 abandoned by Jbond:
[operations/puppet@production] etcd::client::globalconfig: add ca_cert

Reason:
supperseeded

https://gerrit.wikimedia.org/r/556647