Page MenuHomePhabricator
Create Task
Maniphest T420993

Rotate discovery intermediate certificate (expires 2026-05-03)
Open, HighPublic
Actions

Description

I was chasing down certificate issues related to T419289, when I noticed that the dse-k8s issuer's intermediate certificate expires in early May:

gnutls-cli --print-cert opensearch-semantic-search.svc.eqiad.wmnet:30443  | grep -i expire
- subject `CN=opensearch-semantic-search.discovery.wmnet', issuer `CN=discovery,OU=SRE Foundations,O=Wikimedia Foundation\, Inc,L=San Francisco,C=US', serial 0x5672410c6fc4f152d8f03362685dddaf0b60997c, RSA key 2048 bits, signed using ECDSA-SHA512, activated `2026-03-13 06:20:00 UTC', expires `2026-04-10 06:20:00 UTC', pin-sha256="IXvDF4K9qvKm/oQEH191dDBC+Wav2jSNZGSKAV2ARLU="
- subject `CN=discovery,OU=SRE Foundations,O=Wikimedia Foundation\, Inc,L=San Francisco,C=US', issuer `CN=Wikimedia_Internal_Root_CA,OU=Cloud Services,O=Wikimedia Foundation\, Inc,L=San Francisco,ST=California,C=US', serial 0x715331115b69e7112b0e3c7f8c89ce15c51a4639, EC/ECDSA key 528 bits, signed using ECDSA-SHA512, activated `2021-05-04 13:54:00 UTC', expires `2026-05-03 13:54:00 UTC', pin-sha256="PbgfDlEHVB4Zw0a42zNqqnEQbcYF9jYp/dbT4eSdOb8="

Creating this ticket to:

  • Find and read relevant wikitech docs
  • Consult with IF/Service Ops if necessary
  • Rotate intermediate certificate before 2026-05-03 13:54:00 UTC
  • Change the alerting for certificate expiry to create a task

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
tlsproxy::envoy: Bump default now that services have movedoperations/puppetproduction+1 -1
role::pki: remove the 'discovery' intermediate's configoperations/puppetproduction+0 -56
role::crm: update postfix's cfssl pki intermediateoperations/puppetproduction+1 -0
wcqs: Migrate to new discovery intermediateoperations/puppetproduction+3 -0
peopleweb: Switch to discovery2026 intermediate for Envoyoperations/puppetproduction+1 -0
phabricator: Switch to discovery2026 intermediate for Envoyoperations/puppetproduction+1 -0
aphlict: Switch to discovery2026 intermediate for Envoyoperations/puppetproduction+2 -0
etherpad: Switch to discovery2026 intermediate for Envoyoperations/puppetproduction+2 -1
prometheus::pop: switch to discovery2026operations/puppetproduction+1 -0
prometheus: switch to discovery2026 for envoyoperations/puppetproduction+2 -1
titan: switch to discovery2026 for envoyoperations/puppetproduction+1 -0
role::crm: move the pki intermediate to discovery2026operations/puppetproduction+2 -0
profile::mediabackup: move to the discovery2026 pki intermediateoperations/puppetproduction+2 -2
puppetboard: Switch to discovery2026 intermediate for Envoyoperations/puppetproduction+1 -0
envoyproxy: trigger the envoy's config re-creation if deletedoperations/puppetproduction+1 -0
apt/staging: Switch to discovery2026 intermediate for Envoyoperations/puppetproduction+1 -0
profile::hcaptcha: move to the discovery2026 pki intermediateoperations/puppetproduction+2 -2
profile::opensearch::cirrus::server: move to a new pki intermediateoperations/puppetproduction+1 -1
debmonitor: Switch to discovery2026 intermediate for Envoyoperations/puppetproduction+2 -0
Switch remaining Ganeti clusters to discovery2026 intermediateoperations/puppetproduction+1 -3
admin_ng: Move all clusters to the pki discovery2026 intermediateoperations/deployment-chartsmaster+3 -6
wdqs: enable new discovery intermediate certificateoperations/puppetproduction+11 -0
profile::cache::purge: move to the new pki intermediateoperations/puppetproduction+1 -1
profile::etcd::tlsproxy: move to the new pki intermediateoperations/puppetproduction+1 -1
profile::docker_registry: move to the new pki intermediateoperations/puppetproduction+1 -1
profile::dragonfly: move to the new pki intermediateoperations/puppetproduction+1 -1
profile::pki::get_cert: add lookup() to the label argumentoperations/puppetproduction+26 -24
profile::puppetdb: move to the discovery2026 pki intermediateoperations/puppetproduction+1 -1
Move netbox and presto to the new PKI intermediateoperations/puppetproduction+2 -2
profile::tlsproxy::envoy: add condition to cfss base optionsoperations/puppetproduction+1 -0
Move ganeti-test to the 2026 PKI discovery intermediateoperations/puppetproduction+2 -0
ganeti: Make the cfssl label configurable via Hieraoperations/puppetproduction+2 -2
Move netbox, debmonitor and presto to the discovery2026 pki intermediateoperations/puppetproduction+6 -0
cfssl::cert: add require for csr when swapping intermediateoperations/puppetproduction+1 -0
wdqs-internal-scholarly: Provision certificates with new intermediateoperations/puppetproduction+3 -0
wdqs-test: use new intermediateoperations/puppetproduction+3 -0
admin_ng: move staging clusters to the pki discovery2026 intermediateoperations/deployment-chartsmaster+3 -0
profile::pki::get_cert: add lookup() to the label argumentoperations/puppetproduction+1 -1
profile::pki::get_cert: add lookup() to the label argumentoperations/puppetproduction+1 -1
profile::pki::intermediates: refresh discovery's public keyoperations/puppetproduction+12 -12
Add fake private secrets for discovery2026 PKI intermediatelabs/privatemaster+28 -0
role::pki::multiroot: configure discovery2026operations/puppetproduction+55 -0
profile::pki::intermediates: add discovery2026operations/puppetproduction+22 -0
profile::pki::root_ca: create a new discovery intermediateoperations/puppetproduction+1 -0
admin_ng: set cert-manager and cfssl-issuer replicas to 0 in stagingoperations/deployment-chartsmaster+7 -1
debmonitor: use chained TLS cert for server and clientoperations/puppetproduction+2 -2
profile::pki::intermediates: update debmonitor's public keyoperations/puppetproduction+12 -12
cfssl::cert: handle the rotation of the intermediate keypairoperations/puppetproduction+13 -0
discovery: Replace soon-to-be-expired intermediate certificateoperations/puppetproduction+12 -12
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
elukey added a comment.Mon, Apr 27, 7:26 AM
In T420993#11856386, @bking wrote:

@elukey was nice enough to let me help out today, here's what we found on wdqs2025 after updating its hiera values to use the new intermediate:

/etc/cfssl/csr/discovery__query-experimental_eqiad_wmnet_server.csr is removed but /etc/cfssl/csr/discovery2026__query-experimental_eqiad_wmnet_server.csr doesn't exist yet when cfssl gencert runs. I'm guessing this will be fixed early next week, but for now I'm pretty sure that just running Puppet twice would be good enough to refresh the certificate. I'll try another test host and post an update.

To keep archives happy - this should be fixed with https://gerrit.wikimedia.org/r/1277175

On a related topic, do you think it would be useful to add the intermediates to the wmf-certificates deb pkg? Currently it only contains the root CA.

Yeah this is by design, so we keep the clients as light as possible. The services run with the chained certificate that contains both leaf and intermediate, so on the client it is sufficient to deploy the Root CA only.

In T420993#11857129, @bking wrote:

Promised update: I added a few more hosts for testing. It turns out that running Puppet twice isn't enough on its own, and Envoy as a service proxy requires more config than Envoy as a TLS terminator. More details in P91477.

@elukey wdqs2026 and/or wdqs2027 have not been touched other than applying the puppet patches above. If want to use them to test next week, feel free. Just be sure to depool them.

Thanks for the tests! The bug should be fixed, I'll check those hosts today!

gerritbot added a comment.Mon, Apr 27, 9:25 AM

Change #1277438 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] envoyproxy: trigger the envoy's config re-creation if deleted

https://gerrit.wikimedia.org/r/1277438

gerritbot added a comment.Mon, Apr 27, 9:40 AM

Change #1275960 abandoned by Elukey:

[operations/puppet@production] Move netbox, debmonitor and presto to the discovery2026 pki intermediate

https://gerrit.wikimedia.org/r/1275960

gerritbot added a comment.Mon, Apr 27, 9:49 AM

Change #1277449 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Move netbox and presto to the new PKI intermediate

https://gerrit.wikimedia.org/r/1277449

gerritbot added a comment.Mon, Apr 27, 9:59 AM

Change #1277452 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::tlsproxy::envoy: add condition to cfss base options

https://gerrit.wikimedia.org/r/1277452

gerritbot added a comment.Mon, Apr 27, 10:26 AM

Change #1277458 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] ganeti: Make the cfssl label configurable via Hiera

https://gerrit.wikimedia.org/r/1277458

gerritbot added a comment.Mon, Apr 27, 10:44 AM

Change #1277458 merged by Muehlenhoff:

[operations/puppet@production] ganeti: Make the cfssl label configurable via Hiera

https://gerrit.wikimedia.org/r/1277458

gerritbot added a comment.Mon, Apr 27, 11:10 AM

Change #1277484 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Move ganeti-test to the 2026 PKI discovery intermediate

https://gerrit.wikimedia.org/r/1277484

gerritbot added a comment.Mon, Apr 27, 11:46 AM

Change #1277506 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::mediabackup: move to the discovery2026 pki intermediate

https://gerrit.wikimedia.org/r/1277506

gerritbot added a comment.Mon, Apr 27, 11:46 AM

Change #1277507 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::puppetdb: move to the discovery2026 pki intermediate

https://gerrit.wikimedia.org/r/1277507

gerritbot added a comment.Mon, Apr 27, 11:46 AM

Change #1277508 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::opensearch::cirrus::server: move to a new pki intermediate

https://gerrit.wikimedia.org/r/1277508

gerritbot added a comment.Mon, Apr 27, 11:46 AM

Change #1277509 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::hcaptcha: move to the discovery2026 pki intermediate

https://gerrit.wikimedia.org/r/1277509

gerritbot added a comment.Mon, Apr 27, 11:46 AM

Change #1277510 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::dragonfly: move to the new pki intermediate

https://gerrit.wikimedia.org/r/1277510

gerritbot added a comment.Mon, Apr 27, 11:46 AM

Change #1277511 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::docker_registry: move to the new pki intermediate

https://gerrit.wikimedia.org/r/1277511

gerritbot added a comment.Mon, Apr 27, 11:46 AM

Change #1277512 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::cache::purge: move to the new pki intermediate

https://gerrit.wikimedia.org/r/1277512

gerritbot added a comment.Mon, Apr 27, 11:46 AM

Change #1277513 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::etcd::tlsproxy: move to the new pki intermediate

https://gerrit.wikimedia.org/r/1277513

gerritbot added a comment.Mon, Apr 27, 12:05 PM

Change #1277484 merged by Muehlenhoff:

[operations/puppet@production] Move ganeti-test to the 2026 PKI discovery intermediate

https://gerrit.wikimedia.org/r/1277484

ops-monitoring-bot added a comment.Mon, Apr 27, 12:27 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: testvm2002.codfw.wmnet

  • testvm2002.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox
gerritbot added a comment.Mon, Apr 27, 12:53 PM

Change #1277438 merged by Elukey:

[operations/puppet@production] envoyproxy: trigger the envoy's config re-creation if deleted

https://gerrit.wikimedia.org/r/1277438

gerritbot added a comment.Mon, Apr 27, 1:23 PM

Change #1277452 merged by Elukey:

[operations/puppet@production] profile::tlsproxy::envoy: add condition to cfss base options

https://gerrit.wikimedia.org/r/1277452

gerritbot added a comment.Mon, Apr 27, 1:30 PM

Change #1277449 merged by Elukey:

[operations/puppet@production] Move netbox and presto to the new PKI intermediate

https://gerrit.wikimedia.org/r/1277449

gerritbot added a comment.Mon, Apr 27, 1:40 PM

Change #1277507 merged by Elukey:

[operations/puppet@production] profile::puppetdb: move to the discovery2026 pki intermediate

https://gerrit.wikimedia.org/r/1277507

Dzahn added a subtask: T424372: CertAlmostExpired.Mon, Apr 27, 2:17 PM
gerritbot added a comment.Mon, Apr 27, 2:42 PM

Change #1275956 abandoned by Elukey:

[operations/puppet@production] profile::pki::get_cert: add lookup() to the label argument

https://gerrit.wikimedia.org/r/1275956

elukey added a comment.Mon, Apr 27, 2:54 PM

@bking the scholar internal wdqs hosts should be fixed now, lemme know if you want to test more!

gerritbot added a comment.Mon, Apr 27, 3:09 PM

Change #1277512 merged by Elukey:

[operations/puppet@production] profile::cache::purge: move to the new pki intermediate

https://gerrit.wikimedia.org/r/1277512

gerritbot added a comment.Mon, Apr 27, 3:31 PM

Change #1277510 merged by Elukey:

[operations/puppet@production] profile::dragonfly: move to the new pki intermediate

https://gerrit.wikimedia.org/r/1277510

gerritbot added a comment.Mon, Apr 27, 3:41 PM

Change #1277511 merged by Elukey:

[operations/puppet@production] profile::docker_registry: move to the new pki intermediate

https://gerrit.wikimedia.org/r/1277511

gerritbot added a comment.Mon, Apr 27, 3:51 PM

Change #1277513 merged by Elukey:

[operations/puppet@production] profile::etcd::tlsproxy: move to the new pki intermediate

https://gerrit.wikimedia.org/r/1277513

• bd808 mentioned this in T424549: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep.Mon, Apr 27, 4:20 PM
• bd808 added a subtask: T424549: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep.Mon, Apr 27, 4:29 PM
gerritbot added a comment.Mon, Apr 27, 4:43 PM

Change #1277622 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] admin_ng: Move all clusters to the pki discovery2026 intermediate

https://gerrit.wikimedia.org/r/1277622

gerritbot added a comment.Mon, Apr 27, 7:30 PM

Change #1277713 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs: enable new discovery intermediate certificate

https://gerrit.wikimedia.org/r/1277713

gerritbot added a comment.Mon, Apr 27, 8:19 PM

Change #1277713 merged by Bking:

[operations/puppet@production] wdqs: enable new discovery intermediate certificate

https://gerrit.wikimedia.org/r/1277713

bking added a comment.EditedMon, Apr 27, 8:49 PM

@elukey thanks for getting those done! I just merged the above patch to migrate all of WDQS and I'm still having to do the one-off steps I described in P91477 to get envoy to actually serve the new certificate on the production WDQS hosts.

Looking at the history in wdqs2026 it looks like you had to do some one-off steps as well. Is that expected? I (perhaps foolishly) assumed that the Puppet code was working when you said they were done.

Let me know if it is fixed and I just missed something, or if I will still need to do some one-off steps. I'm totally fine either way.

gerritbot added a comment.Tue, Apr 28, 6:56 AM

Change #1277622 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: Move all clusters to the pki discovery2026 intermediate

https://gerrit.wikimedia.org/r/1277622

Stashbot added a comment.Tue, Apr 28, 7:24 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-28T07:24:11Z] <jayme> switching cfss-issuer instances on all clusters to use discovery2026 - T420993

gerritbot added a comment.Tue, Apr 28, 7:39 AM

Change #1278256 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Switch remaining Ganeti clusters to discovery2026 intermediate

https://gerrit.wikimedia.org/r/1278256

Stashbot added a comment.Tue, Apr 28, 7:55 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-28T07:55:50Z] <jayme> started renewal of certificates on codfw kubernetes clusters - T420993

gerritbot added a comment.Tue, Apr 28, 8:39 AM

Change #1278256 merged by Muehlenhoff:

[operations/puppet@production] Switch remaining Ganeti clusters to discovery2026 intermediate

https://gerrit.wikimedia.org/r/1278256

Stashbot added a comment.Tue, Apr 28, 8:42 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-28T08:42:22Z] <jayme> started renewal of certificates on eqiad kubernetes clusters - T420993

Stashbot added a comment.Tue, Apr 28, 8:44 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-28T08:44:41Z] <moritzm> migrate Ganeti clusters to the new discovery2026 intermediate, starting for the edges T420993

Stashbot added a comment.Tue, Apr 28, 9:21 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-28T09:21:30Z] <moritzm> migrate eqiad/codfw Ganeti clusters to the new discovery2026 intermediate T420993

JMeybohm added a comment.Tue, Apr 28, 10:37 AM

The renewal of all cert-manager managed certificates (in all k8s clusters) has been completed. All certificates are now issued by discovery2026.

gerritbot added a comment.Tue, Apr 28, 10:48 AM

Change #1278391 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] debmonitor: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1278391

gerritbot added a comment.Tue, Apr 28, 11:14 AM

Change #1278391 merged by Muehlenhoff:

[operations/puppet@production] debmonitor: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1278391

gerritbot added a comment.Tue, Apr 28, 12:12 PM

Change #1278426 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] apt/staging: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1278426

gerritbot added a comment.Tue, Apr 28, 1:20 PM

Change #1277508 merged by Bking:

[operations/puppet@production] profile::opensearch::cirrus::server: move to a new pki intermediate

https://gerrit.wikimedia.org/r/1277508

gerritbot added a comment.Tue, Apr 28, 1:35 PM

Change #1277509 merged by Ssingh:

[operations/puppet@production] profile::hcaptcha: move to the discovery2026 pki intermediate

https://gerrit.wikimedia.org/r/1277509

gerritbot added a comment.Tue, Apr 28, 1:43 PM

Change #1278426 merged by Muehlenhoff:

[operations/puppet@production] apt/staging: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1278426

gerritbot added a comment.Tue, Apr 28, 2:57 PM

Change #1278491 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] puppetboard: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1278491

gerritbot added a comment.Tue, Apr 28, 3:03 PM

Change #1278491 merged by Muehlenhoff:

[operations/puppet@production] puppetboard: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1278491

MoritzMuehlenhoff created subtask T424669: Migrate Collab Envoy TLS proxy services to the 2026 discovery intermediate.Tue, Apr 28, 3:20 PM
atsuko subscribed.Tue, Apr 28, 3:24 PM
MoritzMuehlenhoff created subtask T424671: Migrate ServiceOps Envoy TLS proxy services to the 2026 discovery intermediate.Tue, Apr 28, 3:25 PM
gerritbot added a comment.Tue, Apr 28, 3:28 PM

Change #1277506 merged by Jcrespo:

[operations/puppet@production] profile::mediabackup: move to the discovery2026 pki intermediate

https://gerrit.wikimedia.org/r/1277506

MoritzMuehlenhoff created subtask T424672: Migrate Data Platform Envoy TLS proxy services to the 2026 discovery intermediate.Tue, Apr 28, 3:30 PM
MoritzMuehlenhoff created subtask T424673: Migrate o11y Envoy TLS proxy services to the 2026 discovery intermediate.Tue, Apr 28, 3:34 PM
MoritzMuehlenhoff created subtask T424674: Migrate Data Persistence Envoy TLS proxy services to the 2026 discovery intermediate.Tue, Apr 28, 3:38 PM
MoritzMuehlenhoff created subtask T424675: Migrate Tools Infrastructure Envoy TLS proxy services to the 2026 discovery intermediate.Tue, Apr 28, 3:41 PM
gerritbot added a comment.Tue, Apr 28, 3:42 PM

Change #1278499 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::crm: move the pki intermediate to discovery2026

https://gerrit.wikimedia.org/r/1278499

jcrespo added a comment.Tue, Apr 28, 3:45 PM

I thought my puppet code had a bug caused by a refresh not being enough to reload the tls configuration, but that wasn't the issue, the automatic refresh from puppet was enough; thus the problem I had above was due to the long-running client requests, not the server. Nevertheless, I did a full restart of the service for testing and then verified it all got the discovery2026 cert (checked with openssl for all open ports). All good on my side.

MoritzMuehlenhoff created subtask T424676: Migrate SRE IF Envoy TLS proxy services to the 2026 discovery intermediate.Tue, Apr 28, 3:45 PM
elukey added a comment.Tue, Apr 28, 4:07 PM
In T420993#11863146, @bking wrote:

@elukey thanks for getting those done! I just merged the above patch to migrate all of WDQS and I'm still having to do the one-off steps I described in P91477 to get envoy to actually serve the new certificate on the production WDQS hosts.

Looking at the history in wdqs2026 it looks like you had to do some one-off steps as well. Is that expected? I (perhaps foolishly) assumed that the Puppet code was working when you said they were done.

Let me know if it is fixed and I just missed something, or if I will still need to do some one-off steps. I'm totally fine either way.

To keep archives happy - I rolled back https://gerrit.wikimedia.org/r/c/operations/puppet/+/1278480 and from Moritz's tests, everything seems to work fine. The other change, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1277452/, was the only one needed.

Brian is going to force a envoy config rebuild of every wdqs host since puppet cannot fix them. Example: cumin -m async 'wdqs1026*' 'depool' '/usr/local/sbin/build-envoy-config -c /etc/envoy' 'systemctl restart envoyproxy' 'pool'`

Moritz created subtasks for all teams that manage tlsproxy instances, to force the new intermediate to be picked up.

gerritbot added a comment.Tue, Apr 28, 4:33 PM

Change #1278499 merged by Elukey:

[operations/puppet@production] role::crm: move the pki intermediate to discovery2026

https://gerrit.wikimedia.org/r/1278499

gerritbot added a comment.Tue, Apr 28, 6:13 PM

Change #1278527 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] prometheus: switch to discovery2026 for envoy

https://gerrit.wikimedia.org/r/1278527

gerritbot added a comment.Tue, Apr 28, 7:27 PM

Change #1278546 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] titan: switch to discovery2026 for envoy

https://gerrit.wikimedia.org/r/1278546

gerritbot added a comment.Tue, Apr 28, 8:08 PM

Change #1278561 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs: remove references to defunct role wdqs::internal

https://gerrit.wikimedia.org/r/1278561

gerritbot added a comment.Tue, Apr 28, 8:25 PM

Change #1278546 merged by Herron:

[operations/puppet@production] titan: switch to discovery2026 for envoy

https://gerrit.wikimedia.org/r/1278546

gerritbot added a comment.Tue, Apr 28, 8:35 PM

Change #1278527 merged by Herron:

[operations/puppet@production] prometheus: switch to discovery2026 for envoy

https://gerrit.wikimedia.org/r/1278527

gerritbot added a comment.Tue, Apr 28, 9:04 PM

Change #1278595 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] prometheus::pop: switch to discovery2026

https://gerrit.wikimedia.org/r/1278595

gerritbot added a comment.Tue, Apr 28, 9:05 PM

Change #1278595 merged by Herron:

[operations/puppet@production] prometheus::pop: switch to discovery2026

https://gerrit.wikimedia.org/r/1278595

gerritbot added a comment.Tue, Apr 28, 9:23 PM

Change #1278610 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wcqs: Migrate to new discovery intermediate

https://gerrit.wikimedia.org/r/1278610

JMeybohm closed subtask T424671: Migrate ServiceOps Envoy TLS proxy services to the 2026 discovery intermediate as Resolved.Wed, Apr 29, 6:48 AM
elukey closed subtask T424676: Migrate SRE IF Envoy TLS proxy services to the 2026 discovery intermediate as Resolved.Wed, Apr 29, 9:18 AM
MoritzMuehlenhoff closed subtask T424675: Migrate Tools Infrastructure Envoy TLS proxy services to the 2026 discovery intermediate as Resolved.Wed, Apr 29, 9:53 AM
gerritbot added a comment.Wed, Apr 29, 10:45 AM

Change #1279268 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] etherpad: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1279268

gerritbot added a comment.Wed, Apr 29, 10:58 AM

Change #1279268 merged by Jelto:

[operations/puppet@production] etherpad: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1279268

gerritbot added a comment.Wed, Apr 29, 11:11 AM

Change #1279273 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] aphlict: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1279273

gerritbot added a comment.Wed, Apr 29, 11:13 AM

Change #1279273 merged by Jelto:

[operations/puppet@production] aphlict: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1279273

gerritbot added a comment.Wed, Apr 29, 11:23 AM

Change #1279274 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] phabricator: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1279274

gerritbot added a comment.Wed, Apr 29, 11:28 AM

Change #1279274 merged by Jelto:

[operations/puppet@production] phabricator: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1279274

gerritbot added a comment.Wed, Apr 29, 11:38 AM

Change #1279282 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] peopleweb: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1279282

gerritbot added a comment.Wed, Apr 29, 11:41 AM

Change #1279282 merged by Jelto:

[operations/puppet@production] peopleweb: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1279282

gerritbot added a comment.Wed, Apr 29, 11:55 AM

Change #1279287 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] doc: Switch to discovery2026 intermediate for Envoy

https://gerrit.wikimedia.org/r/1279287

elukey closed subtask T424673: Migrate o11y Envoy TLS proxy services to the 2026 discovery intermediate as Resolved.Wed, Apr 29, 12:52 PM
Jelto closed subtask T424669: Migrate Collab Envoy TLS proxy services to the 2026 discovery intermediate as Resolved.Wed, Apr 29, 1:05 PM
gerritbot added a comment.Wed, Apr 29, 1:19 PM

Change #1279340 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] tlsproxy::envoy: Bump default now that services have moved

https://gerrit.wikimedia.org/r/1279340

gerritbot added a comment.Wed, Apr 29, 1:22 PM

Change #1278610 merged by Bking:

[operations/puppet@production] wcqs: Migrate to new discovery intermediate

https://gerrit.wikimedia.org/r/1278610

MatthewVernon closed subtask T424674: Migrate Data Persistence Envoy TLS proxy services to the 2026 discovery intermediate as Resolved.Wed, Apr 29, 1:39 PM
MoritzMuehlenhoff added a comment.Wed, Apr 29, 1:40 PM

I ran a fleet-wide grep tls_cert -A 4 /etc/envoy/envoy.yaml in Cumin on C:profile::envoy and only the new discovery2026 certs and a handful of ACME certs come up

bking closed subtask T424672: Migrate Data Platform Envoy TLS proxy services to the 2026 discovery intermediate as Resolved.Wed, Apr 29, 1:42 PM
gerritbot added a comment.Wed, Apr 29, 1:45 PM

Change #1279347 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::crm: update postfix's cfssl pki intermediate

https://gerrit.wikimedia.org/r/1279347

gerritbot added a comment.Wed, Apr 29, 1:55 PM

Change #1279347 merged by Elukey:

[operations/puppet@production] role::crm: update postfix's cfssl pki intermediate

https://gerrit.wikimedia.org/r/1279347

Volans subscribed.EditedThu, Apr 30, 8:42 AM

To keep everyone up to date, this morning Moritz asked me to have a look at why debmonitor-client was failing on the two pki hosts (pki1001.eqiad.wmnet,pki2002.codfw.wmnet) with:

SSLError(SSLError(1, '[SSL: SSLV3_ALERT_CERTIFICATE_EXPIRED] sslv3 alert certificate expired (_ssl.c:2636)')): /hosts/pki1001.eqiad.wmnet/update

After debugging it for a bit I discovered that the certificate in /etc/debmonitor/ssl/debmonitor__pki1001_eqiad_wmnet.chained.pem was having a valid leaf certificate and a correct and valid intermediated chained. The problem was that the signer of the leaf certificate (X509v3 Authority Key Identifier) was not matching the intermediate cert identifier (X509v3 Subject Key Identifier) and was most likely the old one.
As this happened only on the two PKI hosts it's possible that either by design or because of the rollout procedure, the way those certificates were generated on the PKI hosts themselves showed this issue.
I've solved the problem moving the .pem and .chained.pem files and running puppet on the two hosts. They now got a leaf with the proper signer ID and debmonitor client is working fine.
I've left the frankenstein certs on disk with .wrong extension if anyone wants to dig more and fully understand the chain of events that generated the wrong "package".

This could possibly also classify as a cfssl (or our way of using it) bug that was able to generate a chained cert where the leaf is not signed by the intermediate chained in there.

elukey closed subtask T424549: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep as Resolved.Mon, May 4, 10:11 AM
elukey updated the task description. (Show Details)Mon, May 4, 10:13 AM
gerritbot added a comment.Mon, May 4, 12:57 PM

Change #1282350 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::pki: remove the 'discovery' intermediate's config

https://gerrit.wikimedia.org/r/1282350

elukey added a comment.Thu, May 7, 9:01 AM

Updated the documentation: https://wikitech.wikimedia.org/wiki/PKI/CA_Operations#Renewing_an_existing_intermediate

Next steps:

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits