Page MenuHomePhabricator
Create Task
Maniphest T360439

Phase out cergen for Search Platform services
Closed, ResolvedPublic
Actions

Description

cergen is our legacy tooling to manage/generate TLS certificates (https://wikitech.wikimedia.org/wiki/Cergen). It has been replaced by an installation of cfssl (https://wikitech.wikimedia.org/wiki/PKI) and the majority of services uses it.

Our cergen installation is co-hosted on one of the Puppet master (5) frontends (puppetmaster1001), which runs Buster. cergen is based on legacy libraries (it uses networkx v1, which is incompatible with current networkx releases (networkx 2 was released in 2017) and even when the puppetmasters were moved to Buster, this needed a hack to build a co-installable legacy package in a compomnent (T235405).

Instead of forward-porting it yet again to the new installation we'll use the Puppet 5 -> Puppet 7 migration to also phase out cergen and only use cfssl.

Most of those certs are used by Envoy and our Puppet integration makes switching relatively straightforward by switching the profile::tlsproxy::envoy::ssl_provider Hiera flag to "cfssl" (along with specifying SNI names via profile::tlsproxy::envoy::cfssl_options/hosts)

Some examples for this can be found at
https://github.com/wikimedia/operations-puppet/commit/66fbddeac3a4b2dfa1d8e19a49cc649dcb745f18
https://github.com/wikimedia/operations-puppet/commit/a00d0441b4509e736d8abd6ff63f25224e306239

For use cases outside of Envoy the profile::pki::get_cert define provides a convenient method to request certificates. An example how the gradual migration was implemented for the Ganeti RAPI endpoint can be found at https://github.com/wikimedia/operations-puppet/commit/98350d2dff51bb9bf57263fe50f409374892ae1d

There are currently 5 certificate YAML specs defined in /srv/private/modules/secret/secrets/certificates/certificate.manifests.d which need to be moved to PKI/cfssl. Some services are likely also ported already and only the YAML spec file and the legacy certs were forgotten and fixing it might be a simple as removing the legacy cert material.

nginx based
envoy based

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT330490 Next steps for Puppet 7
 OpenMoritzMuehlenhoffT357750 Phase out cergen
 ResolvedBTullisT360439 Phase out cergen for Search Platform services

Event Timeline

MoritzMuehlenhoff created this task.Mar 19 2024, 2:43 PM
MoritzMuehlenhoff mentioned this in T357750: Phase out cergen.
Gehel triaged this task as Medium priority.Mar 20 2024, 9:00 AM
Gehel moved this task from Incoming to Toil / Automation on the Data-Platform-SRE board.
Gehel moved this task from Toil / Automation to 2024.03.25 - 2024.04.14 on the Data-Platform-SRE board.Mar 22 2024, 8:58 AM
Gehel edited projects, added Data-Platform-SRE (2024.03.25 - 2024.04.14); removed Data-Platform-SRE.
jcrespo moved this task from Backlog to Acknowledged on the SRE board.Mar 25 2024, 9:07 AM
Gehel edited projects, added Data-Platform-SRE (2024.04.15 - 2024.05.05); removed Data-Platform-SRE (2024.03.25 - 2024.04.14).Apr 15 2024, 12:50 PM
BTullis claimed this task.Tue, Apr 23, 11:46 AM
BTullis moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.04.15 - 2024.05.05) board.
Paladox unsubscribed.Tue, Apr 23, 12:12 PM
BTullis added a comment.Tue, Apr 23, 12:26 PM

I am starting by looking at the relforge cluster. I see that the certificates are served by nginx and they are still using the puppet CA based certificates.

btullis@relforge1003:/etc/nginx$ openssl x509 -in /etc/ssl/localcerts/relforge.svc.eqiad.wmnet.chained.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 7899 (0x1edb)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = Puppet CA: palladium.eqiad.wmnet
        Validity
            Not Before: Mar 18 02:55:32 2021 GMT
            Not After : Mar 18 02:55:32 2026 GMT
        Subject: CN = relforge.svc.eqiad.wmnet

I'll check to see if there is any code ready to deploy cfssl based certificates for nginx.

MoritzMuehlenhoff added a comment.Tue, Apr 23, 12:50 PM
In T360439#9735349, @BTullis wrote:

I'll check to see if there is any code ready to deploy cfssl based certificates for nginx.

John added support for using cfssl as the provider used by profile::elasticsearch::cirrus::ssl_provider two years ago, but it's not yet used.

gerritbot added a comment.Tue, Apr 23, 1:02 PM

Change #1023426 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Switch relforge certificates from cergen to pki

https://gerrit.wikimedia.org/r/1023426

gerritbot added a project: Patch-For-Review.Tue, Apr 23, 1:02 PM
gerritbot added a comment.Tue, Apr 23, 2:35 PM

Change #1023440 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Replace tabs with 4 spaces in tlsproxy nginx.conf

https://gerrit.wikimedia.org/r/1023440

BTullis added subscribers: RKemper, hnowlan.Tue, Apr 23, 3:00 PM

I have a whitespace-only change in the nginx configuration for tlsproxy here: https://gerrit.wikimedia.org/r/1023440
It looks safe to me, but since it touches all the maps servers and every elasticsearch::cirrus server, I think that I had better get a review from @hnowlan and either @bking or @RKemper.

I have tried out the new cfssl support in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1023426 but unless I'm mistaken I think that we will need to modify profile::elasticsearch::cirrus a little. The server names and aliases aren't being passed through to the cfssl based certificate and I'm not sure that the resulting nginx config will be correct.

image.png (180×816 px, 41 KB)

gerritbot added a comment.Tue, Apr 23, 4:41 PM

Change #1023469 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add server aliases to the cirrus/cfssl proxy config

https://gerrit.wikimedia.org/r/1023469

BTullis updated the task description. (Show Details)Wed, Apr 24, 9:36 AM
BTullis removed subscribers: Volans, Sethabathaba, daniel and 11 others.
gerritbot added a comment.Wed, Apr 24, 9:44 AM

Change #1023813 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Switch elasticsearch::cirrus tlsproxy to pki

https://gerrit.wikimedia.org/r/1023813

BTullis updated the task description. (Show Details)Wed, Apr 24, 9:45 AM
gerritbot added a comment.Wed, Apr 24, 10:06 AM

Change #1023815 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Switch the wcqs tlsproxy to use pki

https://gerrit.wikimedia.org/r/1023815

BTullis updated the task description. (Show Details)Wed, Apr 24, 10:20 AM
BTullis updated the task description. (Show Details)Wed, Apr 24, 10:23 AM
gerritbot added a comment.Wed, Apr 24, 10:29 AM

Change #1023819 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Switch wdqs::public tlsproxy from cergen to pki

https://gerrit.wikimedia.org/r/1023819

BTullis updated the task description. (Show Details)Wed, Apr 24, 10:32 AM
gerritbot added a comment.Wed, Apr 24, 10:36 AM

Change #1023825 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Switch wdqs::internal tlsproxy from cergen to pki

https://gerrit.wikimedia.org/r/1023825

BTullis updated the task description. (Show Details)Wed, Apr 24, 10:37 AM
BTullis updated the task description. (Show Details)
BTullis moved this task from In Progress to Needs Review on the Data-Platform-SRE (2024.04.15 - 2024.05.05) board.Wed, Apr 24, 11:09 AM
gerritbot added a comment.Wed, Apr 24, 9:14 PM

Change #1023440 merged by Bking:

[operations/puppet@production] Replace tabs with 4 spaces in tlsproxy nginx.conf

https://gerrit.wikimedia.org/r/1023440

gerritbot added a comment.Thu, Apr 25, 9:20 AM

Change #1023815 merged by Btullis:

[operations/puppet@production] Switch the wcqs tlsproxy to use pki

https://gerrit.wikimedia.org/r/1023815

BTullis moved this task from Needs Review to In Progress on the Data-Platform-SRE (2024.04.15 - 2024.05.05) board.Thu, Apr 25, 9:36 AM
BTullis updated the task description. (Show Details)
gerritbot added a comment.Thu, Apr 25, 10:08 AM

Change #1023825 merged by Btullis:

[operations/puppet@production] Switch wdqs::internal tlsproxy from cergen to pki

https://gerrit.wikimedia.org/r/1023825

BTullis updated the task description. (Show Details)Thu, Apr 25, 10:14 AM
gerritbot added a comment.Thu, Apr 25, 10:15 AM

Change #1023819 merged by Btullis:

[operations/puppet@production] Switch wdqs::public tlsproxy from cergen to pki

https://gerrit.wikimedia.org/r/1023819

BTullis updated the task description. (Show Details)Thu, Apr 25, 10:33 AM
gerritbot added a comment.Thu, Apr 25, 10:35 AM

Change #1023469 merged by Btullis:

[operations/puppet@production] Add server aliases to the cirrus/cfssl proxy config

https://gerrit.wikimedia.org/r/1023469

gerritbot added a comment.Thu, Apr 25, 1:31 PM

Change #1023426 merged by Bking:

[operations/puppet@production] Switch relforge certificates from cergen to pki

https://gerrit.wikimedia.org/r/1023426

BTullis updated the task description. (Show Details)Thu, Apr 25, 1:44 PM
BTullis added a comment.Thu, Apr 25, 1:46 PM

We rolled out the change to relforge. It works but the Icinga checks on certificate expiry triggered because they fire on the discovery intermediate's default certificate expiry.

image.png (531×1 px, 207 KB)

I think it's best to fix this before we roll it out to the main elasticsearch::cirrus clusters.

gerritbot added a comment.Thu, Apr 25, 3:21 PM

Change #1024420 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove obsolete certs for wdqs/wcqs

https://gerrit.wikimedia.org/r/1024420

gerritbot added a comment.Thu, Apr 25, 3:22 PM

Change #1024421 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Remove obsolete dummy certs

https://gerrit.wikimedia.org/r/1024421

bking mentioned this in T363502: Migrate TLS certificate checks from icinga to prometheus blackbox.Thu, Apr 25, 5:01 PM
gerritbot added a comment.Thu, Apr 25, 7:44 PM

Change #1024481 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elasticsearch: Configure alerts for short-lived certs

https://gerrit.wikimedia.org/r/1024481

gerritbot added a comment.Fri, Apr 26, 11:29 AM

Change #1024421 merged by Muehlenhoff:

[labs/private@master] Remove obsolete dummy certs

https://gerrit.wikimedia.org/r/1024421

Muehlenhoff mentioned this in rLPRI6e0379e13782: Remove obsolete dummy certs.Fri, Apr 26, 11:30 AM
gerritbot added a comment.Tue, Apr 30, 9:57 AM

Change #1024420 merged by Muehlenhoff:

[operations/puppet@production] Remove obsolete certs for wdqs/wcqs

https://gerrit.wikimedia.org/r/1024420

gerritbot added a comment.Tue, Apr 30, 1:24 PM

Change #1025775 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove obsolete certificate

https://gerrit.wikimedia.org/r/1025775

gerritbot added a comment.Tue, Apr 30, 1:30 PM

Change #1025777 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Remove obsolete stub cert

https://gerrit.wikimedia.org/r/1025777

BTullis moved this task from In Progress to To Be Deployed on the Data-Platform-SRE (2024.04.15 - 2024.05.05) board.Tue, Apr 30, 3:25 PM
gerritbot added a comment.Tue, Apr 30, 9:00 PM

Change #1023813 merged by Bking:

[operations/puppet@production] Switch elasticsearch::cirrus tlsproxy to pki

https://gerrit.wikimedia.org/r/1023813

gerritbot added a comment.Tue, Apr 30, 9:29 PM

Change #1024481 merged by Bking:

[operations/puppet@production] elasticsearch: Configure alerts for short-lived certs

https://gerrit.wikimedia.org/r/1024481

bking updated the task description. (Show Details)Wed, May 1, 1:18 PM
bking closed this task as Resolved.Wed, May 1, 1:23 PM
bking moved this task from To Be Deployed to Done on the Data-Platform-SRE (2024.04.15 - 2024.05.05) board.

As of yesterday, the production Elastic clusters are using CFSSL, which means we've accomplished our migration off of Cergen. Thanks to @BTullis
and everyone else who helped with this.

gerritbot added a comment.Thu, May 2, 6:53 AM

Change #1025777 merged by Muehlenhoff:

[labs/private@master] Remove obsolete stub cert

https://gerrit.wikimedia.org/r/1025777

Muehlenhoff mentioned this in rLPRI0e43a7994222: Remove obsolete stub cert.Thu, May 2, 6:56 AM
gerritbot added a comment.Thu, May 2, 7:30 AM

Change #1025775 merged by Muehlenhoff:

[operations/puppet@production] Remove obsolete certificate

https://gerrit.wikimedia.org/r/1025775

gerritbot added a comment.Thu, May 2, 7:36 AM

Change #1026438 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove obsolete cert

https://gerrit.wikimedia.org/r/1026438

gerritbot added a comment.Thu, May 2, 7:37 AM

Change #1026439 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Remove obsolete dummy cert

https://gerrit.wikimedia.org/r/1026439

gerritbot added a comment.Fri, May 3, 8:56 AM

Change #1026803 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] elasticsearch: Remove support for sslcert SSL provider

https://gerrit.wikimedia.org/r/1026803

gerritbot added a comment.Mon, May 6, 6:28 AM

Change #1026439 merged by Muehlenhoff:

[labs/private@master] Remove obsolete dummy cert

https://gerrit.wikimedia.org/r/1026439

Muehlenhoff mentioned this in rLPRIc6310c733b8f: Remove obsolete dummy cert.Mon, May 6, 6:31 AM
gerritbot added a comment.Mon, May 6, 3:14 PM

Change #1026438 merged by Muehlenhoff:

[operations/puppet@production] Remove obsolete cert

https://gerrit.wikimedia.org/r/1026438

gerritbot added a comment.Tue, May 7, 1:40 PM

Change #1026803 merged by Muehlenhoff:

[operations/puppet@production] elasticsearch: Remove support for sslcert SSL provider

https://gerrit.wikimedia.org/r/1026803

Maintenance_bot removed a project: Patch-For-Review.Tue, May 7, 2:31 PM
gerritbot added a comment.Wed, May 8, 8:20 AM

Change #1029121 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] elasticsearch::tlsproxy: Stop passing certs to tlsproxy::localssl

https://gerrit.wikimedia.org/r/1029121

gerritbot added a project: Patch-For-Review.Wed, May 8, 8:20 AM
gerritbot added a comment.Mon, May 13, 8:41 AM

Change #1029121 merged by Muehlenhoff:

[operations/puppet@production] elasticsearch::tlsproxy: Stop passing certs to tlsproxy::localssl

https://gerrit.wikimedia.org/r/1029121

Maintenance_bot removed a project: Patch-For-Review.Mon, May 13, 9:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL