Page MenuHomePhabricator
Create Task
Maniphest T375569

Remove RSA certificates from puppet
Closed, ResolvedPublic
Actions

Description

$ git grep rsa-2048 | wc -l
71

Help out our fellow SREs to get these cert configs updated!

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

BCornwall created this task.Sep 24 2024, 10:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 24 2024, 10:05 PM
gerritbot added a comment.Sep 24 2024, 10:14 PM

Change #1075326 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] Remove rsa-2048 certs from services

https://gerrit.wikimedia.org/r/1075326

gerritbot added a project: Patch-For-Review.Sep 24 2024, 10:14 PM
MoritzMuehlenhoff subscribed.Sep 25 2024, 10:05 AM
gerritbot added a comment.Sep 25 2024, 5:10 PM

Change #1075326 abandoned by BCornwall:

[operations/puppet@production] Remove rsa-2048 certs from services

Reason:

Splitting up the commits

https://gerrit.wikimedia.org/r/1075326

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075604 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] Remove rsa-2048 certs from mail services

https://gerrit.wikimedia.org/r/1075604

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075605 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] archiva: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075605

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075606 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] dynamicproxy: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075606

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075607 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] ldap: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075607

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075608 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] idp: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075608

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075609 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] toolforge: Remove rsa-2048 certs from nginx config

https://gerrit.wikimedia.org/r/1075609

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075610 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] dumps: Remove rsa-2048 certs from nginx config

https://gerrit.wikimedia.org/r/1075610

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075611 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] docker-registry: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075611

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075612 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] tlsproxy: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075612

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075613 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] durum: Remove rsa-2048 certs from nginx config

https://gerrit.wikimedia.org/r/1075613

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075614 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] gerrit: Remove rsa-2048 certs from apache config

https://gerrit.wikimedia.org/r/1075614

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075615 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] icinga: Remove external monitoring rsa-2048 certs

https://gerrit.wikimedia.org/r/1075615

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075616 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] librenms: Remove rsa-2048 certs from Apache config

https://gerrit.wikimedia.org/r/1075616

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075617 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] mirrors: Remove rsa-2048 certs from Apache config

https://gerrit.wikimedia.org/r/1075617

gerritbot added a comment.Sep 25 2024, 5:11 PM

Change #1075618 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] novaproxy: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075618

gerritbot added a comment.Sep 25 2024, 5:12 PM

Change #1075619 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] openstack: Remove rsa-2048 certs from nginx config

https://gerrit.wikimedia.org/r/1075619

gerritbot added a comment.Sep 25 2024, 5:21 PM

Change #1075619 merged by Majavah:

[operations/puppet@production] openstack: Remove rsa-2048 certs from nginx config

https://gerrit.wikimedia.org/r/1075619

gerritbot added a comment.Sep 25 2024, 5:27 PM

Change #1075618 merged by Majavah:

[operations/puppet@production] novaproxy: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075618

gerritbot added a comment.Sep 25 2024, 5:33 PM

Change #1075611 merged by Majavah:

[operations/puppet@production] docker-registry: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075611

gerritbot added a comment.Sep 25 2024, 5:43 PM

Change #1075621 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: legacy_redirector: Remove rsa-2048 certs from nginx config

https://gerrit.wikimedia.org/r/1075621

gerritbot added a comment.Sep 25 2024, 5:47 PM

Change #1075621 merged by Majavah:

[operations/puppet@production] P:toolforge: legacy_redirector: Remove rsa-2048 certs from nginx config

https://gerrit.wikimedia.org/r/1075621

BCornwall changed the task status from Open to In Progress.Sep 25 2024, 6:03 PM
BCornwall triaged this task as Medium priority.
BCornwall moved this task from Backlog to Actively Servicing on the Traffic board.
gerritbot added a comment.Sep 26 2024, 4:58 PM

Change #1075609 merged by BCornwall:

[operations/puppet@production] P:toolforge: proxy: Remove rsa-2048 certs from nginx config

https://gerrit.wikimedia.org/r/1075609

gerritbot added a comment.Oct 28 2024, 6:50 PM

Change #1075605 merged by BCornwall:

[operations/puppet@production] archiva: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075605

gerritbot added a comment.Oct 28 2024, 7:07 PM

Change #1075613 merged by BCornwall:

[operations/puppet@production] durum: Remove rsa-2048 certs from nginx config

https://gerrit.wikimedia.org/r/1075613

gerritbot added a comment.Oct 28 2024, 7:18 PM

Change #1075607 merged by BCornwall:

[operations/puppet@production] ldap: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075607

gerritbot added a comment.Oct 28 2024, 7:25 PM

Change #1075617 merged by BCornwall:

[operations/puppet@production] mirrors: Remove rsa-2048 certs from Apache config

https://gerrit.wikimedia.org/r/1075617

gerritbot added a comment.Oct 28 2024, 7:31 PM

Change #1075610 merged by BCornwall:

[operations/puppet@production] dumps: Remove rsa-2048 certs from nginx config

https://gerrit.wikimedia.org/r/1075610

Stashbot added a comment.Oct 28 2024, 7:33 PM

Mentioned in SAL (#wikimedia-operations) [2024-10-28T19:33:26Z] <brett> Removed RSA certificate support from mirrors, dumps (T375569)

gerritbot added a comment.Oct 28 2024, 7:46 PM

Change #1075612 merged by BCornwall:

[operations/puppet@production] tlsproxy: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075612

Stashbot added a comment.Oct 28 2024, 7:52 PM

Mentioned in SAL (#wikimedia-operations) [2024-10-28T19:52:27Z] <brett> Removed RSA certificate support from tlsproxy (T375569)

gerritbot added a comment.Oct 29 2024, 5:46 PM

Change #1075616 merged by BCornwall:

[operations/puppet@production] librenms: Remove rsa-2048 certs from Apache config

https://gerrit.wikimedia.org/r/1075616

gerritbot added a comment.Oct 29 2024, 5:47 PM

Change #1075615 merged by BCornwall:

[operations/puppet@production] icinga: Remove external monitoring rsa-2048 certs

https://gerrit.wikimedia.org/r/1075615

Stashbot added a comment.Oct 29 2024, 5:49 PM

Mentioned in SAL (#wikimedia-operations) [2024-10-29T17:49:31Z] <brett> Remove RSA cert support from Icinga, librenms (T375569)

gerritbot added a comment.Oct 29 2024, 5:53 PM

Change #1075606 merged by Majavah:

[operations/puppet@production] dynamicproxy: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075606

Stashbot added a comment.Nov 6 2024, 6:41 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-06T18:41:27Z] <brett> Remove RSA cert support from P:idp clients (icinga, karma, klaxon, librenms, orchestrator) (T375569)

gerritbot added a comment.Nov 6 2024, 6:41 PM

Change #1075608 merged by BCornwall:

[operations/puppet@production] idp: Remove rsa-2048 certs

https://gerrit.wikimedia.org/r/1075608

BCornwall added a subscriber: Legoktm.EditedNov 26 2024, 1:50 AM

I don't know if this is something that should apply to mail servers or not since compatibility is always behind.

I took a look at lists1004's exim logs with:

$ zgrep -E ' X=TLS\S+ ' mainlog* -ho | sort | uniq -c | sort -n | tail -n10
1615X=TLS1.2:ECDHE_X25519RSA_PSS_RSAE_SHA256AES_128_GCM:128
1996X=TLS1.2:ECDHE_SECP256R1RSA_SHA512AES_128_GCM:128
4391X=TLS1.2:RSA__AES_256_GCM:256
4805X=TLS1.2:ECDHE_SECP256R1RSA_SHA512AES_256_GCM:256
8637X=TLS1.3:ECDHE_X25519RSA_PSS_RSAE_SHA256AES_256_GCM:256
10373X=TLS1.2:ECDHE_SECP384R1RSA_SHA512AES_256_GCM:256
10896X=TLS1.2:ECDHE_SECP256R1RSA_SHA256AES_256_GCM:256
40893X=TLS1.3:ECDHE_X25519RSA_PSS_RSAE_SHA256AES_128_GCM:128
153648X=TLS1.3:ECDHE_SECP256R1RSA_PSS_RSAE_SHA256AES_256_GCM:256
399541X=TLS1.3:ECDHE_X25519ECDSA_SECP256R1_SHA256AES_256_GCM:256

That went through ~600k connections in total. I also noticed that TLS 1.0 seems to be accepted (X=DHE_CUSTOM2048RSA_SHA1AES_256_CBC__SHA1 is in there)

It does sound to me like mail services should be left alone with this effort. @Legoktm would you agree?

Vgutierrez subscribed.Nov 26 2024, 4:14 AM

right now exim is configured with RSA certs only and not with a dual stack (RSA+ECDSA) setup, from lists1004's exim configuration:

# TLS           
                
tls_certificate = /etc/acmecerts/lists/live/rsa-2048.chained.crt
tls_privatekey = /etc/acmecerts/lists/live/rsa-2048.key

From exim's documentation:

For dual-stack (eg. RSA and ECDSA) configurations, these options can be colon-separated lists of file paths.

I'd suggest configuring exim to support ECDSA+RSA first (in that order) and track the evolution of TLS usage

gerritbot added a comment.Jan 29 2025, 8:20 AM

Change #1075614 merged by Jelto:

[operations/puppet@production] gerrit: Remove rsa-2048 certs from apache config

https://gerrit.wikimedia.org/r/1075614

Vgutierrez closed this task as Resolved.Jan 29 2025, 8:41 AM

nice work @BCornwall, could you open a task for lists1004 exim dual-stack (RSA+ECDSA) configuration?

Thanks!

BCornwall added a subtask: T385067: Set up dual-stack ECDSA/RSA certificate support for Exim.Jan 29 2025, 5:00 PM
Vgutierrez closed subtask T385067: Set up dual-stack ECDSA/RSA certificate support for Exim as Resolved.Mar 17 2025, 3:15 PM
Vgutierrez mentioned this in T398019: Set up TLS dual stack for mx-in[12]001.wikimedia.org.Jun 27 2025, 9:28 AM
Vgutierrez mentioned this in T398020: Stop issuing RSA certificates.
gerritbot added a comment.Jun 27 2025, 9:43 AM

Change #1164399 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::proxy: api: Use ec-prime256v1 certificates

https://gerrit.wikimedia.org/r/1164399

gerritbot added a comment.Jun 27 2025, 10:03 AM

Change #1164399 merged by Majavah:

[operations/puppet@production] P:toolforge::proxy: api: Use ec-prime256v1 certificates

https://gerrit.wikimedia.org/r/1164399

Vgutierrez closed subtask T398019: Set up TLS dual stack for mx-in[12]001.wikimedia.org as Invalid.Jun 27 2025, 10:09 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits