Page MenuHomePhabricator

Vgutierrez (Valentín Gutiérrez)
Traffic Security Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Feb 12 2018, 9:51 AM (62 w, 1 d)
Availability
Available
IRC Nick
vgutierrez
LDAP User
Vgutierrez
MediaWiki User
Unknown

Recent Activity

Today

Vgutierrez committed rOSAC3a882a904cd4: acme_chief: Prevalidate CN/SNI list (authored by Vgutierrez).
acme_chief: Prevalidate CN/SNI list
Tue, Apr 23, 12:56 PM
Vgutierrez committed rOSACa06ab2ad5407: CI: Run tests with minimum and latest dependencies (authored by Vgutierrez).
CI: Run tests with minimum and latest dependencies
Tue, Apr 23, 9:00 AM
Vgutierrez moved T221594: Puppetize ATS TLS configuration for incoming traffic from Triage to TLS on the Traffic board.
Tue, Apr 23, 8:33 AM · Patch-For-Review, Traffic, Operations
Vgutierrez triaged T221594: Puppetize ATS TLS configuration for incoming traffic as Normal priority.
Tue, Apr 23, 8:33 AM · Patch-For-Review, Traffic, Operations
Vgutierrez created T221594: Puppetize ATS TLS configuration for incoming traffic.
Tue, Apr 23, 8:33 AM · Patch-For-Review, Traffic, Operations

Thu, Apr 18

Vgutierrez committed rOSACb8a07272a161: acme_chief: Prevalidate CN/SNI list (authored by Vgutierrez).
acme_chief: Prevalidate CN/SNI list
Thu, Apr 18, 1:27 PM
Vgutierrez committed rOSAC12eca669aef3: acme_chief: Prevalidate CN/SNI list (authored by Vgutierrez).
acme_chief: Prevalidate CN/SNI list
Thu, Apr 18, 1:22 PM
Vgutierrez committed rOSACc9cdc62d3978: acme_chief: Prevalidate CN/SNI list (authored by Vgutierrez).
acme_chief: Prevalidate CN/SNI list
Thu, Apr 18, 1:05 PM
Vgutierrez committed rOSAC2c2653891ac3: dns: Move DNS operations to its own module (authored by Vgutierrez).
dns: Move DNS operations to its own module
Thu, Apr 18, 1:05 PM
Vgutierrez added a comment to T221343: puppet fails to run in cp1008 under certain conditions.

for the record, LC_CTYPE=UTF-8

Thu, Apr 18, 9:02 AM · Packaging, Puppet, Operations
Vgutierrez renamed T221343: puppet fails to run in cp1008 under certain conditions from puppet fails to run in cp1008 to puppet fails to run in cp1008 under certain conditions.
Thu, Apr 18, 9:02 AM · Packaging, Puppet, Operations
Vgutierrez added a comment to T221343: puppet fails to run in cp1008 under certain conditions.

so... this is caused by my locales:

vgutierrez@cp1008:~$ unset LC_CTYPE
vgutierrez@cp1008:~$ sudo -i puppet agent -t
Warning: Support for ruby version 2.1.5 is deprecated and will be removed in a future release. See https://puppet.com/docs/puppet/latest/system_requirements.html for a list of supported ruby versions.
   (location: /usr/lib/ruby/vendor_ruby/puppet.rb:130:in `<module:Puppet>')
Warning: Downgrading to PSON for future requests
Info: Using configured environment 'production'
Info: Retrieving pluginfacts

but this was working as expected before

Thu, Apr 18, 8:55 AM · Packaging, Puppet, Operations
Vgutierrez created T221343: puppet fails to run in cp1008 under certain conditions.
Thu, Apr 18, 8:53 AM · Packaging, Puppet, Operations

Wed, Apr 17

Vgutierrez moved T221217: Allow running several ATS instances on the same server from Triage to Caching on the Traffic board.
Wed, Apr 17, 10:30 AM · Patch-For-Review, Operations, Traffic
Vgutierrez added a parent task for T221217: Allow running several ATS instances on the same server: T220383: Evaluate ATS TLS stack.
Wed, Apr 17, 10:25 AM · Patch-For-Review, Operations, Traffic
Vgutierrez added a subtask for T220383: Evaluate ATS TLS stack: T221217: Allow running several ATS instances on the same server.
Wed, Apr 17, 10:25 AM · Traffic, Operations
Vgutierrez created T221217: Allow running several ATS instances on the same server.
Wed, Apr 17, 10:24 AM · Patch-For-Review, Operations, Traffic
Vgutierrez committed rOSAC2dc2c0a28d21: acme_chief: Prevalidate CN/SNI list (authored by Vgutierrez).
acme_chief: Prevalidate CN/SNI list
Wed, Apr 17, 8:45 AM
Vgutierrez committed rOSACbac82390f268: dns: Move DNS operations to its own module (authored by Vgutierrez).
dns: Move DNS operations to its own module
Wed, Apr 17, 8:45 AM
Vgutierrez committed rOSAC11b7f1118228: acme_chief: Prevalidate CN/SNI list (authored by Vgutierrez).
acme_chief: Prevalidate CN/SNI list
Wed, Apr 17, 8:45 AM
Vgutierrez committed rOSAC4148b5a493e2: config: Move ACMEChiefConfig to its own module (authored by Vgutierrez).
config: Move ACMEChiefConfig to its own module
Wed, Apr 17, 8:45 AM
Vgutierrez committed rOSACd44e30ac5202: dns: Move DNS operations to its own module (authored by Vgutierrez).
dns: Move DNS operations to its own module
Wed, Apr 17, 8:45 AM
Vgutierrez committed rOSACc7b80c5222eb: acme_chief: Prevalidate CN/SNI list (authored by Vgutierrez).
acme_chief: Prevalidate CN/SNI list
Wed, Apr 17, 8:45 AM
Vgutierrez committed rOSAC2885c7ef047a: config: Move ACMEChiefConfig to its own module (authored by Vgutierrez).
config: Move ACMEChiefConfig to its own module
Wed, Apr 17, 8:45 AM
Vgutierrez committed rOSAC2aea9a3aa250: dns: Move DNS operations to its own module (authored by Vgutierrez).
dns: Move DNS operations to its own module
Wed, Apr 17, 8:45 AM

Tue, Apr 16

Vgutierrez updated the task description for T220786: Add SPF record for non-canonical domains that are not parked.
Tue, Apr 16, 2:13 PM · Patch-For-Review, Operations, Traffic, DNS

Mon, Apr 15

Vgutierrez added a comment to T219414: acme-chief fails to issue certificates against LE staging environment.

it looks like gdnsd sets a minimum TTL of 60 seconds for dns-01 ACME challenges:

acme_challenge_ttl
           Integer seconds, range 60-3600, default 600.  For temporary ACME DNS-01 challenge data added via "gdnsdctl acme-dns-01 ...", this sets both the time until the TXT records auto-expire from the server and dissappear, and also the TTL of the RRs themselves.  The TTL of static TXT records in zonefiles which happen to have "_acme-challenge" as their leading label are also forced to this TTL regardless of the zonefile-level TTL, to avoid cases of mixed TTLs when mixing static and dynamic records in server outputs.  See the gdnsdctl(8) documentation for more details.
Mon, Apr 15, 9:08 AM · Patch-For-Review, Acme-chief

Fri, Apr 12

Vgutierrez changed the status of T219414: acme-chief fails to issue certificates against LE staging environment from Stalled to Open.
Fri, Apr 12, 2:08 PM · Patch-For-Review, Acme-chief
Vgutierrez updated subscribers of T219414: acme-chief fails to issue certificates against LE staging environment.

According to https://community.letsencrypt.org/t/unable-to-issue-ecdsa-rsa-in-acmev2-staging-environment/90835/9, LE caches dns-01 challenges for 60 seconds but they do respect lower TTLs:

Fri, Apr 12, 1:55 PM · Patch-For-Review, Acme-chief
Vgutierrez added a comment to T155359: wikiba.se should use HTTPS.

that's right, operations/dns manages wikiba.se

Fri, Apr 12, 1:09 PM · User-Addshore, wikiba.se website, Wikidata
Vgutierrez added a parent task for T131930: Set SPF (... -all) for toolserver.org: T220786: Add SPF record for non-canonical domains that are not parked.
Fri, Apr 12, 6:31 AM · cloud-services-team (Kanban), Traffic, Mail, Cloud-VPS, Patch-For-Review, Operations, DNS
Vgutierrez added a subtask for T220786: Add SPF record for non-canonical domains that are not parked: T131930: Set SPF (... -all) for toolserver.org.
Fri, Apr 12, 6:31 AM · Patch-For-Review, Operations, Traffic, DNS
Vgutierrez moved T220786: Add SPF record for non-canonical domains that are not parked from Triage to DNS Names on the Traffic board.
Fri, Apr 12, 6:27 AM · Patch-For-Review, Operations, Traffic, DNS
Vgutierrez triaged T220786: Add SPF record for non-canonical domains that are not parked as Normal priority.
Fri, Apr 12, 6:27 AM · Patch-For-Review, Operations, Traffic, DNS
Vgutierrez created T220786: Add SPF record for non-canonical domains that are not parked.
Fri, Apr 12, 6:26 AM · Patch-For-Review, Operations, Traffic, DNS
Vgutierrez closed T210134: wikidata.org lacks SPF record as Resolved.

Fixed by T193408

Fri, Apr 12, 6:01 AM · Mail, Patch-For-Review, Wikidata, User-revi, Traffic, Operations, DNS
Vgutierrez closed T210134: wikidata.org lacks SPF record, a subtask of T193408: SPF record for canonical domains, as Resolved.
Fri, Apr 12, 6:01 AM · Patch-For-Review, Mail, Operations
Vgutierrez added a comment to T219414: acme-chief fails to issue certificates against LE staging environment.

I've manually added an artificial wait of 90 seconds between the dns-01 challenges have been validated on the acme-chief side and the confirmation to LE, and that apparently solves the issue. So I'm inclined to believe that there is some DNS caching issue on the LE staging environment side.

Fri, Apr 12, 5:45 AM · Patch-For-Review, Acme-chief

Thu, Apr 11

Vgutierrez added a comment to T219414: acme-chief fails to issue certificates against LE staging environment.

I've tested acme-chief 0.16 with T207461 already implemented in acmechief-test1001, it doesn't solve the issue.

Thu, Apr 11, 3:24 PM · Patch-For-Review, Acme-chief
Vgutierrez closed T207461: Validate DNS-01 challenges against every DNS server as Resolved.
Thu, Apr 11, 3:21 PM · Acme-chief
Vgutierrez committed rOSACf4e85294e286: debian: Fix changelog 0.16 entry (authored by Vgutierrez).
debian: Fix changelog 0.16 entry
Thu, Apr 11, 2:44 PM
Vgutierrez committed rOSACcefcf253ec24: debian: Add release 0.16 to changelog (authored by Vgutierrez).
debian: Add release 0.16 to changelog
Thu, Apr 11, 2:38 PM
Vgutierrez committed rOSACab936d77f339: Release 0.16 (authored by Vgutierrez).
Release 0.16
Thu, Apr 11, 2:33 PM
Vgutierrez committed rOSAC267463d32fdf: acme_requests: Validate dns-01 challenges against all the DNS servers (authored by Vgutierrez).
acme_requests: Validate dns-01 challenges against all the DNS servers
Thu, Apr 11, 2:33 PM
Vgutierrez committed rOSAC1b9ce513d7ef: Release 0.16 (authored by Vgutierrez).
Release 0.16
Thu, Apr 11, 2:18 PM
Vgutierrez committed rOSAC336b26f2406e: Release 0.15 (authored by Vgutierrez).
Release 0.15
Thu, Apr 11, 2:13 PM
Vgutierrez committed rOSACc7d707122b25: acme_requests: Validate dns-01 challenges against all the DNS servers (authored by Vgutierrez).
acme_requests: Validate dns-01 challenges against all the DNS servers
Thu, Apr 11, 1:56 PM
Vgutierrez committed rOSAC0f8ae91f7e61: acme_requests: Validate dns-01 challenges against all the DNS servers (authored by Vgutierrez).
acme_requests: Validate dns-01 challenges against all the DNS servers
Thu, Apr 11, 12:14 PM
Vgutierrez committed rOSAC94faa1b2c996: acme_requests: Validate dns-01 challenges against all the DNS servers (authored by Vgutierrez).
acme_requests: Validate dns-01 challenges against all the DNS servers
Thu, Apr 11, 12:14 PM

Wed, Apr 10

Vgutierrez claimed T207461: Validate DNS-01 challenges against every DNS server.
Wed, Apr 10, 3:05 PM · Acme-chief
Vgutierrez added a comment to T219414: acme-chief fails to issue certificates against LE staging environment.

Initial response by Let's Encrypt engineers point to that this could be related to T207461, aka for some reason gdnsd is not returning the proper TXT record. acme-chief currently validates it but only against one of the three authoritative DNS servers. We could increase our confidence by solving T207461

Wed, Apr 10, 3:04 PM · Patch-For-Review, Acme-chief
Vgutierrez changed the status of T219414: acme-chief fails to issue certificates against LE staging environment from Open to Stalled.

Reported to Let's Encrypt in https://community.letsencrypt.org/t/unable-to-issue-ecdsa-rsa-in-acmev2-staging-environment/90835

Wed, Apr 10, 1:36 PM · Patch-For-Review, Acme-chief
Vgutierrez added a comment to T219414: acme-chief fails to issue certificates against LE staging environment.

On a second attempt... some of the failing certificates have been issued successfully:

vgutierrez@acmechief-test1001:~$ fgrep tendril acme-chief-2nd.log
Apr 10 09:29:38 acmechief-test1001 acme-chief-backend[8612]: New configured certificates: {'ldap', 'ldap-labtest', 'lists', 'librenms', 'unified', 'icinga', 'cloudelastic', 'non-canonical-redirect-4', 'mx', 'non-canonical-redirect-1', 'non-canonical-redirect-2', 'wikibase', 'non-canonical-redirect-3', 'ldap-codfw1dev', 'pinkunicorn', 'archiva', 'netbox', 'dumps', 'tendril', 'gerrit', 'mirrors'}
Apr 10 09:29:43 acmechief-test1001 acme-chief-backend[8612]: Creating new certificate version 64488bf20668403186f1a7ceeab45615 for tendril
Apr 10 09:29:43 acmechief-test1001 acme-chief-backend[8612]: Creating initial self-signed certificate for tendril / ec-prime256v1
Apr 10 09:29:43 acmechief-test1001 acme-chief-backend[8612]: Pushing the new certificate for tendril
Apr 10 09:29:43 acmechief-test1001 acme-chief-backend[8612]: Waiting till tendril / rsa-2048 is generated to be able to push the new certificate
Apr 10 09:29:43 acmechief-test1001 acme-chief-backend[8612]: Creating initial self-signed certificate for tendril / rsa-2048
Apr 10 09:29:43 acmechief-test1001 acme-chief-backend[8612]: Pushing the new certificate for tendril
Apr 10 09:33:30 acmechief-test1001 acme-chief-backend[8612]: Handling new certificate event for tendril / ec-prime256v1
Apr 10 09:33:30 acmechief-test1001 acme-chief-backend[8612]: Creating new certificate version d2974d62fa8b4484b4e3eddced82aafd for tendril
Apr 10 09:33:30 acmechief-test1001 acme-chief-backend[8612]: Running subprocess ['/usr/local/bin/acme-chief-gdnsd-sync.py', '--remote-servers', 'authdns1001.wikimedia.org', 'authdns2001.wikimedia.org', 'multatuli.wikimedia.org', '--', '_acme-challenge.tendril.wikimedia.org', 'eZEj0891mrN6bdKf6Qg3ocaPdurvno6u0aKa3ZZLyws']
Apr 10 09:33:33 acmechief-test1001 acme-chief-backend[8612]: Handling pushed CSR event for tendril / ec-prime256v1
Apr 10 09:33:33 acmechief-test1001 acme-chief-backend[8612]: Handling validated challenges event for tendril / ec-prime256v1
Apr 10 09:33:33 acmechief-test1001 acme-chief-backend[8612]: Handling pushed challenges event for tendril / ec-prime256v1
Apr 10 09:33:34 acmechief-test1001 acme-chief-backend[8612]: Handling order finalized event for tendril / ec-prime256v1
Apr 10 09:33:36 acmechief-test1001 acme-chief-backend[8612]: Pushing the new certificate for tendril
Apr 10 09:33:36 acmechief-test1001 acme-chief-backend[8612]: Waiting till tendril / rsa-2048 is generated to be able to push the new certificate
Apr 10 09:33:36 acmechief-test1001 acme-chief-backend[8612]: Handling new certificate event for tendril / rsa-2048
Apr 10 09:33:36 acmechief-test1001 acme-chief-backend[8612]: Running subprocess ['/usr/local/bin/acme-chief-gdnsd-sync.py', '--remote-servers', 'authdns1001.wikimedia.org', 'authdns2001.wikimedia.org', 'multatuli.wikimedia.org', '--', '_acme-challenge.tendril.wikimedia.org', 'mngAKhYePDExCl80HhVcB97bRt64YoRWPq3O4vp4LiE']
Apr 10 09:33:38 acmechief-test1001 acme-chief-backend[8612]: Handling pushed CSR event for tendril / rsa-2048
Apr 10 09:33:38 acmechief-test1001 acme-chief-backend[8612]: Handling validated challenges event for tendril / rsa-2048
Apr 10 09:33:38 acmechief-test1001 acme-chief-backend[8612]: Handling pushed challenges event for tendril / rsa-2048
Apr 10 09:33:38 acmechief-test1001 acme-chief-backend[8612]: ACME Directory has rejected the challenge(s) for certificate tendril / rsa-2048
Apr 10 09:33:38 acmechief-test1001 acme-chief-backend[8612]: ACME directory has rejected the challenge(s) for order https://acme-staging-v02.api.letsencrypt.org/acme/order/7090084/30282144
--- OUTPUT OMITTED ---
Apr 10 09:36:23 acmechief-test1001 acme-chief-backend[8612]: Number of certificates per status: Counter({'SELF_SIGNED': 34, 'VALID': 10})
Apr 10 09:39:30 acmechief-test1001 acme-chief-backend[8612]: Handling new certificate event for tendril / ec-prime256v1
Apr 10 09:39:30 acmechief-test1001 acme-chief-backend[8612]: Creating new certificate version e182b50179334c15b9e344fdf0cfd914 for tendril
Apr 10 09:39:31 acmechief-test1001 acme-chief-backend[8612]: Running subprocess ['/usr/local/bin/acme-chief-gdnsd-sync.py', '--remote-servers', 'authdns1001.wikimedia.org', 'authdns2001.wikimedia.org', 'multatuli.wikimedia.org', '--', '_acme-challenge.tendril.wikimedia.org', 'hCDCpO1qpnPTWp-y6MJ5bo_BGzbuyjY0vXImO-VFDnU']
Apr 10 09:39:33 acmechief-test1001 acme-chief-backend[8612]: Handling pushed CSR event for tendril / ec-prime256v1
Apr 10 09:39:33 acmechief-test1001 acme-chief-backend[8612]: Handling validated challenges event for tendril / ec-prime256v1
Apr 10 09:39:33 acmechief-test1001 acme-chief-backend[8612]: Handling pushed challenges event for tendril / ec-prime256v1
Apr 10 09:39:35 acmechief-test1001 acme-chief-backend[8612]: Handling order finalized event for tendril / ec-prime256v1
Apr 10 09:39:36 acmechief-test1001 acme-chief-backend[8612]: Pushing the new certificate for tendril
Apr 10 09:39:36 acmechief-test1001 acme-chief-backend[8612]: Waiting till tendril / rsa-2048 is generated to be able to push the new certificate
Apr 10 09:39:36 acmechief-test1001 acme-chief-backend[8612]: Handling new certificate event for tendril / rsa-2048
Apr 10 09:39:37 acmechief-test1001 acme-chief-backend[8612]: Running subprocess ['/usr/local/bin/acme-chief-gdnsd-sync.py', '--remote-servers', 'authdns1001.wikimedia.org', 'authdns2001.wikimedia.org', 'multatuli.wikimedia.org', '--', '_acme-challenge.tendril.wikimedia.org', 'COvB8oOd_FM8sVmyLXmKBIzd0HdQo0e-ZQZ2PFN5jDY']
Apr 10 09:39:39 acmechief-test1001 acme-chief-backend[8612]: Handling pushed CSR event for tendril / rsa-2048
Apr 10 09:39:39 acmechief-test1001 acme-chief-backend[8612]: Handling validated challenges event for tendril / rsa-2048
Apr 10 09:39:39 acmechief-test1001 acme-chief-backend[8612]: Handling pushed challenges event for tendril / rsa-2048
Apr 10 09:39:43 acmechief-test1001 acme-chief-backend[8612]: Handling order finalized event for tendril / rsa-2048
Apr 10 09:39:44 acmechief-test1001 acme-chief-backend[8612]: Pushing the new certificate for tendril
Wed, Apr 10, 9:49 AM · Patch-For-Review, Acme-chief
Vgutierrez added a comment to T219414: acme-chief fails to issue certificates against LE staging environment.

After letting the staging environment get the whole certificate list, some are being issued successfully and some fail with the same behaviour as the one described in this task's description:

Apr 10 09:36:23 acmechief-test1001 acme-chief-backend[8612]: Number of certificates per status: Counter({'SELF_SIGNED': 34, 'VALID': 10})
Wed, Apr 10, 9:37 AM · Patch-For-Review, Acme-chief
Vgutierrez lowered the priority of T219414: acme-chief fails to issue certificates against LE staging environment from High to Normal.

I'm unable to reproduce this with the brand new acme-chief staging environment.. both certificates have been issued successfully for apt.wikimedia.org:

Apr 10 08:57:16 acmechief-test1001 acme-chief-backend[8612]: Missing/invalid DNS zone updater CMD timeout, using the default one: 60.00
Apr 10 08:57:16 acmechief-test1001 acme-chief-backend[8612]: Number of certificates per status: Counter({'SELF_SIGNED': 2})
Apr 10 08:57:16 acmechief-test1001 acme-chief-backend[8612]: Starting main loop...
Apr 10 08:57:16 acmechief-test1001 acme-chief-backend[8612]: Handling new certificate event for apt / ec-prime256v1
Apr 10 08:57:17 acmechief-test1001 acme-chief-backend[8612]: Triggering DNS zone update...
Apr 10 08:57:17 acmechief-test1001 acme-chief-backend[8612]: Running subprocess ['/usr/local/bin/acme-chief-gdnsd-sync.py', '--remote-servers', 'authdns1001.wikimedia.org', 'authdns2001.wikimedia.org', 'multatuli.wikimedia.org', '--', '_acme-challenge.apt.wikimedia.org', 'eJP2fR8wwwFFe1wi1yv1eBcM4VHHItOZinsOpy8Lv38']
Apr 10 08:57:19 acmechief-test1001 acme-chief-backend[8612]: Handling pushed CSR event for apt / ec-prime256v1
Apr 10 08:57:19 acmechief-test1001 acme-chief-backend[8612]: Handling validated challenges event for apt / ec-prime256v1
Apr 10 08:57:20 acmechief-test1001 acme-chief-backend[8612]: Handling pushed challenges event for apt / ec-prime256v1
Apr 10 08:57:23 acmechief-test1001 acme-chief-backend[8612]: Handling order finalized event for apt / ec-prime256v1
Apr 10 08:57:24 acmechief-test1001 acme-chief-backend[8612]: Pushing the new certificate for apt
Apr 10 08:57:24 acmechief-test1001 acme-chief-backend[8612]: Waiting till apt / rsa-2048 is generated to be able to push the new certificate
Apr 10 08:57:24 acmechief-test1001 acme-chief-backend[8612]: Handling new certificate event for apt / rsa-2048
Apr 10 08:57:25 acmechief-test1001 acme-chief-backend[8612]: Triggering DNS zone update...
Apr 10 08:57:25 acmechief-test1001 acme-chief-backend[8612]: Running subprocess ['/usr/local/bin/acme-chief-gdnsd-sync.py', '--remote-servers', 'authdns1001.wikimedia.org', 'authdns2001.wikimedia.org', 'multatuli.wikimedia.org', '--', '_acme-challenge.apt.wikimedia.org', '79x-XzQ5fiRlLsOYEyZW9yPdzz4BhZXt4AwMKiGCpuc']
Apr 10 08:57:27 acmechief-test1001 acme-chief-backend[8612]: Handling pushed CSR event for apt / rsa-2048
Apr 10 08:57:27 acmechief-test1001 acme-chief-backend[8612]: Handling validated challenges event for apt / rsa-2048
Apr 10 08:57:27 acmechief-test1001 acme-chief-backend[8612]: Handling pushed challenges event for apt / rsa-2048
Apr 10 08:57:29 acmechief-test1001 acme-chief-backend[8612]: Handling order finalized event for apt / rsa-2048
Apr 10 08:57:30 acmechief-test1001 acme-chief-backend[8612]: Pushing the new certificate for apt
root@acmechief-test1001:/var/lib/acme-chief/certs/apt/live# openssl x509 -text -noout -in ec-prime256v1.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            fa:4b:81:27:71:d0:f8:d9:3e:5b:40:24:2d:57:50:47:4b:d3
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = Fake LE Intermediate X1
        Validity
            Not Before: Apr 10 07:57:21 2019 GMT
            Not After : Jul  9 07:57:21 2019 GMT
        Subject: CN = apt.wikimedia.org
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:09:1b:00:31:21:26:61:dd:9d:a4:c2:c2:99:3a:
                    91:77:48:cb:d5:62:a9:05:e5:ab:ce:79:71:b9:e7:
                    3a:bc:b0:63:8d:94:77:ff:95:6a:f3:b9:f5:97:de:
                    98:77:c0:73:29:f1:9d:8c:ac:c2:ee:a4:ca:0d:5c:
                    91:34:a3:7f:3a
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                1C:0E:07:4C:C5:10:9A:CD:17:2B:FA:32:28:51:F5:E6:B4:38:0E:B4
            X509v3 Authority Key Identifier:
                keyid:C0:CC:03:46:B9:58:20:CC:5C:72:70:F3:E1:2E:CB:20:A6:F5:68:3A
Wed, Apr 10, 9:10 AM · Patch-For-Review, Acme-chief
Vgutierrez closed T220378: Provide an staging environment for acme-chief as Resolved.
Wed, Apr 10, 9:01 AM · Operations, Traffic, Acme-chief
Vgutierrez closed T220378: Provide an staging environment for acme-chief, a subtask of T219414: acme-chief fails to issue certificates against LE staging environment, as Resolved.
Wed, Apr 10, 9:01 AM · Patch-For-Review, Acme-chief
Vgutierrez changed the status of T220359: Benefit from acme-chief features in acme-chief clients from Open to Stalled.
Wed, Apr 10, 8:27 AM · Operations, Traffic, Acme-chief
Vgutierrez moved T220518: acme-chief: Validate that configured certificates can be actually issued from Triage to TLS on the Traffic board.
Wed, Apr 10, 7:16 AM · Patch-For-Review, Acme-chief, HTTPS, Traffic, Operations

Tue, Apr 9

Vgutierrez triaged T220518: acme-chief: Validate that configured certificates can be actually issued as Normal priority.
Tue, Apr 9, 3:14 PM · Patch-For-Review, Acme-chief, HTTPS, Traffic, Operations
Vgutierrez created T220518: acme-chief: Validate that configured certificates can be actually issued.
Tue, Apr 9, 3:14 PM · Patch-For-Review, Acme-chief, HTTPS, Traffic, Operations
Vgutierrez added a comment to T209707: tagged_interface sometimes exceeds IFNAMSIZ.

So we are effectively stripping the common part of every ethernet interface name: en. We don't lose a bit of information. I don't see the problem to be honest.

Tue, Apr 9, 10:30 AM · Patch-For-Review, Traffic, Operations
Vgutierrez created P8375 (An Untitled Masterwork).
Tue, Apr 9, 8:26 AM
Vgutierrez created P8374 (An Untitled Masterwork).
Tue, Apr 9, 8:21 AM
Vgutierrez updated subscribers of T209707: tagged_interface sometimes exceeds IFNAMSIZ.

so this is currently a blocker on cloudvirt1024.eqiad.wmnet for @Andrew. The suggested approach by @faidon of using systemd >= 239 doesn't seem to work. I've rebased https://gerrit.wikimedia.org/r/474272 and ran pcc against our whole fleet of lvs and cloudvirt1024, it shows the expected changes:

  • lvs: NOOP
  • cloudvirt1024: trimmed tagged network interface names: enp175s0f1d1.1105 -> p175s0f1d1.1105
Tue, Apr 9, 7:11 AM · Patch-For-Review, Traffic, Operations
Vgutierrez added a comment to T209707: tagged_interface sometimes exceeds IFNAMSIZ.

so systemd 241 shows the same behaviour as 232 in lvs2010:

vgutierrez@lvs2010:~$ apt-cache policy systemd
systemd:
  Installed: 241-1~bpo9+1
  Candidate: 241-1~bpo9+1
  Version table:
 *** 241-1~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
        100 /var/lib/dpkg/status
     232-25+deb9u11 500
        500 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
     232-25+deb9u8 500
        500 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages
vgutierrez@lvs2010:~$ sudo dmesg |grep rename
[    5.107385] bnxt_en 0000:3b:00.0 enp59s0f0: renamed from eth0
[    5.297922] bnxt_en 0000:af:00.0 enp175s0f0: renamed from eth2
[    5.320634] bnxt_en 0000:3b:00.1 enp59s0f1d1: renamed from eth1
[    5.404452] bnxt_en 0000:af:00.1 enp175s0f1d1: renamed from eth3
Tue, Apr 9, 6:52 AM · Patch-For-Review, Traffic, Operations
Vgutierrez added a comment to T209707: tagged_interface sometimes exceeds IFNAMSIZ.

that's actually pretty easy to test in lvs2010 (currently a spare system):

vgutierrez@lvs2010:~$ apt-cache policy systemd
systemd:
  Installed: 232-25+deb9u9
  Candidate: 232-25+deb9u11
  Version table:
     241-1~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
     232-25+deb9u11 500
        500 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
 *** 232-25+deb9u9 100
        100 /var/lib/dpkg/status
     232-25+deb9u8 500
        500 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages
Tue, Apr 9, 6:36 AM · Patch-For-Review, Traffic, Operations

Mon, Apr 8

Vgutierrez updated the task description for T193408: SPF record for canonical domains.
Mon, Apr 8, 2:32 PM · Patch-For-Review, Mail, Operations
Vgutierrez closed T217002: Make sure that services available for NDA-only users are using strong TLS ciphersuites as Resolved.
Mon, Apr 8, 2:00 PM · Operations, Traffic, HTTPS
Vgutierrez closed T217002: Make sure that services available for NDA-only users are using strong TLS ciphersuites, a subtask of T104681: HTTPS Plans (tracking / high-level info), as Resolved.
Mon, Apr 8, 2:00 PM · Tracking-Neverending, Operations, Traffic, HTTPS
Vgutierrez updated the task description for T220383: Evaluate ATS TLS stack.
Mon, Apr 8, 1:39 PM · Traffic, Operations
Vgutierrez moved T220383: Evaluate ATS TLS stack from Triage to TLS on the Traffic board.
Mon, Apr 8, 1:36 PM · Traffic, Operations
Vgutierrez triaged T220383: Evaluate ATS TLS stack as Normal priority.
Mon, Apr 8, 1:36 PM · Traffic, Operations
Vgutierrez created T220383: Evaluate ATS TLS stack.
Mon, Apr 8, 1:36 PM · Traffic, Operations
Vgutierrez triaged T220378: Provide an staging environment for acme-chief as Normal priority.
Mon, Apr 8, 12:32 PM · Operations, Traffic, Acme-chief
Vgutierrez created T220378: Provide an staging environment for acme-chief.
Mon, Apr 8, 12:31 PM · Operations, Traffic, Acme-chief
Vgutierrez moved T220359: Benefit from acme-chief features in acme-chief clients from Triage to TLS on the Traffic board.
Mon, Apr 8, 10:51 AM · Operations, Traffic, Acme-chief
Vgutierrez triaged T220359: Benefit from acme-chief features in acme-chief clients as Normal priority.
Mon, Apr 8, 10:51 AM · Operations, Traffic, Acme-chief
Vgutierrez created T220359: Benefit from acme-chief features in acme-chief clients.
Mon, Apr 8, 10:50 AM · Operations, Traffic, Acme-chief

Fri, Apr 5

Vgutierrez claimed T219414: acme-chief fails to issue certificates against LE staging environment.
Fri, Apr 5, 12:57 PM · Patch-For-Review, Acme-chief
Vgutierrez claimed T170567: Support TLSv1.3.
Fri, Apr 5, 12:54 PM · Goal, Patch-For-Review, Traffic, Operations

Thu, Apr 4

Vgutierrez updated subscribers of T219856: wicipediacymraeg.org is on clientHold.

Then IMHO we should get rid of it at operations/dns and in redirects.dat in operations/puppet, what are your thoughts @BBlack?

Thu, Apr 4, 12:22 PM · Patch-For-Review, Domains, Operations, Traffic

Wed, Apr 3

Vgutierrez added a comment to T219856: wicipediacymraeg.org is on clientHold.

regarding TLS wicipediacymraeg.org should benefit from T133548 that should be implemented during Q4

Wed, Apr 3, 7:29 AM · Patch-For-Review, Domains, Operations, Traffic

Tue, Apr 2

Vgutierrez claimed T133548: Create a secure redirect service for large count of non-canonical / junk domains.
Tue, Apr 2, 3:13 PM · Goal, Patch-For-Review, HTTPS, Operations, Traffic
Vgutierrez updated the task description for T213705: Deploy managed LetsEncrypt certs for all public use-cases.
Tue, Apr 2, 9:25 AM · Patch-For-Review, Traffic, Operations, Acme-chief, Goal
Vgutierrez closed T213705: Deploy managed LetsEncrypt certs for all public use-cases as Resolved.

The non-canonical certs have been issued successfully:

Tue, Apr 2, 9:25 AM · Patch-For-Review, Traffic, Operations, Acme-chief, Goal
Vgutierrez created P8327 testing the final list of SNIs for the non-canonical redirect list.
Tue, Apr 2, 9:03 AM · Acme-chief
Vgutierrez triaged T219856: wicipediacymraeg.org is on clientHold as Normal priority.
Tue, Apr 2, 8:47 AM · Patch-For-Review, Domains, Operations, Traffic
Vgutierrez moved T219856: wicipediacymraeg.org is on clientHold from Triage to DNS Names on the Traffic board.
Tue, Apr 2, 8:47 AM · Patch-For-Review, Domains, Operations, Traffic
Vgutierrez created T219856: wicipediacymraeg.org is on clientHold.
Tue, Apr 2, 8:47 AM · Patch-For-Review, Domains, Operations, Traffic
Vgutierrez edited P8326 checking non-canonical SNIs after merging Ib064d25b82cdc1fcf9372a7881d8caece2433507.
Tue, Apr 2, 8:21 AM
Vgutierrez created P8326 checking non-canonical SNIs after merging Ib064d25b82cdc1fcf9372a7881d8caece2433507.
Tue, Apr 2, 8:17 AM
Vgutierrez created P8325 checking non-canonical SNIs.
Tue, Apr 2, 8:10 AM · DNS, Acme-chief

Mon, Apr 1

Vgutierrez updated the task description for T213705: Deploy managed LetsEncrypt certs for all public use-cases.
Mon, Apr 1, 4:50 PM · Patch-For-Review, Traffic, Operations, Acme-chief, Goal
Vgutierrez updated the task description for T213705: Deploy managed LetsEncrypt certs for all public use-cases.
Mon, Apr 1, 2:50 PM · Patch-For-Review, Traffic, Operations, Acme-chief, Goal
Vgutierrez triaged T219765: Implement server-side OCSP stapling as Normal priority.
Mon, Apr 1, 10:56 AM · Patch-For-Review, Acme-chief
Vgutierrez created T219765: Implement server-side OCSP stapling.
Mon, Apr 1, 9:29 AM · Patch-For-Review, Acme-chief

Thu, Mar 28

Vgutierrez added a comment to T213705: Deploy managed LetsEncrypt certs for all public use-cases.
vgutierrez@acmechief1001:~$ sudo -i openssl x509 -text -noout -in /var/lib/acme-chief/certs/wikibase/live/rsa-2048.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:09:1c:d8:1f:6b:2f:dc:9f:40:ac:df:f8:dd:16:c6:22:eb
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
        Validity
            Not Before: Mar 28 13:09:23 2019 GMT
            Not After : Jun 26 13:09:23 2019 GMT
        Subject: CN = wikiba.se
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:e4:24:39:5e:d0:74:ea:b9:5c:92:6a:e1:f4:d4:
                    36:16:5f:47:69:67:f2:ec:c7:63:51:17:b4:d6:84:
                    17:88:88:f6:15:c4:88:44:28:a1:e8:64:fa:cf:e9:
                    4a:55:6e:40:c3:25:d7:c6:3d:e9:e5:34:3e:5e:38:
                    71:31:c5:96:2e:d1:3c:85:2f:ce:e6:39:53:c3:a9:
                    cc:86:d6:48:c5:c9:c1:e1:fb:32:59:9f:70:ec:c7:
                    5c:3e:5e:07:ce:9f:75:45:0f:b1:23:08:5b:8f:3c:
                    2f:ae:26:c3:f4:af:63:db:1c:07:b3:03:4e:b3:de:
                    d2:ef:e4:c4:9b:3c:ae:84:50:cd:fc:d3:0d:98:e2:
                    76:1d:3f:25:bb:a7:eb:1e:22:08:0c:cd:43:ef:92:
                    a4:eb:79:9e:49:c8:4d:19:73:ef:00:df:4c:41:58:
                    2f:0d:c6:1c:10:f9:8c:bd:68:07:b7:a3:ba:75:95:
                    30:d0:ec:f5:3c:62:c8:69:e2:8e:5c:37:db:87:32:
                    8a:16:ca:81:6a:c7:72:1a:fc:5d:f1:0b:24:33:1c:
                    67:3d:39:05:73:bc:d4:5b:d9:45:1c:d3:11:36:9f:
                    3c:3c:15:d3:15:11:e4:1e:df:91:ee:2a:29:01:79:
                    b6:71:c7:7e:9c:bf:5b:a9:88:89:f9:2e:81:58:cf:
                    52:f3
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                EE:EE:D6:A3:82:15:C6:CE:A5:C5:98:50:5E:45:51:FF:88:C4:46:65
            X509v3 Authority Key Identifier:
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1
Thu, Mar 28, 2:17 PM · Patch-For-Review, Traffic, Operations, Acme-chief, Goal
Krenair awarded T213705: Deploy managed LetsEncrypt certs for all public use-cases a Party Time token.
Thu, Mar 28, 9:12 AM · Patch-For-Review, Traffic, Operations, Acme-chief, Goal
Vgutierrez added a comment to T213705: Deploy managed LetsEncrypt certs for all public use-cases.
root@acmechief1001:~# openssl x509 -text -noout -in /var/lib/acme-chief/certs/unified/live/rsa-2048.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:92:37:dd:0b:55:1a:07:fc:2c:b9:19:6c:c4:bd:ec:0f:c1
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
        Validity
            Not Before: Mar 28 07:10:05 2019 GMT
            Not After : Jun 26 07:10:05 2019 GMT
        Subject: CN = *.wikipedia.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:db:5c:75:9f:45:03:47:ae:0b:6f:70:7e:da:f3:
                    f9:cc:da:f8:4f:c5:c7:cd:b9:93:64:1a:47:c9:eb:
                    f2:58:9f:81:cc:d5:de:b5:a8:20:86:4b:9f:c7:4d:
                    4c:9b:39:82:22:34:17:ce:6f:6c:d3:b3:dd:2f:c1:
                    54:34:6f:99:b8:49:92:fc:09:09:a7:cd:61:9a:17:
                    80:83:95:47:da:54:d0:e5:fe:fd:75:37:a7:52:01:
                    73:f5:57:1e:d7:2b:d4:d6:c3:e3:62:60:cc:c4:5c:
                    d8:1f:58:cd:b0:0a:98:9b:19:43:90:3f:13:95:f3:
                    f4:b4:2e:c3:79:27:4f:f5:e8:85:91:02:8f:16:69:
                    6d:e9:38:17:f9:77:1d:52:20:c0:71:db:3a:24:77:
                    b8:be:d9:e4:9a:b7:57:2a:43:8e:7f:bb:a5:33:be:
                    4d:e5:12:fb:9b:50:d9:7f:aa:aa:08:04:80:6e:e3:
                    e1:14:46:c4:3b:3d:d5:29:ff:87:b8:e5:02:88:70:
                    86:4a:4a:54:6b:d6:29:1b:18:90:2e:f0:7f:fb:33:
                    a5:0a:44:96:2a:9a:37:f8:41:d8:da:de:28:a2:9c:
                    f0:b0:0d:37:76:ab:a4:d7:f3:ec:a7:9d:2c:c6:cb:
                    0f:7f:0e:3f:e7:e7:36:2e:b3:41:c2:45:05:12:56:
                    f7:7b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                A0:3C:D2:C9:4D:5E:F7:02:9C:84:60:9A:25:0D:E3:9A:AA:88:82:E8
            X509v3 Authority Key Identifier:
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1
Thu, Mar 28, 8:14 AM · Patch-For-Review, Traffic, Operations, Acme-chief, Goal