Vgutierrez (Valentín Gutiérrez)
Traffic Security Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Feb 12 2018, 9:51 AM (43 w, 7 h)
Availability
Available
IRC Nick
vgutierrez
LDAP User
Vgutierrez
MediaWiki User
Unknown

Recent Activity

Fri, Nov 30

Vgutierrez changed the status of T209337: lvs2006 crashed into (what it seems) an unrecoverable state from Open to Stalled.

lvs2010 replacement is currently blocked by T203194

Fri, Nov 30, 3:39 PM · Patch-For-Review, ops-codfw, Operations, Traffic

Thu, Nov 29

Vgutierrez updated the task description for T207050: Migrate most standard public TLS certificates to CertCentral issuance.
Thu, Nov 29, 9:09 AM · Patch-For-Review, Traffic, Operations

Wed, Nov 28

Vgutierrez updated the task description for T207050: Migrate most standard public TLS certificates to CertCentral issuance.
Wed, Nov 28, 3:38 PM · Patch-For-Review, Traffic, Operations
Vgutierrez closed T209976: puppet still restarts certcentral on config changes instead of reloading it as Resolved.

After applying change XXXX certcentral gets reloaded instead of restarted:

Nov 28 14:59:24 certcentral1001 systemd[1]: Reloading Central Certificates Service.
Nov 28 14:59:24 certcentral1001 certcentral-backend[8314]: SIGHUP received
Nov 28 14:59:24 certcentral1001 systemd[1]: Reloaded Central Certificates Service.
Nov 28 14:59:24 certcentral1001 certcentral-backend[8314]: Missing/invalid DNS zone updater CMD timeout, using the default one: 60.00
Nov 28 14:59:24 certcentral1001 certcentral-backend[8314]: New configured certificates: {'dumps'}
Nov 28 14:59:24 certcentral1001 certcentral-backend[8314]: Number of certificates per status: Counter({'VALID': 14, 'INITIAL': 2})
.....
Wed, Nov 28, 3:01 PM · Certcentral
Vgutierrez updated the task description for T207050: Migrate most standard public TLS certificates to CertCentral issuance.
Wed, Nov 28, 9:36 AM · Patch-For-Review, Traffic, Operations
Vgutierrez triaged T209980: certcentral crashes on network errors as Normal priority.
Wed, Nov 28, 7:36 AM · Certcentral

Tue, Nov 27

Vgutierrez updated the task description for T207050: Migrate most standard public TLS certificates to CertCentral issuance.
Tue, Nov 27, 2:29 PM · Patch-For-Review, Traffic, Operations

Mon, Nov 26

Vgutierrez updated the task description for T207050: Migrate most standard public TLS certificates to CertCentral issuance.
Mon, Nov 26, 3:33 PM · Patch-For-Review, Traffic, Operations
Vgutierrez committed rOSCCd58b2b391b18: acme_requests: Handle TCP/HTTPS errors (authored by Vgutierrez).
acme_requests: Handle TCP/HTTPS errors
Mon, Nov 26, 9:06 AM

Fri, Nov 23

Vgutierrez updated the task description for T207050: Migrate most standard public TLS certificates to CertCentral issuance.
Fri, Nov 23, 3:40 PM · Patch-For-Review, Traffic, Operations

Thu, Nov 22

Vgutierrez updated the task description for T207050: Migrate most standard public TLS certificates to CertCentral issuance.
Thu, Nov 22, 4:18 PM · Patch-For-Review, Traffic, Operations
Vgutierrez triaged T209976: puppet still restarts certcentral on config changes instead of reloading it as Normal priority.
Thu, Nov 22, 10:00 AM · Certcentral

Tue, Nov 20

Vgutierrez created T209980: certcentral crashes on network errors.
Tue, Nov 20, 5:12 PM · Certcentral
Vgutierrez created T209976: puppet still restarts certcentral on config changes instead of reloading it.
Tue, Nov 20, 5:09 PM · Certcentral
Vgutierrez closed T208859: certcentral: keep track of orders and authorizations IDs when issuing certificates as Resolved.
Tue, Nov 20, 4:58 PM · Certcentral
Vgutierrez closed T208970: certcentral wrongly handles acme.errors.ValidationError exception as Resolved.
Tue, Nov 20, 4:57 PM · Certcentral
Vgutierrez closed T208948: certcentral "wrongly" assumes that a new order always implies fulfilling new challenges as Resolved.
Tue, Nov 20, 4:56 PM · Patch-For-Review, Certcentral
Vgutierrez closed T208967: Avoid using acme.client poll_and_finalize() method as Resolved.
Tue, Nov 20, 4:55 PM · Certcentral
Vgutierrez closed T209856: Deploy a certcentral managed TLS certificate for librenms as Resolved.
Tue, Nov 20, 4:05 PM · Certcentral, Traffic, Operations
Vgutierrez closed T209856: Deploy a certcentral managed TLS certificate for librenms, a subtask of T207050: Migrate most standard public TLS certificates to CertCentral issuance, as Resolved.
Tue, Nov 20, 4:05 PM · Patch-For-Review, Traffic, Operations

Mon, Nov 19

Vgutierrez added a comment to T209856: Deploy a certcentral managed TLS certificate for librenms.

looking good:

vgutierrez@neodymium:~$ sudo cumin netmon1002.wikimedia.org,netmon2001.wikimedia.org 'sha256sum /etc/centralcerts/librenms.rsa-2048.crt'
2 hosts will be targeted:
netmon[1002,2001].wikimedia.org
Confirm to continue [y/n]? y
===== NODE GROUP =====
(2) netmon[1002,2001].wikimedia.org
----- OUTPUT of 'sha256sum /etc/c...nms.rsa-2048.crt' -----
e9828e3c7261ea693cb010479c978715234228ea0d1cd5f85ee31a5ac96ff673  /etc/centralcerts/librenms.rsa-2048.crt
================
PASS:  |#######################################################################################################################################################| 100% (2/2) [00:00<00:00,  3.01hosts/s]
FAIL:  |                                                                                                                                                               |   0% (0/2) [00:00<?, ?hosts/s]
100.0% (2/2) success ratio (>= 100.0% threshold) for command: 'sha256sum /etc/c...nms.rsa-2048.crt'.
100.0% (2/2) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
vgutierrez@neodymium:~$ sudo cumin netmon1002.wikimedia.org,netmon2001.wikimedia.org 'openssl x509 -text -noout -in /etc/centralcerts/librenms.rsa-2048.crt'
2 hosts will be targeted:
netmon[1002,2001].wikimedia.org
Confirm to continue [y/n]? y
===== NODE GROUP =====
(2) netmon[1002,2001].wikimedia.org
----- OUTPUT of 'openssl x509 -te...nms.rsa-2048.crt' -----
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:a7:17:10:ae:0a:3e:dc:a6:e9:3b:f4:20:88:33:4c:dd:3a
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
        Validity
            Not Before: Nov 19 15:50:45 2018 GMT
            Not After : Feb 17 15:50:45 2019 GMT
        Subject: CN = librenms.wikimedia.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ad:40:86:b8:4c:26:8f:7d:88:30:0a:73:e1:f2:
                    51:cd:0b:e9:64:c2:5a:02:4a:bb:8c:ff:53:07:43:
                    ce:99:7c:35:37:f4:90:ef:43:82:ab:da:8c:24:e0:
                    7f:b1:1b:cf:7e:07:2e:42:e6:f3:88:96:ed:25:79:
                    d8:a6:fb:cf:83:0b:3d:37:bd:8c:2f:32:42:42:5e:
                    9f:aa:7e:9f:e8:95:c3:07:49:c0:c0:b2:d9:4a:21:
                    2f:3a:9d:8d:74:a8:36:91:8b:b9:41:df:5f:12:52:
                    c4:1e:31:4c:06:4b:e8:ec:be:04:48:28:ef:67:ac:
                    db:b0:68:4c:d4:c9:04:ba:f7:ca:86:b4:61:ab:ba:
                    ee:79:5e:08:c2:af:08:99:12:41:de:f5:68:73:6b:
                    5f:b8:86:c0:f2:27:91:f6:7a:33:5a:f0:54:b1:30:
                    e8:01:c5:66:8a:99:87:7d:5d:f4:8b:2b:a9:18:ac:
                    18:7f:ba:7f:56:c9:4c:c5:4d:83:17:a5:60:ee:36:
                    61:2f:b5:5d:b3:a1:9c:64:a2:e9:0b:f9:65:18:51:
                    28:4a:52:e9:2a:12:6c:73:32:d9:e3:fb:cc:52:de:
                    56:ec:09:25:e8:0d:d9:3c:4c:8c:ef:51:e9:f0:4d:
                    6e:d9:20:ff:70:61:3d:cc:a4:be:10:92:5d:03:30:
                    18:25
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                8B:9A:37:A7:0B:65:75:43:F8:60:74:6F:0D:E0:AA:C0:AC:D2:5C:93
            X509v3 Authority Key Identifier:
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1
Mon, Nov 19, 5:24 PM · Certcentral, Traffic, Operations
Vgutierrez moved T209856: Deploy a certcentral managed TLS certificate for librenms from Triage to TLS on the Traffic board.
Mon, Nov 19, 3:52 PM · Certcentral, Traffic, Operations
Vgutierrez added a project to T209856: Deploy a certcentral managed TLS certificate for librenms: Certcentral.
Mon, Nov 19, 3:52 PM · Certcentral, Traffic, Operations
Vgutierrez triaged T209856: Deploy a certcentral managed TLS certificate for librenms as Normal priority.
Mon, Nov 19, 3:52 PM · Certcentral, Traffic, Operations

Fri, Nov 16

Vgutierrez created P7818 (An Untitled Masterwork).
Fri, Nov 16, 4:27 PM
Vgutierrez created T209707: tagged_interface sometimes exceeds IFNAMSIZ.
Fri, Nov 16, 3:16 PM · Patch-For-Review, Traffic, Operations
Vgutierrez closed T209475: store non-config files in /var/lib/certcentral as Resolved.
Fri, Nov 16, 8:11 AM · Patch-For-Review, Certcentral

Thu, Nov 15

Vgutierrez committed rOSCC925f30781e3e: debian: Add release 0.7 to changelog (authored by Vgutierrez).
debian: Add release 0.7 to changelog
Thu, Nov 15, 3:27 PM
Vgutierrez committed rOSCC1b703fe3cf77: Release 0.7 (authored by Vgutierrez).
Release 0.7
Thu, Nov 15, 3:24 PM
Vgutierrez committed rOSCCf88358d69e2e: acme_requests: Fix finalize_order() exception handling (authored by Vgutierrez).
acme_requests: Fix finalize_order() exception handling
Thu, Nov 15, 3:24 PM
Vgutierrez committed rOSCCdc9bbe53740b: Release 0.7 (authored by Vgutierrez).
Release 0.7
Thu, Nov 15, 3:12 PM
Vgutierrez committed rOSCCc5d7c0d1d12e: certcentral: split base path in config and certificates path (authored by Vgutierrez).
certcentral: split base path in config and certificates path
Thu, Nov 15, 10:35 AM
Vgutierrez committed rOSCCefe0e7cc41be: debian: Take into account /var/lib/certcentral (authored by Vgutierrez).
debian: Take into account /var/lib/certcentral
Thu, Nov 15, 10:35 AM

Wed, Nov 14

Vgutierrez created T209475: store non-config files in /var/lib/certcentral.
Wed, Nov 14, 11:49 AM · Patch-For-Review, Certcentral
Vgutierrez closed T209161: switch certcentral servers from active/active to active/passive as Resolved.
Wed, Nov 14, 11:08 AM · Patch-For-Review, Certcentral

Tue, Nov 13

Vgutierrez added a comment to T196560: rack/setup/install LVS200[7-10].

@Papaul enp59s0f0

Tue, Nov 13, 7:49 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Vgutierrez added a comment to T196560: rack/setup/install LVS200[7-10].

@Papaul so at least in lvs2010, debian installer seems to think that enp175s0f0 is the first NIC, the mac addr is 00:0a:f7:f0:0c:10.
in lvs2009 the mac address is 00:0a:f7:f0:0b:70

Tue, Nov 13, 7:20 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Vgutierrez added a comment to T196560: rack/setup/install LVS200[7-10].

@Papaul we need to re-wire lvs2009 & lvs2010 to connect the first interface (enp175s0f0) to the main row for each server.

Tue, Nov 13, 6:33 PM · Patch-For-Review, ops-codfw, Traffic, Operations
Vgutierrez added a comment to T209337: lvs2006 crashed into (what it seems) an unrecoverable state.

we will be replacing lvs2006 with lvs2010

Tue, Nov 13, 4:28 PM · Patch-For-Review, ops-codfw, Operations, Traffic
Vgutierrez added a comment to T209337: lvs2006 crashed into (what it seems) an unrecoverable state.

The system is online since 07:30 UTC

Tue, Nov 13, 3:34 PM · Patch-For-Review, ops-codfw, Operations, Traffic

Nov 9 2018

Vgutierrez triaged T209161: switch certcentral servers from active/active to active/passive as High priority.
Nov 9 2018, 4:29 PM · Patch-For-Review, Certcentral
Vgutierrez created T209161: switch certcentral servers from active/active to active/passive.
Nov 9 2018, 4:29 PM · Patch-For-Review, Certcentral
Vgutierrez committed rOSCC4ef41c70e1f0: acme_requests: Fix finalize_order() exception handling (authored by Vgutierrez).
acme_requests: Fix finalize_order() exception handling
Nov 9 2018, 3:44 PM
Vgutierrez added a comment to T208948: certcentral "wrongly" assumes that a new order always implies fulfilling new challenges.

A second attempt, 10 minutes shows that certcentral1001 is able to fetch the certificates this time:

Nov 09 12:37:42 certcentral1001 systemd[1]: Reloading Central Certificates Service.
Nov 09 12:37:42 certcentral1001 certcentral-backend[30803]: SIGHUP received
Nov 09 12:37:42 certcentral1001 systemd[1]: Reloaded Central Certificates Service.
Nov 09 12:37:42 certcentral1001 certcentral-backend[30803]: Missing/invalid DNS zone updater CMD timeout, using the default one: 60.00
Nov 09 12:37:42 certcentral1001 certcentral-backend[30803]: Number of certificates per status: Counter({'VALID': 2, 'SELF_SIGNED': 2})
Nov 09 12:37:47 certcentral1001 certcentral-backend[30803]: Handling new certificate event for pinkunicorn2 / rsa-2048
Nov 09 12:37:47 certcentral1001 certcentral-backend[30803]: Triggering DNS zone update...
Nov 09 12:37:47 certcentral1001 certcentral-backend[30803]: Running subprocess ['/usr/local/bin/certcentral-gdnsd-sync.py', '--remote-servers', 'authdns1001.wikimedia.org', 'authdns2001.wikimedia.org', 'multatuli.wikimedia.org', '--', '_acme-challenge.pinkunicorn2.wikimedia.org', '3r0gzNp1jxAJBvwwq9gKNWgdDQNA7w801ubYd7LIo6E']
Nov 09 12:37:50 certcentral1001 certcentral-backend[30803]: Handling pushed CSR event for pinkunicorn2 / rsa-2048
Nov 09 12:37:50 certcentral1001 certcentral-backend[30803]: Handling validated challenges event for pinkunicorn2 / rsa-2048
Nov 09 12:37:50 certcentral1001 certcentral-backend[30803]: Handling pushed challenges event for pinkunicorn2 / rsa-2048
Nov 09 12:37:53 certcentral1001 certcentral-backend[30803]: Handling order finalized event for pinkunicorn2 / rsa-2048
Nov 09 12:37:54 certcentral1001 certcentral-backend[30803]: Pushing the new certificate for pinkunicorn2 / rsa-2048
Nov 09 12:37:54 certcentral1001 certcentral-backend[30803]: Handling new certificate event for pinkunicorn2 / ec-prime256v1
Nov 09 12:37:55 certcentral1001 certcentral-backend[30803]: Skipping challenge validation for certificate pinkunicorn2 / ec-prime256v1
Nov 09 12:38:00 certcentral1001 certcentral-backend[30803]: Handling pushed challenges event for pinkunicorn2 / ec-prime256v1
Nov 09 12:38:01 certcentral1001 certcentral-backend[30803]: Handling order finalized event for pinkunicorn2 / ec-prime256v1
Nov 09 12:38:02 certcentral1001 certcentral-backend[30803]: Pushing the new certificate for pinkunicorn2 / ec-prime256v1
Nov 9 2018, 12:38 PM · Patch-For-Review, Certcentral
Vgutierrez added a comment to T208948: certcentral "wrongly" assumes that a new order always implies fulfilling new challenges.

I ran a test to issue new certificates for a non already challenged hostname: pinkunicorn2.wikimedia.org, these are the results:

certcentral1001
Nov 09 12:26:15 certcentral1001 systemd[1]: Reloading Central Certificates Service.
Nov 09 12:26:15 certcentral1001 certcentral-backend[30803]: SIGHUP received
Nov 09 12:26:15 certcentral1001 systemd[1]: Reloaded Central Certificates Service.
Nov 09 12:26:15 certcentral1001 certcentral-backend[30803]: Missing/invalid DNS zone updater CMD timeout, using the default one: 60.00
Nov 09 12:26:15 certcentral1001 certcentral-backend[30803]: New configured certificates: {'pinkunicorn2'}
Nov 09 12:26:15 certcentral1001 certcentral-backend[30803]: Number of certificates per status: Counter({'INITIAL': 2, 'VALID': 2})
Nov 09 12:26:15 certcentral1001 certcentral-backend[30803]: Creating initial self-signed certificate for pinkunicorn2 / rsa-2048
Nov 09 12:26:15 certcentral1001 certcentral-backend[30803]: Creating initial self-signed certificate for pinkunicorn2 / ec-prime256v1
Nov 09 12:26:19 certcentral1001 certcentral-backend[30803]: Handling new certificate event for pinkunicorn2 / rsa-2048
Nov 09 12:26:20 certcentral1001 certcentral-backend[30803]: Triggering DNS zone update...
Nov 09 12:26:20 certcentral1001 certcentral-backend[30803]: Running subprocess ['/usr/local/bin/certcentral-gdnsd-sync.py', '--remote-servers', 'authdns1001.wikimedia.org', 'authdns2001.wikimedia.org', 'multatuli.wikimedia.org', '--', '_acme-challenge.pinkunicorn2.wikimedia.org', 'zgs7KCjRNsewJmWjKT6RYBNz0FP__R1VGFJT3Qz6zzA']
Nov 09 12:26:22 certcentral1001 certcentral-backend[30803]: Handling pushed CSR event for pinkunicorn2 / rsa-2048
Nov 09 12:26:22 certcentral1001 certcentral-backend[30803]: Handling validated challenges event for pinkunicorn2 / rsa-2048
Nov 09 12:26:22 certcentral1001 certcentral-backend[30803]: Handling pushed challenges event for pinkunicorn2 / rsa-2048
Nov 09 12:26:22 certcentral1001 certcentral-backend[30803]: ACME directory has rejected the challenge(s) for order https://acme-staging-v02.api.letsencrypt.org/acme/order/7090084/12874112
Nov 09 12:26:22 certcentral1001 certcentral-backend[30803]: ACME Directory has rejected the challenge(s) for certificate pinkunicorn2 / rsa-2048
Nov 09 12:26:22 certcentral1001 certcentral-backend[30803]: Handling new certificate event for pinkunicorn2 / ec-prime256v1
Nov 09 12:26:22 certcentral1001 certcentral-backend[30803]: Triggering DNS zone update...
Nov 09 12:26:22 certcentral1001 certcentral-backend[30803]: Running subprocess ['/usr/local/bin/certcentral-gdnsd-sync.py', '--remote-servers', 'authdns1001.wikimedia.org', 'authdns2001.wikimedia.org', 'multatuli.wikimedia.org', '--', '_acme-challenge.pinkunicorn2.wikimedia.org', 'xFp3Dy08tIYnLqNNhJpRQRalpAe5bfZvs0jy38pkUJU']
Nov 09 12:26:25 certcentral1001 certcentral-backend[30803]: Handling pushed CSR event for pinkunicorn2 / ec-prime256v1
Nov 09 12:26:25 certcentral1001 certcentral-backend[30803]: Handling validated challenges event for pinkunicorn2 / ec-prime256v1
Nov 09 12:26:25 certcentral1001 certcentral-backend[30803]: Handling pushed challenges event for pinkunicorn2 / ec-prime256v1
Nov 09 12:26:26 certcentral1001 certcentral-backend[30803]: ACME directory has rejected the challenge(s) for order https://acme-staging-v02.api.letsencrypt.org/acme/order/7090084/12874117
Nov 09 12:26:26 certcentral1001 certcentral-backend[30803]: ACME Directory has rejected the challenge(s) for certificate pinkunicorn2 / ec-prime256v1
certcentral2002
Nov 09 12:26:15 certcentral2001 systemd[1]: Reloading Central Certificates Service.
Nov 09 12:26:15 certcentral2001 certcentral-backend[3275]: SIGHUP received
Nov 09 12:26:15 certcentral2001 systemd[1]: Reloaded Central Certificates Service.
Nov 09 12:26:15 certcentral2001 certcentral-backend[3275]: Missing/invalid DNS zone updater CMD timeout, using the default one: 60.00
Nov 09 12:26:15 certcentral2001 certcentral-backend[3275]: New configured certificates: {'pinkunicorn2'}
Nov 09 12:26:15 certcentral2001 certcentral-backend[3275]: Number of certificates per status: Counter({'INITIAL': 2, 'VALID': 2})
Nov 09 12:26:15 certcentral2001 certcentral-backend[3275]: Creating initial self-signed certificate for pinkunicorn2 / ec-prime256v1
Nov 09 12:26:15 certcentral2001 certcentral-backend[3275]: Creating initial self-signed certificate for pinkunicorn2 / rsa-2048
Nov 09 12:26:16 certcentral2001 certcentral-backend[3275]: Handling new certificate event for pinkunicorn2 / ec-prime256v1
Nov 09 12:26:16 certcentral2001 certcentral-backend[3275]: Triggering DNS zone update...
Nov 09 12:26:16 certcentral2001 certcentral-backend[3275]: Running subprocess ['/usr/local/bin/certcentral-gdnsd-sync.py', '--remote-servers', 'authdns1001.wikimedia.org', 'authdns2001.wikimedia.org', 'multatuli.wikimedia.org', '--', '_acme-challenge.pinkunicorn2.wikimedia.org', 'b6_IMxhS361pfcNPJ_X3RlpXO75LMZ_5zC6IpJxXCrU']
Nov 09 12:26:19 certcentral2001 certcentral-backend[3275]: Handling pushed CSR event for pinkunicorn2 / ec-prime256v1
Nov 09 12:26:19 certcentral2001 certcentral-backend[3275]: Handling validated challenges event for pinkunicorn2 / ec-prime256v1
Nov 09 12:26:19 certcentral2001 certcentral-backend[3275]: Handling pushed challenges event for pinkunicorn2 / ec-prime256v1
Nov 09 12:26:21 certcentral2001 certcentral-backend[3275]: Handling order finalized event for pinkunicorn2 / ec-prime256v1
Nov 09 12:26:23 certcentral2001 certcentral-backend[3275]: Pushing the new certificate for pinkunicorn2 / ec-prime256v1
Nov 09 12:26:23 certcentral2001 certcentral-backend[3275]: Handling new certificate event for pinkunicorn2 / rsa-2048
Nov 09 12:26:23 certcentral2001 certcentral-backend[3275]: Skipping challenge validation for certificate pinkunicorn2 / rsa-2048
Nov 09 12:26:28 certcentral2001 certcentral-backend[3275]: Handling pushed challenges event for pinkunicorn2 / rsa-2048
Nov 09 12:26:29 certcentral2001 certcentral-backend[3275]: Handling order finalized event for pinkunicorn2 / rsa-2048
Nov 09 12:26:30 certcentral2001 certcentral-backend[3275]: Pushing the new certificate for pinkunicorn2 / rsa-2048
Nov 9 2018, 12:37 PM · Patch-For-Review, Certcentral
Vgutierrez added a comment to T208967: Avoid using acme.client poll_and_finalize() method.

Test results against LE staging environment are really promising:

certcentral1001
Nov 09 12:05:31 certcentral1001 certcentral-backend[30803]: SIGHUP received
Nov 09 12:05:31 certcentral1001 certcentral-backend[30803]: Missing/invalid DNS zone updater CMD timeout, using the default one: 60.00
Nov 09 12:05:31 certcentral1001 certcentral-backend[30803]: Number of certificates per status: Counter({'INITIAL': 2})
Nov 09 12:05:31 certcentral1001 certcentral-backend[30803]: Creating initial self-signed certificate for pinkunicorn / rsa-2048
Nov 09 12:05:31 certcentral1001 certcentral-backend[30803]: Creating initial self-signed certificate for pinkunicorn / ec-prime256v1
Nov 09 12:05:31 certcentral1001 certcentral-backend[30803]: Starting main loop...
Nov 09 12:05:31 certcentral1001 certcentral-backend[30803]: Handling new certificate event for pinkunicorn / rsa-2048
Nov 09 12:05:33 certcentral1001 certcentral-backend[30803]: Skipping challenge validation for certificate pinkunicorn / rsa-2048
Nov 09 12:05:33 certcentral1001 certcentral-backend[30803]: Handling new certificate event for pinkunicorn / ec-prime256v1
Nov 09 12:05:33 certcentral1001 certcentral-backend[30803]: Skipping challenge validation for certificate pinkunicorn / ec-prime256v1
Nov 09 12:05:38 certcentral1001 certcentral-backend[30803]: Handling pushed challenges event for pinkunicorn / rsa-2048
Nov 09 12:05:39 certcentral1001 certcentral-backend[30803]: Handling order finalized event for pinkunicorn / rsa-2048
Nov 09 12:05:40 certcentral1001 certcentral-backend[30803]: Pushing the new certificate for pinkunicorn / rsa-2048
Nov 09 12:05:40 certcentral1001 certcentral-backend[30803]: Handling pushed challenges event for pinkunicorn / ec-prime256v1
Nov 09 12:05:41 certcentral1001 certcentral-backend[30803]: Handling order finalized event for pinkunicorn / ec-prime256v1
Nov 09 12:05:43 certcentral1001 certcentral-backend[30803]: Pushing the new certificate for pinkunicorn / ec-prime256v1
certcentral2001
Nov 09 12:05:30 certcentral2001 certcentral-backend[3275]: SIGHUP received
Nov 09 12:05:30 certcentral2001 certcentral-backend[3275]: Missing/invalid DNS zone updater CMD timeout, using the default one: 60.00
Nov 09 12:05:30 certcentral2001 certcentral-backend[3275]: Number of certificates per status: Counter({'INITIAL': 2})
Nov 09 12:05:30 certcentral2001 certcentral-backend[3275]: Creating initial self-signed certificate for pinkunicorn / ec-prime256v1
Nov 09 12:05:30 certcentral2001 certcentral-backend[3275]: Creating initial self-signed certificate for pinkunicorn / rsa-2048
Nov 09 12:05:31 certcentral2001 certcentral-backend[3275]: Starting main loop...
Nov 09 12:05:31 certcentral2001 certcentral-backend[3275]: Handling new certificate event for pinkunicorn / ec-prime256v1
Nov 09 12:05:32 certcentral2001 certcentral-backend[3275]: Skipping challenge validation for certificate pinkunicorn / ec-prime256v1
Nov 09 12:05:32 certcentral2001 certcentral-backend[3275]: Handling new certificate event for pinkunicorn / rsa-2048
Nov 09 12:05:33 certcentral2001 certcentral-backend[3275]: Skipping challenge validation for certificate pinkunicorn / rsa-2048
Nov 09 12:05:38 certcentral2001 certcentral-backend[3275]: Handling pushed challenges event for pinkunicorn / ec-prime256v1
Nov 09 12:05:39 certcentral2001 certcentral-backend[3275]: Handling order finalized event for pinkunicorn / ec-prime256v1
Nov 09 12:05:40 certcentral2001 certcentral-backend[3275]: Pushing the new certificate for pinkunicorn / ec-prime256v1
Nov 09 12:05:40 certcentral2001 certcentral-backend[3275]: Handling pushed challenges event for pinkunicorn / rsa-2048
Nov 09 12:05:43 certcentral2001 certcentral-backend[3275]: Handling order finalized event for pinkunicorn / rsa-2048
Nov 09 12:05:44 certcentral2001 certcentral-backend[3275]: Pushing the new certificate for pinkunicorn / rsa-2048
Nov 9 2018, 12:18 PM · Certcentral
Vgutierrez committed rOSCC39e0a2e35866: debian: Add release 0.6 to changelog (authored by Vgutierrez).
debian: Add release 0.6 to changelog
Nov 9 2018, 8:59 AM
Vgutierrez committed rOSCC69fff94a8845: Release 0.6 (authored by Vgutierrez).
Release 0.6
Nov 9 2018, 8:52 AM
Vgutierrez committed rOSCC058861ec08f1: certcentral: Stop using acme.client.poll_and_finalize() (authored by Vgutierrez).
certcentral: Stop using acme.client.poll_and_finalize()
Nov 9 2018, 8:52 AM
Vgutierrez committed rOSCC0c179f70f6c3: certcentral: Evaluate order status after creation (authored by Vgutierrez).
certcentral: Evaluate order status after creation
Nov 9 2018, 8:52 AM
Vgutierrez committed rOSCCa647b4a1ca55: acme_requests: log order URI on non-recoverable finalization errors (authored by Vgutierrez).
acme_requests: log order URI on non-recoverable finalization errors
Nov 9 2018, 8:52 AM
Vgutierrez committed rOSCCd1583469c43c: Release 0.6 (authored by Vgutierrez).
Release 0.6
Nov 9 2018, 8:46 AM

Nov 8 2018

Vgutierrez committed rOSCC8969ef7d32ed: certcentral: Stop using acme.client.poll_and_finalize() (authored by Vgutierrez).
certcentral: Stop using acme.client.poll_and_finalize()
Nov 8 2018, 6:50 PM
Vgutierrez committed rOSCCb6fadc7738a6: certcentral: Evaluate order status after creation (authored by Vgutierrez).
certcentral: Evaluate order status after creation
Nov 8 2018, 6:50 PM
Vgutierrez committed rOSCC6169b5facae5: certcentral: Stop using acme.client.poll_and_finalize() (authored by Vgutierrez).
certcentral: Stop using acme.client.poll_and_finalize()
Nov 8 2018, 5:40 PM
Vgutierrez committed rOSCCd8ecb5c6b2d9: certcentral: Stop using acme.client.poll_and_finalize() (authored by Vgutierrez).
certcentral: Stop using acme.client.poll_and_finalize()
Nov 8 2018, 4:08 PM
Vgutierrez committed rOSCC2402b587c2ef: certcentral: Stop using acme.client.poll_and_finalize() (authored by Vgutierrez).
certcentral: Stop using acme.client.poll_and_finalize()
Nov 8 2018, 4:01 PM
Vgutierrez committed rOSCC563632b94b04: certcentral: Evaluate order status after creation (authored by Vgutierrez).
certcentral: Evaluate order status after creation
Nov 8 2018, 3:55 PM

Nov 7 2018

Vgutierrez renamed T208970: certcentral wrongly handles acme.errors.ValidationError exception from certcentral handles interprets acme.errors.ValidationError exception to certcentral wrongly handles acme.errors.ValidationError exception.
Nov 7 2018, 5:54 PM · Certcentral
Vgutierrez renamed T208970: certcentral wrongly handles acme.errors.ValidationError exception from certcentral wrongly interprets acme.errors.ValidationError exception to certcentral handles interprets acme.errors.ValidationError exception.
Nov 7 2018, 5:54 PM · Certcentral
Vgutierrez created T208970: certcentral wrongly handles acme.errors.ValidationError exception.
Nov 7 2018, 5:53 PM · Certcentral
Vgutierrez created T208967: Avoid using acme.client poll_and_finalize() method.
Nov 7 2018, 5:48 PM · Certcentral
Vgutierrez added a comment to T208948: certcentral "wrongly" assumes that a new order always implies fulfilling new challenges.

I've already tested this manually on certcentral1001 because it's impossible to reproduce this behaviour with pebble.

Nov 07 16:19:58 certcentral1001 certcentral-backend[20234]: Creating initial self-signed certificate for pinkunicorn / rsa-2048
Nov 07 16:19:58 certcentral1001 certcentral-backend[20234]: Creating initial self-signed certificate for pinkunicorn / ec-prime256v1
Nov 07 16:19:58 certcentral1001 certcentral-backend[20234]: Starting main loop...
Nov 07 16:19:58 certcentral1001 certcentral-backend[20234]: Handling new certificate event for pinkunicorn / rsa-2048
Nov 07 16:19:59 certcentral1001 certcentral-backend[20234]: Skipping challenge validation for certificate pinkunicorn / rsa-2048
Nov 07 16:19:59 certcentral1001 certcentral-backend[20234]: Handling new certificate event for pinkunicorn / ec-prime256v1
Nov 07 16:20:00 certcentral1001 certcentral-backend[20234]: Skipping challenge validation for certificate pinkunicorn / ec-prime256v1
Nov 07 16:20:05 certcentral1001 certcentral-backend[20234]: Handling pushed challenges event for pinkunicorn / rsa-2048
Nov 07 16:20:07 certcentral1001 certcentral-backend[20234]: Pushing the new certificate for pinkunicorn / rsa-2048
Nov 7 2018, 4:46 PM · Patch-For-Review, Certcentral
Vgutierrez committed rOSCC200529d2433d: certcentral: Evaluate order status after creation (authored by Vgutierrez).
certcentral: Evaluate order status after creation
Nov 7 2018, 4:41 PM
Vgutierrez renamed T208948: certcentral "wrongly" assumes that a new order always implies fulfilling new challenges from certcentral "wrongly" assumes that a new order always implies fulfulling new challenges to certcentral "wrongly" assumes that a new order always implies fulfilling new challenges.
Nov 7 2018, 1:58 PM · Patch-For-Review, Certcentral
Vgutierrez created T208948: certcentral "wrongly" assumes that a new order always implies fulfilling new challenges.
Nov 7 2018, 1:58 PM · Patch-For-Review, Certcentral

Nov 6 2018

Vgutierrez created P7765 (An Untitled Masterwork).
Nov 6 2018, 4:52 PM
Vgutierrez committed rOSCC93e09e3a8852: acme_requests: log order URI on non-recoverable finalization errors (authored by Vgutierrez).
acme_requests: log order URI on non-recoverable finalization errors
Nov 6 2018, 4:14 PM
Vgutierrez committed rOSCC07421cec086d: acme_requests: log order URI on non-recoverable finalization errors (authored by Vgutierrez).
acme_requests: log order URI on non-recoverable finalization errors
Nov 6 2018, 4:14 PM
Vgutierrez created T208859: certcentral: keep track of orders and authorizations IDs when issuing certificates.
Nov 6 2018, 3:12 PM · Certcentral
Vgutierrez closed T207737: LE rejects issuing two certificates with the same CSR on a short timespan as Resolved.
Nov 6 2018, 1:12 PM · Patch-For-Review, Certcentral
Vgutierrez closed T207927: Take into account LE rate limits on sensitive operations as Resolved.
Nov 6 2018, 1:12 PM · Patch-For-Review, Certcentral
Vgutierrez closed T208212: Provision unique LE accounts for each certcentral node as Resolved.
Nov 6 2018, 1:11 PM · Patch-For-Review, Certcentral
Vgutierrez closed T208378: retrying policy currently ignores self_signed status as Resolved.
Nov 6 2018, 1:10 PM · Patch-For-Review, Certcentral
Vgutierrez closed T208572: Report number of certificates on config (re)load as Resolved.
Nov 6 2018, 1:09 PM · Patch-For-Review, Certcentral
Vgutierrez closed T208833: package upgrade overwrites certcentral systemd service unit file as Resolved.

It's has been solved as @Joe suggested in -traffic. Using an override we keep the systemd unit file provided by the debian package and we just provide the required environment variables via puppet.

Nov 6 2018, 12:40 PM · Patch-For-Review, Certcentral
Vgutierrez triaged T208833: package upgrade overwrites certcentral systemd service unit file as Normal priority.
Nov 6 2018, 8:54 AM · Patch-For-Review, Certcentral
Vgutierrez created T208833: package upgrade overwrites certcentral systemd service unit file.
Nov 6 2018, 8:54 AM · Patch-For-Review, Certcentral

Nov 5 2018

Vgutierrez committed rOSCC211487d522e3: debian: add release 0.5 to changelog (authored by Vgutierrez).
debian: add release 0.5 to changelog
Nov 5 2018, 4:38 PM
Vgutierrez committed rOSCCda32480becc0: debian: add release 0.5 to changelog (authored by Vgutierrez).
debian: add release 0.5 to changelog
Nov 5 2018, 4:38 PM
Vgutierrez committed rOSCC446a4d009ce2: Release 0.5 (authored by Vgutierrez).
Release 0.5
Nov 5 2018, 4:33 PM
Vgutierrez committed rOSCCec00898b9707: certcentral: report certificate status on config (re)load (authored by Vgutierrez).
certcentral: report certificate status on config (re)load
Nov 5 2018, 4:33 PM
Vgutierrez committed rOSCC57bb46e78d91: certcentral: Stop abusing SELF_SIGNED status to signal errors (authored by Vgutierrez).
certcentral: Stop abusing SELF_SIGNED status to signal errors
Nov 5 2018, 4:33 PM
Vgutierrez committed rOSCC844e10dc96a0: Release 0.5 (authored by Vgutierrez).
Release 0.5
Nov 5 2018, 4:21 PM
Vgutierrez added a comment to T207476: Create production LE accounts.

As per T208212 two Lets Encrypt production accounts are going to be created

Nov 5 2018, 3:44 PM · Patch-For-Review, Certcentral, Traffic, Operations
Vgutierrez committed rOSCC9f045fc66858: certcentral: report certificate status on config (re)load (authored by Vgutierrez).
certcentral: report certificate status on config (re)load
Nov 5 2018, 3:40 PM
Vgutierrez committed rOSCC8886a49b3ce0: certcentral: report certificate status on config (re)load (authored by Vgutierrez).
certcentral: report certificate status on config (re)load
Nov 5 2018, 1:38 PM
Vgutierrez committed rOSCC4c2b5a01693a: certcentral: report certificate status on config (re)load (authored by Vgutierrez).
certcentral: report certificate status on config (re)load
Nov 5 2018, 1:38 PM

Nov 2 2018

Vgutierrez closed T208583: Reimage eeden to test role as Resolved.
Nov 2 2018, 2:58 PM · decommission, ops-esams, Operations, Traffic
Vgutierrez claimed T208572: Report number of certificates on config (re)load.
Nov 2 2018, 10:33 AM · Patch-For-Review, Certcentral
Vgutierrez created T208572: Report number of certificates on config (re)load.
Nov 2 2018, 10:33 AM · Patch-For-Review, Certcentral

Oct 31 2018

Vgutierrez closed T208424: Test wildcard certificate issuance with certcentral as Resolved.
Oct 31 2018, 4:38 PM · Patch-For-Review, Certcentral
Vgutierrez closed T208390: Allow Let's Encrypt issue wildcard certificates as Resolved.
Oct 31 2018, 4:38 PM · Patch-For-Review, Certcentral, Operations, Traffic, DNS
Vgutierrez closed T208390: Allow Let's Encrypt issue wildcard certificates, a subtask of T208424: Test wildcard certificate issuance with certcentral, as Resolved.
Oct 31 2018, 4:38 PM · Patch-For-Review, Certcentral
Vgutierrez added a comment to T208424: Test wildcard certificate issuance with certcentral.

certcentral has been able to get the certificates in both nodes. No manual operation has been required, the change https://gerrit.wikimedia.org/r/470846 has been merged and afterwards puppet ran in both nodes triggering the certcentral restart.

Oct 31 2018, 4:05 PM · Patch-For-Review, Certcentral
Vgutierrez created P7746 certcentral1001 logs during wildcard certificate issuance.
Oct 31 2018, 4:04 PM · Certcentral
Vgutierrez created P7745 certcentral2001 logs during wildcard certificate issuance.
Oct 31 2018, 4:02 PM · Certcentral
Vgutierrez added a subtask for T208424: Test wildcard certificate issuance with certcentral: T208390: Allow Let's Encrypt issue wildcard certificates.
Oct 31 2018, 2:34 PM · Patch-For-Review, Certcentral
Vgutierrez added a parent task for T208390: Allow Let's Encrypt issue wildcard certificates: T208424: Test wildcard certificate issuance with certcentral.
Oct 31 2018, 2:34 PM · Patch-For-Review, Certcentral, Operations, Traffic, DNS
Vgutierrez created T208424: Test wildcard certificate issuance with certcentral.
Oct 31 2018, 2:34 PM · Patch-For-Review, Certcentral