Page MenuHomePhabricator
Feed Advanced Search

Today

jcrespo awarded T237650: Renew and deploy GlobalSign unified cert (2019) a Like token.
Wed, Nov 13, 2:16 PM · Traffic, Operations
BBlack added a comment to T237650: Renew and deploy GlobalSign unified cert (2019).

A certificate warning here https://social.imirhil.fr/@aeris/103126273693383568 the user still had the cert 2018 at 2019-11-12 17:07 UTC although the sites served the cert 2019 at that time, I guess because of some cache on their side.
Possibly it would be better to have a longer transition, where the new cert is served and preferred, but the old one is still valid and served.

Wed, Nov 13, 2:13 PM · Traffic, Operations

Yesterday

BBlack closed T237650: Renew and deploy GlobalSign unified cert (2019) as Resolved.

In live use now.

Tue, Nov 12, 2:19 PM · Traffic, Operations

Thu, Nov 7

BBlack added a comment to T237687: ATS doesn't support X-Wikimedia-Debug.

Maybe this is closer to a Lua replacement for all of it, although it still has issues!

Thu, Nov 7, 11:35 PM · Performance-Team (Radar), Traffic, Operations
BBlack added a comment to T237687: ATS doesn't support X-Wikimedia-Debug.

Reading up on the debug_proxy stuff a bit more.... currently hassium/hassaleh are proxies into mwdebug[12]00[12], and use the header to select the destination host, and also has some backwards compatibility for older values. We could potentially skip/eliminate the debug proxy layer and handle this directly as well. The underlying mwdebug hosts actually do have TLS configured already (like the non-debug appservers).

Thu, Nov 7, 10:38 PM · Performance-Team (Radar), Traffic, Operations
BBlack triaged T237687: ATS doesn't support X-Wikimedia-Debug as High priority.
Thu, Nov 7, 8:58 PM · Performance-Team (Radar), Traffic, Operations
BBlack added a subtask for T237650: Renew and deploy GlobalSign unified cert (2019): Unknown Object (Task).
Thu, Nov 7, 4:13 PM · Traffic, Operations
BBlack triaged T237650: Renew and deploy GlobalSign unified cert (2019) as High priority.
Thu, Nov 7, 4:13 PM · Traffic, Operations
BBlack added a comment to T236497: cp3056 hardware issue.

Sorry I missed that you already had a patch! But in any case, we only need commenting from cache::nodes to fix up this case (there's no good reason to e.g. churn it out of conftool or the various iptables rules defined from the other stuff).

Thu, Nov 7, 3:42 PM · DC-Ops, ops-esams, Traffic, Operations
BBlack updated subscribers of T237492: Create a second text-lb IP address for test purposes.
Thu, Nov 7, 3:26 PM · Patch-For-Review, Traffic, Operations

Wed, Nov 6

BBlack triaged T237492: Create a second text-lb IP address for test purposes as Normal priority.
Wed, Nov 6, 4:21 AM · Patch-For-Review, Traffic, Operations

Tue, Nov 5

BBlack removed a parent task for T98006: Anycast AuthDNS: T104442: Investigate better DNS cache/lookup solutions.
Tue, Nov 5, 6:08 PM · Performance-Team (Radar), Patch-For-Review, netops, Operations, Traffic
BBlack removed a subtask for T104442: Investigate better DNS cache/lookup solutions: T98006: Anycast AuthDNS.
Tue, Nov 5, 6:08 PM · Patch-For-Review, Traffic, Operations
BBlack renamed T98006: Anycast AuthDNS from Anycast (Auth)DNS to Anycast AuthDNS.
Tue, Nov 5, 6:08 PM · Performance-Team (Radar), Patch-For-Review, netops, Operations, Traffic
BBlack added a comment to T171498: Implement machine-local forwarding DNS caches.

I think this is actually fairly orthogonal to some of the other improvements. Not sure what current/modern thinking is on this either, probably needs re-evaluation. My gut feeling it to lean against bothering with this right now.

Tue, Nov 5, 6:08 PM · Traffic, Operations
BBlack closed T104442: Investigate better DNS cache/lookup solutions as Resolved.

With anycast recdns deployed at all sites with fallback routing towards the cores (or to the opposite core, as the case may be), I think we're in pretty good shape here at this point. If there are other specific improvements we want to make, they should probably be re-evaluated in current context and considered in smaller-scoped tickets like T171498

Tue, Nov 5, 6:07 PM · Patch-For-Review, Traffic, Operations
BBlack removed a subtask for T98006: Anycast AuthDNS: T186550: Anycast recdns.
Tue, Nov 5, 6:06 PM · Performance-Team (Radar), Patch-For-Review, netops, Operations, Traffic
BBlack edited parent tasks for T186550: Anycast recdns, added: T104442: Investigate better DNS cache/lookup solutions; removed: T98006: Anycast AuthDNS.
Tue, Nov 5, 6:06 PM · Patch-For-Review, netops, Operations, Traffic
BBlack added a subtask for T104442: Investigate better DNS cache/lookup solutions: T186550: Anycast recdns.
Tue, Nov 5, 6:06 PM · Patch-For-Review, Traffic, Operations
BBlack closed T101525: Set up LVS for current AuthDNS, a subtask of T98006: Anycast AuthDNS, as Declined.
Tue, Nov 5, 5:59 PM · Performance-Team (Radar), Patch-For-Review, netops, Operations, Traffic
BBlack closed T101525: Set up LVS for current AuthDNS, a subtask of T140365: Lower geodns TTLs from 600 (10min) to 300 (5min), as Declined.
Tue, Nov 5, 5:58 PM · Traffic, Operations
BBlack closed T101525: Set up LVS for current AuthDNS as Declined.

I don't think we'll go the LVS route here.

Tue, Nov 5, 5:58 PM · Operations, Traffic
BBlack closed T228190: Roll out Anycast RecDNS to more servers, a subtask of T186550: Anycast recdns, as Resolved.
Tue, Nov 5, 5:56 PM · Patch-For-Review, netops, Operations, Traffic
BBlack closed T228190: Roll out Anycast RecDNS to more servers as Resolved.
Tue, Nov 5, 5:56 PM · Operations, Traffic
BBlack added a comment to T128559: store.wikimedia.org HTTPS issues.

@MBeat33 / @Jseddon - Any update yet?

Tue, Nov 5, 5:52 PM · Operations, Traffic, Wikimedia-Shop, HTTPS
BBlack closed T118181: Planning for phasing out non-Forward-Secret TLS ciphers as Resolved.

Yes, this task was long-ago completed. See also https://phabricator.wikimedia.org/phame/post/view/111/wikipedia_goes_100_forward_secret/

Tue, Nov 5, 5:49 PM · Operations, Traffic
BBlack closed T127482: Enable VCL source-DC switching via confd as Declined.

We're not going down this road at all. cache::route_table will just go away when all cache backends have converted to ATS in T227432, which doesn't use a tiered setup to reach the origins.

Tue, Nov 5, 5:43 PM · codfw-rollout, Traffic, Operations
BBlack added a comment to T233661: Publish tls related info to webrequest via varnish.

@BBlack: once we deploy the VCL/varnish-kafka chnages we need to change our refine pipeline to read these values, when we deploy those changes values will be available in webrequest table, after that we ill re-do the indexing of the webrequest dataset into turnilo

Tue, Nov 5, 12:41 PM · Patch-For-Review, Analytics-Kanban, observability, Operations, Analytics, Traffic
BBlack added a comment to T233661: Publish tls related info to webrequest via varnish.

Agreed, let's not go down that road right here (because we have a burning need for this data pronto), but side note to keep in mind: "one day" is really really soon (like, we'll probably be migrating varnishkafka stuff to ATS next quarter).

Probably this is not the best place to talk about this, but next quarter seems really close :) Who is going to replace Varnishkafka? Is there any plan from Traffic or should we (as Analytics) schedule time for it? I know that with fifo-log-demux the job of the new "atskafka" should be relatively easy (read from a socket and push to kafka) but we really need metrics (like we have now) to build monitoring on top of them. If the migration is so close, can we open a task (if there is not one) and start the discussion? :)

Tue, Nov 5, 12:39 PM · Patch-For-Review, Analytics-Kanban, observability, Operations, Analytics, Traffic

Mon, Nov 4

BBlack added a comment to T233661: Publish tls related info to webrequest via varnish.

Nevermind, I see it in the gerrit comments

Mon, Nov 4, 7:26 PM · Patch-For-Review, Analytics-Kanban, observability, Operations, Analytics, Traffic
BBlack added a comment to T233661: Publish tls related info to webrequest via varnish.

Hm, would be ok with me, but likely whatever we choose we'll be stuck with forever. I tend to prefer descriptive names in general, but Joseph might more concerned with the efficiency.

Mon, Nov 4, 7:25 PM · Patch-For-Review, Analytics-Kanban, observability, Operations, Analytics, Traffic
BBlack added a comment to T233661: Publish tls related info to webrequest via varnish.

Patches above look sane? I went ahead and shortened the key names down to the minimum to prevent bloat at these layers. We can always give them better descriptive names when they're pulled back to e.g. Turnilo in queries. V is Version, K is Key Exchange, A is Auth, C is Cipher. Too short?

Mon, Nov 4, 6:54 PM · Patch-For-Review, Analytics-Kanban, observability, Operations, Analytics, Traffic
BBlack added a comment to T233661: Publish tls related info to webrequest via varnish.

Agreed, let's not go down that road right here (because we have a burning need for this data pronto), but side note to keep in mind: "one day" is really really soon (like, we'll probably migrating varnishkafka stuff to ATS next quarter).

Mon, Nov 4, 4:35 PM · Patch-For-Review, Analytics-Kanban, observability, Operations, Analytics, Traffic

Fri, Nov 1

BBlack added a comment to T230236: De-noise ipsec alerts (Reduce Icinga alert noise goal).

https://grafana.wikimedia.org/d/B9JpocKZz/ipsec-tunnel-status probably needs some cleanup (some of the graphs are empty, there's a note there to ignore icinga errors, etc). Also fix missing doc link on the alert?

Fri, Nov 1, 5:27 PM · Patch-For-Review, User-herron, Goal, observability
BBlack added a comment to T233661: Publish tls related info to webrequest via varnish.

We probably don't need to send the reused value (it's not that useful for analysis at this level, IMHO), and we don't need to send the full-cipher value either (that's the original string from which some of these fields are extracted, but doesn't cover the Key-Exchange part fully or the Version at all). All we need is the 4 derived fields: Version, Key-Exchange, Auth, and Cipher. The string "CP-" isn't really descriptive and just an implementation detail. Also I'm assuming from how X-Analytics is set up that the format is k1=v1;k2=v2;.... (equal sign rather than colon).

Fri, Nov 1, 3:56 PM · Patch-For-Review, Analytics-Kanban, observability, Operations, Analytics, Traffic
BBlack added a comment to T233661: Publish tls related info to webrequest via varnish.

@Nuria - what you're asking for is something like a combined TLS field with separators? e.g. we contruct a 4-part semicolon-delimited string like: V=1.2;K=X25519;A=ECDSA;C=CHACHA20POLY1305, and then set that in webrequest as the single field TLS ?

Fri, Nov 1, 3:32 PM · Patch-For-Review, Analytics-Kanban, observability, Operations, Analytics, Traffic

Thu, Oct 31

BBlack added a comment to T237020: Ferm should log errors when failing to create all configured rules.

Digging a little deeper on the Net::DNS side of things and the issues with how options parsing in /etc/resolv.conf affects behavior, looking at the actual version of it deployed on db1119 (latest stretch):

Thu, Oct 31, 3:19 PM · Operations
BBlack added a comment to T237020: Ferm should log errors when failing to create all configured rules.

Copying over from IRC: looking at the ferm code itself, a couple of things are notable:

Thu, Oct 31, 2:57 PM · Operations
BBlack added a comment to T237011: Update DNS/NTP servers on the esams PDUs/SCS.

I had already manually updated cr[23]-esams, mr1-esams, and asw2-esams, as appropriate for NTP (DNS should've been already-correct on those), I believe.

Thu, Oct 31, 2:33 PM · DC-Ops, ops-esams, Operations

Wed, Oct 30

BBlack added a comment to T235427: Serve volatile uri from local site.

~15m delays should be ok for the GeoIP stuff, it was already sync'd to various consuming cache and DNS nodes over the ~30 minute splay window of puppet runs without issue.

Wed, Oct 30, 5:32 PM · Traffic, User-jbond, Operations, Puppet
BBlack added a comment to T226840: Consistent HTTP 503 Error on some urls for some logged-in users (CentralAuth Set-Cookie storm).

It sounds like this particular problem is fixed. Was TMH the only main offender? If so, can @BBlack revert the header limit?

Wed, Oct 30, 4:14 PM · Core Platform Team, Patch-For-Review, TimedMediaHandler, MW-1.34-notes (1.34.0-wmf.13; 2019-07-09), Wikimedia-Incident, Performance-Team (Radar), Traffic, MediaWiki-extensions-CentralAuth, Operations
BBlack added a comment to T236497: cp3056 hardware issue.

Tried again this morning, but the kernel panics happen too fast to make much progress once the agent starts actually using the NIC (I've only ever had one agent run complete successfully before a crash, out of many attempts). The crashes (and preceding dmesg outputs) are consistently issues with the card and/or driver for the 10G NIC. I'd say this sounds like our familiar firmware-level issue, but I was able to look at ethtool earlier and it seems like the same firmware version which is stable on the rest of the new esams cache nodes. Given the history, perhaps it really is some kind of actual system board error (which was first affecting the PCIe NVMe drive, and is now affecting the PCIe NIC? I'm at a loss on causes, but if it consistently can't make it through a few puppet runs without crashing, something's wrong....

Wed, Oct 30, 2:32 PM · DC-Ops, ops-esams, Traffic, Operations

Tue, Oct 29

BBlack added a comment to T236497: cp3056 hardware issue.

I've tried imaging, and things mostly work, but I have a hard time keeping it online long enough to get through an initial puppet agent run (or two or three), as the kernel keeps panic-ing somewhere related to the NIC, e.g.

Tue, Oct 29, 11:07 PM · DC-Ops, ops-esams, Traffic, Operations
BBlack closed T233242: rack/setup/install cp30[50-65].esams.wmnet as Resolved.

As a batch these servers are complete in general. Note cp3056 had an early hardware issue that prevented progress, but this is tracked separately in: T236497

Tue, Oct 29, 1:30 PM · Traffic, ops-esams, Operations
BBlack updated the task description for T233242: rack/setup/install cp30[50-65].esams.wmnet.
Tue, Oct 29, 1:28 PM · Traffic, ops-esams, Operations
BBlack renamed T208585: Decommission cp3007-cp3010 from Decommission esams cache_misc hosts to Decommission cp3007-cp3010.
Tue, Oct 29, 1:24 PM · ops-esams, decommission, Operations, Traffic
BBlack closed T225720: poll power data for redeployment of esams/knams as Resolved.
Tue, Oct 29, 1:20 PM · Traffic, DC-Ops, Operations
BBlack merged T227077: decommission cp3037 into T236454: decommission cp3030-3049.
Tue, Oct 29, 1:19 PM · Traffic, Operations, DC-Ops, decommission
BBlack merged task T227077: decommission cp3037 into T236454: decommission cp3030-3049.
Tue, Oct 29, 1:19 PM · DC-Ops, decommission
BBlack merged T202046: cp3032 PS Redundancy Lost into T236454: decommission cp3030-3049.
Tue, Oct 29, 1:18 PM · Traffic, Operations, DC-Ops, decommission
BBlack merged task T202046: cp3032 PS Redundancy Lost into T236454: decommission cp3030-3049.
Tue, Oct 29, 1:18 PM · ops-esams, Traffic, Operations
BBlack merged T202627: cp3036 PS Redundancy Lost into T236454: decommission cp3030-3049.
Tue, Oct 29, 1:17 PM · Traffic, Operations, DC-Ops, decommission
BBlack merged task T202627: cp3036 PS Redundancy Lost into T236454: decommission cp3030-3049.
Tue, Oct 29, 1:17 PM · Traffic, ops-esams, Operations
BBlack closed T236294: rack/setup/install lvs300[567] as Resolved.
Tue, Oct 29, 1:15 PM · Traffic, Operations, ops-esams
BBlack updated the task description for T236294: rack/setup/install lvs300[567].
Tue, Oct 29, 1:15 PM · Traffic, Operations, ops-esams
BBlack closed T236217: rack/setup/install dns300[12], a subtask of T235805: ESAMS Refresh/Rebuild (October 2019), as Resolved.
Tue, Oct 29, 1:14 PM · Patch-For-Review, DC-Ops, ops-esams, Operations
BBlack closed T236217: rack/setup/install dns300[12] as Resolved.
Tue, Oct 29, 1:14 PM · Traffic, DNS, ops-esams, Operations
BBlack updated the task description for T236217: rack/setup/install dns300[12].
Tue, Oct 29, 1:14 PM · Traffic, DNS, ops-esams, Operations
BBlack added a subtask for T235805: ESAMS Refresh/Rebuild (October 2019): T236497: cp3056 hardware issue.
Tue, Oct 29, 1:13 PM · Patch-For-Review, DC-Ops, ops-esams, Operations
BBlack added a parent task for T236497: cp3056 hardware issue: T235805: ESAMS Refresh/Rebuild (October 2019).
Tue, Oct 29, 1:13 PM · DC-Ops, ops-esams, Traffic, Operations

Mon, Oct 28

BBlack closed T236686: bast3004 can't reach mgmt networks as Resolved.

Turns out it was simpler than I thought! Should be done here, re-open if it's still not working.

Mon, Oct 28, 5:14 PM · netops, ops-esams, Operations
BBlack claimed T236686: bast3004 can't reach mgmt networks .

I'll poke at this today since Arzhel's not here (may take a couple hours, squeezing it around meetings)

Mon, Oct 28, 4:50 PM · netops, ops-esams, Operations

Fri, Oct 25

BBlack triaged T236497: cp3056 hardware issue as Normal priority.
Fri, Oct 25, 3:40 PM · DC-Ops, ops-esams, Traffic, Operations
BBlack assigned T236489: decommission multatuli to Papaul.
Fri, Oct 25, 3:03 PM · Traffic, Operations, DC-Ops, decommission
BBlack updated the task description for T236489: decommission multatuli.
Fri, Oct 25, 3:03 PM · Traffic, Operations, DC-Ops, decommission
BBlack added a subtask for T235805: ESAMS Refresh/Rebuild (October 2019): T236489: decommission multatuli.
Fri, Oct 25, 3:03 PM · Patch-For-Review, DC-Ops, ops-esams, Operations
BBlack added a parent task for T236489: decommission multatuli: T235805: ESAMS Refresh/Rebuild (October 2019).
Fri, Oct 25, 3:03 PM · Traffic, Operations, DC-Ops, decommission
BBlack created T236489: decommission multatuli.
Fri, Oct 25, 2:55 PM · Traffic, Operations, DC-Ops, decommission
BBlack triaged T236479: Temporarily use ganeti3003 as ns2 authdns as Normal priority.
Fri, Oct 25, 1:11 PM · Traffic, Operations
BBlack added a subtask for T235805: ESAMS Refresh/Rebuild (October 2019): T208585: Decommission cp3007-cp3010.
Fri, Oct 25, 5:03 AM · Patch-For-Review, DC-Ops, ops-esams, Operations
BBlack added a parent task for T208585: Decommission cp3007-cp3010: T235805: ESAMS Refresh/Rebuild (October 2019).
Fri, Oct 25, 5:03 AM · ops-esams, decommission, Operations, Traffic
BBlack reassigned T236454: decommission cp3030-3049 from BBlack to Papaul.
Fri, Oct 25, 5:00 AM · Traffic, Operations, DC-Ops, decommission
BBlack merged task T200806: cp3031: Power required by the system exceeds the power supplied by the Power Supply Units into T236454: decommission cp3030-3049.
Fri, Oct 25, 4:05 AM · Traffic, Operations
BBlack merged task T190607: cp3048 hardware issues into T236454: decommission cp3030-3049.
Fri, Oct 25, 4:05 AM · Traffic, Operations, ops-esams
BBlack merged task T199677: cp3033 unreacheable since 2018-07-15 11:47:31 into T236454: decommission cp3030-3049.
Fri, Oct 25, 4:05 AM · ops-esams, Operations, Traffic
BBlack merged task T222041: cp3037 is currently unreachable into T236454: decommission cp3030-3049.
Fri, Oct 25, 4:05 AM · ops-esams, Operations, Traffic
BBlack merged task T225035: cp3035 PS Redundancy Lost into T236454: decommission cp3030-3049.
Fri, Oct 25, 4:05 AM · Traffic, Operations, ops-esams
BBlack merged tasks T225035: cp3035 PS Redundancy Lost, T222041: cp3037 is currently unreachable, T199677: cp3033 unreacheable since 2018-07-15 11:47:31, T200806: cp3031: Power required by the system exceeds the power supplied by the Power Supply Units, T190607: cp3048 hardware issues into T236454: decommission cp3030-3049.
Fri, Oct 25, 4:05 AM · Traffic, Operations, DC-Ops, decommission
BBlack added a subtask for T235805: ESAMS Refresh/Rebuild (October 2019): T236454: decommission cp3030-3049.
Fri, Oct 25, 4:04 AM · Patch-For-Review, DC-Ops, ops-esams, Operations
BBlack added a parent task for T236454: decommission cp3030-3049: T235805: ESAMS Refresh/Rebuild (October 2019).
Fri, Oct 25, 4:04 AM · Traffic, Operations, DC-Ops, decommission
BBlack created T236454: decommission cp3030-3049.
Fri, Oct 25, 4:02 AM · Traffic, Operations, DC-Ops, decommission
BBlack added a subtask for T235805: ESAMS Refresh/Rebuild (October 2019): T236452: decommission nescio and maerlant.
Fri, Oct 25, 3:28 AM · Patch-For-Review, DC-Ops, ops-esams, Operations
BBlack added a parent task for T236452: decommission nescio and maerlant: T235805: ESAMS Refresh/Rebuild (October 2019).
Fri, Oct 25, 3:28 AM · Traffic, Operations, DC-Ops, decommission
BBlack assigned T236452: decommission nescio and maerlant to Papaul.
Fri, Oct 25, 3:28 AM · Traffic, Operations, DC-Ops, decommission
BBlack updated the task description for T236452: decommission nescio and maerlant.
Fri, Oct 25, 3:27 AM · Traffic, Operations, DC-Ops, decommission
BBlack created T236452: decommission nescio and maerlant.
Fri, Oct 25, 3:04 AM · Traffic, Operations, DC-Ops, decommission
BBlack assigned T236451: decommission lvs300[1234] to Papaul.
Fri, Oct 25, 2:43 AM · Traffic, Operations, DC-Ops, decommission
BBlack updated the task description for T236451: decommission lvs300[1234].
Fri, Oct 25, 2:43 AM · Traffic, Operations, DC-Ops, decommission
BBlack added a subtask for T235805: ESAMS Refresh/Rebuild (October 2019): T236451: decommission lvs300[1234].
Fri, Oct 25, 2:08 AM · Patch-For-Review, DC-Ops, ops-esams, Operations
BBlack added a parent task for T236451: decommission lvs300[1234]: T235805: ESAMS Refresh/Rebuild (October 2019).
Fri, Oct 25, 2:08 AM · Traffic, Operations, DC-Ops, decommission
BBlack created T236451: decommission lvs300[1234].
Fri, Oct 25, 2:07 AM · Traffic, Operations, DC-Ops, decommission

Wed, Oct 23

BBlack added a comment to T236217: rack/setup/install dns300[12].

confirming above - @Papaul is correct. The total set of new esams Linux boxes AFAIK is: 16x caches, 3x LVS, 2x DNS, 1x Bastion, 3x Ganeti.

Wed, Oct 23, 3:32 PM · DNS, Traffic, Operations, ops-esams

Tue, Oct 22

BBlack triaged T236208: interface-rps.py should have a flag to avoid CPU0 as Normal priority.
Tue, Oct 22, 8:09 PM · Operations, Traffic
BBlack created P9437 broadcom firmware stuff.
Tue, Oct 22, 4:10 PM · Traffic

Thu, Oct 17

BBlack closed T209515: Renew Digicert Unified in 2019 as Resolved.

Digicert-2019 is now in live use at the esams edge and we have full normal redundancy (for now) among commercial cert vendors.

Thu, Oct 17, 5:35 PM · Operations, Traffic
BBlack closed T224033: Fix operations/puppet.git "rebase hell" as Resolved.

It's switched to Rebase-if-necessary now

Thu, Oct 17, 2:45 PM · Release-Engineering-Team (Development services), Gerrit, Release-Engineering-Team-TODO, Continuous-Integration-Config, Operations
BBlack added a comment to T224033: Fix operations/puppet.git "rebase hell".

IRC says the meeting was mostly consumed by OKR discussion, it may have been talked about a little, nobody remembers any new big blocker being raised.

Thu, Oct 17, 2:43 PM · Release-Engineering-Team (Development services), Gerrit, Release-Engineering-Team-TODO, Continuous-Integration-Config, Operations
BBlack added a parent task for T234803: Provide an easy way of picking the traffic serving TLS certificate used by ATS: T230687: Decide/document criteria needed to serve acme-chief LE issued unified certificate to end users.
Thu, Oct 17, 2:15 PM · Operations, Traffic
BBlack added a subtask for T230687: Decide/document criteria needed to serve acme-chief LE issued unified certificate to end users: T234803: Provide an easy way of picking the traffic serving TLS certificate used by ATS.
Thu, Oct 17, 2:15 PM · Operations, Traffic, Acme-chief
BBlack added a comment to T230687: Decide/document criteria needed to serve acme-chief LE issued unified certificate to end users.

@Vgutierrez may have some ideas about how to tackle these, but it's behind other priorities at present (We could manually switch in the LE certs globally in an OCSP service emergency, if that were necessary before this puppetization work were done). We'll probably wait to tackle his until our TLS termination has finished switching over to our new ATS implementation, since that's close on the horizon and the existing puppetization is nginx-based - (T221594 and related).

Thu, Oct 17, 2:15 PM · Operations, Traffic, Acme-chief