Page MenuHomePhabricator

RLazarus (Reuven Lazarus)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 15 2019, 4:02 PM (32 w, 1 d)
Availability
Available
IRC Nick
rlazarus
LDAP User
RLazarus
MediaWiki User
Unknown

Recent Activity

Yesterday

RLazarus added a comment to T251349: Request for srv/phab/phabricator/bin/bulk make-silent --id * command via SSH for moving tasks quarterly.

Should I instead point to the private key I generated (per https://wikitech.wikimedia.org/wiki/Production_access#Generating_your_SSH_key)?

Tue, May 26, 8:27 PM · SRE-Access-Requests, Operations
RLazarus reassigned T252705: Requesting access to sites from Google Search Console for soworu from RLazarus to soworu.

@soworu Ping on my question above. :)

Tue, May 26, 2:17 PM · Operations, SRE-Access-Requests

Fri, May 22

RLazarus added a project to T252826: Whitelist x-wikimedia-debug header field (currently not allowed by Access-Control-Allow-Headers in preflight response): Core Platform Team.

Hi CPT, adding you back -- is it possible this is due to this section of security_response_header_filter.js in Restbase?

Fri, May 22, 6:23 PM · Core Platform Team Workboards (Clinic Duty Team), Developer Productivity, Operations, Traffic
RLazarus triaged T253292: check_http and SNI support as Medium priority.
Fri, May 22, 6:01 PM · Operations, Traffic

Thu, May 21

RLazarus closed T253340: Create SDAW-Internal mailing list as Resolved.

All set!

Thu, May 21, 9:42 PM · Operations, Wikimedia-Mailing-lists
RLazarus triaged T253197: Implement a prometheus exporter for rdkafka in golang as Medium priority.
Thu, May 21, 6:09 PM · Operations, Traffic
RLazarus added a project to T252227: Mobile redirects drop provenance parameters: Traffic.
Thu, May 21, 5:52 PM · Analytics, Traffic, Operations, Readers-Web-Backlog
RLazarus assigned T183990: Publishing project anomaly data for censorship researchers. Evaluate privacy threats to ssingh.

Trying to route this -- @ssingh, should this be assigned to you?

Thu, May 21, 5:51 PM · Privacy Engineering, Operations, Traffic, Analytics
RLazarus closed T249352: Onboarding Wolfgang Kandek as Resolved.
Thu, May 21, 5:47 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations
RLazarus added a comment to T251349: Request for srv/phab/phabricator/bin/bulk make-silent --id * command via SSH for moving tasks quarterly.

@MBinder_WMF On the offchance you'd already tried logging into phab1001, and it didn't work, but you hadn't gotten around to saying anything yet -- that was because of my mistake, just fixed it, so try again. :)

Thu, May 21, 5:04 PM · SRE-Access-Requests, Operations

Wed, May 20

RLazarus closed T250430: LDAP/NDA Access Request for mshaver as Resolved.
Wed, May 20, 10:00 PM · Operations, LDAP-Access-Requests
RLazarus claimed T250430: LDAP/NDA Access Request for mshaver.

Thanks! I've removed mnoor but not mshaver from the wmf LDAP group, let me know if anything suddenly stops working.

Wed, May 20, 8:46 PM · Operations, LDAP-Access-Requests
RLazarus added a comment to T250430: LDAP/NDA Access Request for mshaver.

Hi @MNoorWMF, just checking on this -- can you please confirm that the mshaver account is working for you? If it is, I'll remove access from mnoor so that we don't leave both lying around.

Wed, May 20, 7:55 PM · Operations, LDAP-Access-Requests
RLazarus added a comment to T251349: Request for srv/phab/phabricator/bin/bulk make-silent --id * command via SSH for moving tasks quarterly.

Hi @MBinder_WMF! I've created your shell account and added it to the group that @Dzahn created for you. I just ran puppet on bast4002 (your nearest bastion) and both Phab hosts, so you should be able to test your access now by sshing to phab1001.eqiad.wmnet, after following the Setting_up_your_SSH_config steps in the page that Daniel linked.

Wed, May 20, 7:48 PM · SRE-Access-Requests, Operations
RLazarus updated subscribers of T253155: Give access to the JupyterHub (SWAP) notebooks to (@Rvvalentim).

After chatting with @colewhite (thanks!) I've moved Rodolfo from the wmf group over to the nda group, which should still provide all the access necessary here. Let me know if you have any trouble.

Wed, May 20, 6:08 PM · LDAP-Access-Requests, Operations, Jupyter-Hub
RLazarus added a comment to T253155: Give access to the JupyterHub (SWAP) notebooks to (@Rvvalentim).

This uses your LDAP password -- the same one you use to log into Wikitech. The username should be either your Wikitech username (Rodolfo Valentim) or your uid/"shell username" (rodolfovalentim) -- the login prompt should tell you which.

Wed, May 20, 5:52 PM · LDAP-Access-Requests, Operations, Jupyter-Hub
RLazarus added a comment to T253086: Add Daniel Cipoletti to analytics-privatedata-users.

@dcipoletti Thanks, looks good! All I need you to do now is:

Wed, May 20, 5:38 PM · LDAP-Access-Requests, Operations
RLazarus closed T253155: Give access to the JupyterHub (SWAP) notebooks to (@Rvvalentim), a subtask of T252476: Give access to the Analytics Cluster to Research Inter (Rodolfo), as Resolved.
Wed, May 20, 5:18 PM · SRE-Access-Requests, Operations
RLazarus closed T253155: Give access to the JupyterHub (SWAP) notebooks to (@Rvvalentim) as Resolved.

Thanks Katie!

Wed, May 20, 5:18 PM · LDAP-Access-Requests, Operations, Jupyter-Hub

Tue, May 19

RLazarus triaged T253131: docker-pkg update cli renders unclear guidance as Medium priority.

Looks like this comes from docker_pkg/cli.py line 76, which is the last line here:

Tue, May 19, 10:27 PM · serviceops, docker-pkg, Operations
RLazarus assigned T253155: Give access to the JupyterHub (SWAP) notebooks to (@Rvvalentim) to KFrancis.

I see @Nuria's authorization on the parent task, so going ahead with this.

Tue, May 19, 8:56 PM · LDAP-Access-Requests, Operations, Jupyter-Hub
RLazarus triaged T253086: Add Daniel Cipoletti to analytics-privatedata-users as Medium priority.
Tue, May 19, 8:47 PM · LDAP-Access-Requests, Operations
RLazarus assigned T253086: Add Daniel Cipoletti to analytics-privatedata-users to dcipoletti.

@dcipoletti Can you please ask your manager to comment on this task, approving your access?

Tue, May 19, 8:46 PM · LDAP-Access-Requests, Operations
RLazarus triaged T252749: Reverse proxy supporting XFF-based per-IP concurrency limit and request queueing as Medium priority.

Agree we ought to do this, and I think it's something Envoy can do nearly out of the box. If there's an element Envoy doesn't support, I agree that it's the over-limit FIFO queue. (Out of all the reverse proxies we could use, I'm focusing on Envoy because we're actively deploying it in other parts of the infrastructure, and if it's feasible I think we'd like to use it here too for consistency.)

Tue, May 19, 7:08 PM · serviceops, Sustainability (Incident Followup), Operations, Thumbor
RLazarus assigned T252773: Move kafkamon hosts to Debian Buster to herron.
Tue, May 19, 5:38 PM · observability, Operations, Analytics
RLazarus triaged T252890: scrape ripe atlas data for a few anchors at other large networks as Medium priority.
Tue, May 19, 5:34 PM · netops, Operations
RLazarus triaged T252913: Better manage java updates for ELK7 as Medium priority.
Tue, May 19, 5:31 PM · observability, Operations
RLazarus triaged T253025: dbctl gives user-hostile diffs as Medium priority.
Tue, May 19, 5:27 PM · conftool, Operations
RLazarus triaged T252956: Add Git LFS support for research/wikiworkshop as Medium priority.
Tue, May 19, 5:26 PM · Operations, Research

Mon, May 18

RLazarus closed T252674: Add `dcipoletti` to `wmf` Access Group as Resolved.

Hi Daniel, welcome to the Foundation! I've added you to the wmf LDAP group, feel free to reopen if you need anything else.

Mon, May 18, 9:59 PM · LDAP-Access-Requests, Operations
RLazarus closed T252887: LDAP access to the wmf group for Carly Bogen as Resolved.

Hi Carly, I've added you to the wmf LDAP group -- feel free to reopen if you need anything else.

Mon, May 18, 9:44 PM · LDAP-Access-Requests, Operations
RLazarus removed a project from T252987: Requesting access to wikimedia namespace in packagist: SRE-Access-Requests.

Thanks @bd808! It doesn't sound like there's presently anything for the SRE clinic-duty person to do here, so I'm optimistically removing SRE-Access-Requests, but feel free to add it back if you need anything from me (or subsequent clinicians).

Mon, May 18, 9:25 PM · Patch-For-Review, Composer, Librarization
RLazarus triaged T252705: Requesting access to sites from Google Search Console for soworu as Medium priority.
Mon, May 18, 7:58 PM · Operations, SRE-Access-Requests
RLazarus claimed T252705: Requesting access to sites from Google Search Console for soworu.

Hi Segun, thanks for the clear and complete request! I've granted you "restricted user" access for almost all those domains, which I think is the right level of access for your needs. If you need "full user," let me know and I'll upgrade you (or file another task, if you find out later).

Mon, May 18, 7:57 PM · Operations, SRE-Access-Requests

Thu, May 14

RLazarus created P11196 (An Untitled Masterwork).
Thu, May 14, 5:37 PM

Wed, May 6

RLazarus claimed T244340: Reduce read pressure on memcached servers by adding a machine-local Memcache instance.
Wed, May 6, 5:11 PM · Performance-Team (Radar), Patch-For-Review, Operations, serviceops

Tue, May 5

RLazarus created P11152 (An Untitled Masterwork).
Tue, May 5, 2:58 PM
RLazarus created P11151 (An Untitled Masterwork).
Tue, May 5, 2:44 PM
RLazarus created P11150 (An Untitled Masterwork).
Tue, May 5, 2:35 PM

Mon, May 4

RLazarus renamed T243318: FY2020-2021 Q1 codfw -> eqiad switchback from FY2019-20 Q4 (or later) codfw -> eqiad switchback to FY2020-2021 Q1 codfw -> eqiad switchback.
Mon, May 4, 7:05 PM · Operations
RLazarus renamed T243316: FY2020-2021 Q1 eqiad -> codfw switchover from FY2019-20 Q4 eqiad -> codfw switchover to FY2020-2021 Q1 eqiad -> codfw switchover.
Mon, May 4, 7:04 PM · Operations
RLazarus updated the task description for T243314: FY2020-2021 Q1 DC switchover and switchback.
Mon, May 4, 7:04 PM · Goal, Operations
RLazarus renamed T243314: FY2020-2021 Q1 DC switchover and switchback from FY2019-20 Q4 DC switchover and switchback to FY2020-2021 Q1 DC switchover and switchback.
Mon, May 4, 7:04 PM · Goal, Operations
RLazarus renamed T244808: CommRel support for FY2020-2021 Q1 DC switchover from CommRel support for FY2019-2020 Q4 DC switchover to CommRel support for FY2020-2021 Q1 DC switchover.
Mon, May 4, 7:03 PM · CommRel-Specialists-Support (Jul-Sep-2020), Operations
RLazarus edited projects for T244808: CommRel support for FY2020-2021 Q1 DC switchover, added: CommRel-Specialists-Support (Jul-Sep-2020); removed CommRel-Specialists-Support (Apr-Jun-2020).

We've ruled out a switchover in Q4. We'll continue to do all the non-user-impacting prep work we can, so we might be ready to go early in Q1, if all goes well -- but obviously that's way past the horizon for predicting the continuing impact of COVID-19 on our work capacity, so I'll keep you updated.

Mon, May 4, 7:03 PM · CommRel-Specialists-Support (Jul-Sep-2020), Operations

Thu, Apr 30

RLazarus updated the task description for T249352: Onboarding Wolfgang Kandek.
Thu, Apr 30, 6:02 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations
RLazarus created P11102 (An Untitled Masterwork).
Thu, Apr 30, 3:02 PM

Wed, Apr 29

Dzahn awarded T236699: Build a black-box httpd testing framework a Love token.
Wed, Apr 29, 9:31 AM · Wikimedia-Apache-configuration, Operations, serviceops

Tue, Apr 28

akosiaris awarded T248523: `helmfile --interactive apply` logs to SAL even if cancelled a Love token.
Tue, Apr 28, 1:12 PM · Prod-Kubernetes, serviceops, Operations

Apr 23 2020

Dzahn awarded T248093: Renew certs for mcrouter on all application servers. a Like token.
Apr 23 2020, 7:23 AM · Operations, serviceops

Apr 21 2020

RLazarus closed T248093: Renew certs for mcrouter on all application servers. as Resolved.

Certs renewed! I still need to merge the script for next time, and maybe set it up to run periodically unattended, but I'm resolving this.

Apr 21 2020, 7:49 PM · Operations, serviceops

Apr 20 2020

RLazarus added a comment to T248093: Renew certs for mcrouter on all application servers..

The renewal script works as expected, but the procedure as written caused problems because not every mcrouter host is listed in /etc/cergen/mcrouter.manifests.d/mediawiki-hosts.certs.yaml on puppetmaster1001. That meant the missing hosts didn't have certs re-created by cergen, so they were deleted without replacement, which meant puppet failed on those hosts. I reverted the cert renewal, so it still needs to be done.

Apr 20 2020, 8:57 PM · Operations, serviceops
RLazarus added a member for serviceops-radar: RLazarus.
Apr 20 2020, 3:58 PM

Apr 17 2020

RLazarus created P11012 mwmaint1002 crontabs/www-data updates.
Apr 17 2020, 3:19 PM

Apr 14 2020

RLazarus created T250231: purgeParserCache.php: Cannot purge this kind of parser cache.
Apr 14 2020, 10:15 PM · Core Platform Team Workboards (Clinic Duty Team), MediaWiki-Parser, serviceops, Operations
RLazarus edited P10981 journalctl -u mediawiki_job_parser_cache_purging.service.
Apr 14 2020, 9:36 PM
RLazarus created P10981 journalctl -u mediawiki_job_parser_cache_purging.service.
Apr 14 2020, 9:24 PM

Apr 13 2020

RLazarus updated the task description for T249352: Onboarding Wolfgang Kandek.
Apr 13 2020, 6:49 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations
RLazarus added a member for WMF-NDA-Requests: wkandek.
Apr 13 2020, 6:42 PM
RLazarus added a member for WMF-NDA-Requests: RLazarus.
Apr 13 2020, 6:42 PM
RLazarus added a member for acl*sre-team: wkandek.
Apr 13 2020, 6:40 PM
RLazarus added a member for WMF-NDA: wkandek.
Apr 13 2020, 6:37 PM

Apr 10 2020

RLazarus added a comment to T244808: CommRel support for FY2020-2021 Q1 DC switchover.

Yeah, as you can imagine with some folks working reduced hours, SRE is mostly focusing on critical work and we've pushed this off. I haven't asked around in a little bit, but at a guess I'd say that late Q4 is still possible but extremely uncertain.

Apr 10 2020, 5:16 PM · CommRel-Specialists-Support (Jul-Sep-2020), Operations

Apr 3 2020

jcrespo awarded T249352: Onboarding Wolfgang Kandek a Pterodactyl token.
Apr 3 2020, 6:54 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations
RLazarus added a comment to T249352: Onboarding Wolfgang Kandek.

Welcome Wolfgang! We've already been chatting about some of this stuff, but this Phab task will track progress as we get it taken care of.

Apr 3 2020, 5:16 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations
RLazarus triaged T249352: Onboarding Wolfgang Kandek as High priority.
Apr 3 2020, 5:15 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations

Mar 31 2020

RLazarus reopened T163986: Revamp spec.yaml in citoid as "Open".

Reopening this -- we had some alerts today from Citoid that turned out to be latency associated with only 404s (graph). We suspect the root cause was elevated latency from example.com.

Mar 31 2020, 5:37 PM · Patch-For-Review, User-Ryasmeen, Services (watching), User-mobrovac, VisualEditor, Citoid

Mar 30 2020

RLazarus closed T246868: envoyproxy: CVE-2020-8664 CVE-2020-8661 CVE-2020-8660 CVE-2020-8659 as Resolved.

And done.

Mar 30 2020, 2:50 PM · Patch-For-Review, Operations, Security
RLazarus added a comment to T246868: envoyproxy: CVE-2020-8664 CVE-2020-8661 CVE-2020-8660 CVE-2020-8659.

Just snapshot1006 left, it's on my list for this morning.

Mar 30 2020, 2:46 PM · Patch-For-Review, Operations, Security

Mar 26 2020

RLazarus updated subscribers of T246868: envoyproxy: CVE-2020-8664 CVE-2020-8661 CVE-2020-8660 CVE-2020-8659.
Mar 26 2020, 2:37 PM · Patch-For-Review, Operations, Security

Mar 25 2020

RLazarus added a comment to T246868: envoyproxy: CVE-2020-8664 CVE-2020-8661 CVE-2020-8660 CVE-2020-8659.

All kubernetes services are updated in all clusters. (T246868#6000068 turned out to be operator error, there were no unexpected diffs.)

Mar 25 2020, 10:39 PM · Patch-For-Review, Operations, Security
RLazarus triaged T248523: `helmfile --interactive apply` logs to SAL even if cancelled as Low priority.
Mar 25 2020, 10:35 PM · Prod-Kubernetes, serviceops, Operations

Mar 23 2020

RLazarus updated subscribers of T246868: envoyproxy: CVE-2020-8664 CVE-2020-8661 CVE-2020-8660 CVE-2020-8659.

All hosts updated except snapshot1006, held back until later this week per @ArielGlenn.

Mar 23 2020, 4:06 PM · Patch-For-Review, Operations, Security

Mar 19 2020

RLazarus added a comment to T246868: envoyproxy: CVE-2020-8664 CVE-2020-8661 CVE-2020-8660 CVE-2020-8659.

All remaining MW hosts are updated. That leaves parsoid, snapshot hosts, and a few other odds and ends.

Mar 19 2020, 8:52 PM · Patch-For-Review, Operations, Security
RLazarus claimed T248093: Renew certs for mcrouter on all application servers..
Mar 19 2020, 6:47 PM · Operations, serviceops
RLazarus closed T236699: Build a black-box httpd testing framework as Resolved.

Marking this done: https://gerrit.wikimedia.org/r/581616 deleted apache-fast-test, as httpbb is now complete.

Mar 19 2020, 4:00 PM · Wikimedia-Apache-configuration, Operations, serviceops

Mar 18 2020

RLazarus updated subscribers of T246868: envoyproxy: CVE-2020-8664 CVE-2020-8661 CVE-2020-8660 CVE-2020-8659.

Deployed 1.13.1 to all hosts where we're using envoy as a TLS proxy, that is, C:profile::tlsproxy::envoy. Exception: mendelevium.eqiad.wmnet is still running Jessie for now, and Envoy 1.11.1. (fyi @Dzahn)

Mar 18 2020, 4:03 PM · Patch-For-Review, Operations, Security

Mar 17 2020

RLazarus added a comment to T246868: envoyproxy: CVE-2020-8664 CVE-2020-8661 CVE-2020-8660 CVE-2020-8659.

Belated update: we decided to upgrade to 1.13.1 (not 1.12.3). So far it's deployed to all MW hosts in codfw, plus the MW canaries in eqiad. Monitoring for impact, then we'll proceed to the rest of MW, then Kubernetes hosts.

Mar 17 2020, 3:30 PM · Patch-For-Review, Operations, Security

Mar 10 2020

RLazarus added a project to T244395: Investigate whether we can automatically share incident status docs with WMDE: SRE-OnFire.
Mar 10 2020, 7:15 PM · Sustainability (Incident Followup), SRE-OnFire, Operations

Mar 4 2020

RLazarus added a comment to T246868: envoyproxy: CVE-2020-8664 CVE-2020-8661 CVE-2020-8660 CVE-2020-8659.

I'm pretty sure our envoy-build expert in residence just assigned this to me, but I'm happy to give this a shot anyway.

Mar 4 2020, 3:24 PM · Patch-For-Review, Operations, Security

Mar 3 2020

RLazarus moved T246338: Add graceful-restart to cr2-esams from On-going to Follow-up prevention on the Wikimedia-Incident board.
Mar 3 2020, 4:24 PM · Sustainability (Incident Followup), Operations, netops
RLazarus added a project to T246338: Add graceful-restart to cr2-esams: Wikimedia-Incident.
Mar 3 2020, 4:23 PM · Sustainability (Incident Followup), Operations, netops

Feb 24 2020

RLazarus added a comment to T244395: Investigate whether we can automatically share incident status docs with WMDE.

"On Monday" turned out to be two weeks later -- sorry about that. Conclusions from today's SRE meeting, documented for posterity:

Feb 24 2020, 10:46 PM · Sustainability (Incident Followup), SRE-OnFire, Operations

Feb 18 2020

RLazarus committed rLPRI38addf7a0d35: Add fake keys for mw13[49-84].eqiad.wmnet (authored by RLazarus).
Add fake keys for mw13[49-84].eqiad.wmnet
Feb 18 2020, 11:04 PM

Feb 12 2020

RLazarus added a comment to T244761: Script to point SRE local machine traffic to another LB.

Maybe separate from a script, is there any way we can do this via DNS? Something like grafana.cp-ulsfo.wikimedia.org to specify a site rather than accepting geoip routing.

Feb 12 2020, 10:19 PM · Operations

Feb 10 2020

RLazarus created T244808: CommRel support for FY2020-2021 Q1 DC switchover.
Feb 10 2020, 11:49 PM · CommRel-Specialists-Support (Jul-Sep-2020), Operations
RLazarus renamed T243318: FY2020-2021 Q1 codfw -> eqiad switchback from 2020 Q3 (or later) codfw -> eqiad switchback to 2020 Q4 (or later) codfw -> eqiad switchback.
Feb 10 2020, 10:44 PM · Operations
RLazarus renamed T243316: FY2020-2021 Q1 eqiad -> codfw switchover from 2020 Q3 eqiad -> codfw switchover to 2020 Q4 eqiad -> codfw switchover.
Feb 10 2020, 10:44 PM · Operations
RLazarus renamed T243314: FY2020-2021 Q1 DC switchover and switchback from 2020 Q3 DC switchover and switchback to 2020 Q4 DC switchover and switchback.
Feb 10 2020, 10:43 PM · Goal, Operations

Feb 7 2020

RLazarus added a comment to T244395: Investigate whether we can automatically share incident status docs with WMDE.

Thanks! I'll bring this up in the SRE meeting on Monday and go ahead if no one objects.

Feb 7 2020, 2:26 PM · Sustainability (Incident Followup), SRE-OnFire, Operations
RLazarus updated the task description for T244395: Investigate whether we can automatically share incident status docs with WMDE.
Feb 7 2020, 2:24 PM · Sustainability (Incident Followup), SRE-OnFire, Operations

Feb 5 2020

RLazarus triaged T244395: Investigate whether we can automatically share incident status docs with WMDE as Medium priority.
Feb 5 2020, 7:25 PM · Sustainability (Incident Followup), SRE-OnFire, Operations
RLazarus created T244395: Investigate whether we can automatically share incident status docs with WMDE.
Feb 5 2020, 7:24 PM · Sustainability (Incident Followup), SRE-OnFire, Operations

Feb 4 2020

RLazarus created T244278: Document 2020-02-04 kartotherian incident.
Feb 4 2020, 7:31 PM · SRE-OnFire-Incident-Docs, Patch-For-Review, Operations

Jan 30 2020

RLazarus added a project to T243935: Audit all cumin queries in switchdc scripts: SRE-tools.
Jan 30 2020, 1:21 AM · Patch-For-Review, SRE-tools, Operations
RLazarus created T243935: Audit all cumin queries in switchdc scripts.
Jan 30 2020, 1:21 AM · Patch-For-Review, SRE-tools, Operations

Jan 24 2020

RLazarus created P10266 (An Untitled Masterwork).
Jan 24 2020, 10:20 PM

Jan 21 2020

RLazarus created T243318: FY2020-2021 Q1 codfw -> eqiad switchback.
Jan 21 2020, 6:43 PM · Operations
RLazarus created T243316: FY2020-2021 Q1 eqiad -> codfw switchover.
Jan 21 2020, 6:42 PM · Operations
RLazarus created T243314: FY2020-2021 Q1 DC switchover and switchback.
Jan 21 2020, 6:40 PM · Goal, Operations