Page MenuHomePhabricator

wkandek (Wolfgang Kandek)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Apr 12 2020, 7:16 PM (75 w, 14 h)
Availability
Available
LDAP User
Unknown
MediaWiki User
WKandek (WMF) [ Global Accounts ]

Recent Activity

Wed, Sep 15

wkandek claimed T288844: Update MaxMind GeoIP2 license key and product IDs for application servers.

Yes, we will take a look to see how the new database can be put on all appservers.

Wed, Sep 15, 2:42 PM · Patch-For-Review, Anti-Harassment, IP Info, serviceops, SRE

Aug 20 2021

wkandek added a comment to T288844: Update MaxMind GeoIP2 license key and product IDs for application servers.

In T288375 we are discussing how this extension would have access to the maxmind databases when we migrate MediaWiki to Kubernetes. It would be helpful to know how important the "freshness" of the database is, i.e. how quickly after maxmind publishes an update do we need that update in production? Maxmind updates the database mostly in a weekly rhythm, but for certain elements quicker (and slower). See: https://support.maxmind.com/geoip-faq/databases-and-database-updates/how-often-are-the-geoip2-and-geoip-legacy-databases-updated/. One of the options is too include the maxmind database in the release, which happens mostly daily, but in extreme cases (no deployment weeks for example, christmas holidays, etc) there can be intervals of up to 10 days without a release. Is that acceptable to the users of IP Info? The migration to Kubernetes is currently in the works, a first version can be accessed via the Wikimedia Debug extension for Firefox or Chrome and then selecting "k8s-experimental", we are projecting a full migration for early 2022.

Aug 20 2021, 1:29 AM · Patch-For-Review, Anti-Harassment, IP Info, serviceops, SRE
wkandek added a parent task for T288375: IPInfo MediaWiki extension depends on presence of maxmind db in the container/host: T288844: Update MaxMind GeoIP2 license key and product IDs for application servers.
Aug 20 2021, 1:19 AM · serviceops, MW-on-K8s
wkandek added a subtask for T288844: Update MaxMind GeoIP2 license key and product IDs for application servers: T288375: IPInfo MediaWiki extension depends on presence of maxmind db in the container/host.
Aug 20 2021, 1:19 AM · Patch-For-Review, Anti-Harassment, IP Info, serviceops, SRE
wkandek added a comment to T288375: IPInfo MediaWiki extension depends on presence of maxmind db in the container/host.

Regarding 1 - a maxmind db change requires a new MediaWiki image and deployment. I believe the current frequency of MediaWiki deployments (multiple times a week) is good enough for the maxmind databases. The automated update of databases to the build system and the inclusion in the next image should be good enough.

Aug 20 2021, 1:14 AM · serviceops, MW-on-K8s

Aug 7 2021

wkandek triaged T288375: IPInfo MediaWiki extension depends on presence of maxmind db in the container/host as Medium priority.
Aug 7 2021, 5:14 PM · serviceops, MW-on-K8s
wkandek added a project to T288375: IPInfo MediaWiki extension depends on presence of maxmind db in the container/host: serviceops.
Aug 7 2021, 5:12 PM · serviceops, MW-on-K8s

Aug 6 2021

wkandek created T288375: IPInfo MediaWiki extension depends on presence of maxmind db in the container/host.
Aug 6 2021, 8:52 PM · serviceops, MW-on-K8s

Jul 30 2021

wkandek added a comment to T286463: mw2383 is misbehaving .

Papaul, Effie is on vacation for 2 more weeks. Just FYI, so don't expect any action here soon.

Jul 30 2021, 3:30 AM · SRE, ops-codfw

Jul 26 2021

wkandek added a project to T280881: New Service Request Toolhub: serviceops.
Jul 26 2021, 3:35 PM · Patch-For-Review, serviceops, Toolhub, Service-deployment-requests, Services, SRE

Jul 5 2021

wkandek updated the task description for T286032: Switch buffer re-partition - Eqiad Row A.
Jul 5 2021, 3:09 PM · Traffic, cloud-services-team (Kanban), DBA, Infrastructure-Foundations, SRE, netops

Jul 3 2021

wkandek triaged T286080: Story idea for Blog: June 2021 DC Switchover as Medium priority.
Jul 3 2021, 12:58 AM · Datacenter-Switchover, serviceops, Technical-blog-posts
wkandek moved T286080: Story idea for Blog: June 2021 DC Switchover from Incoming 🐫 to Doing 😎 on the serviceops board.
Jul 3 2021, 12:58 AM · Datacenter-Switchover, serviceops, Technical-blog-posts
wkandek added a comment to T286080: Story idea for Blog: June 2021 DC Switchover.

I have added a bit of text to explain services

Jul 3 2021, 12:42 AM · Datacenter-Switchover, serviceops, Technical-blog-posts

Jul 2 2021

wkandek added a comment to T280497: Benchmark performance of MediaWiki on k8s.

Is this the dashboard? https://grafana.wikimedia.org/d/U7JT--knk/joe-k8s-mwdebug?viewPanel=70&orgId=1&from=1625227688488&to=1625246654342

Jul 2 2021, 5:27 PM · Patch-For-Review, Performance-Team (Radar), MW-on-K8s, serviceops, SRE

Jul 1 2021

wkandek added a comment to T285806: Document communication expectations around planning a DC switchover.

Thanks everybody for the feedback on the communications for the DC switchover process. We will spend some time this quarter (Q1) in working through the typical timelines for the process and define the groups involved and the best mechanism to make sure that everybody is aware of the status and progress. Please take a look at https://wikitech.wikimedia.org/wiki/Switch_Datacenter and https://wikitech.wikimedia.org/wiki/Switch_Datacenter/Coordination to see if what we have captured there makes sense.

Jul 1 2021, 7:18 PM · serviceops, SRE, Datacenter-Switchover

Jun 30 2021

wkandek updated subscribers of T268819: Blubber needs to check if a user is present before creating it as part of its runs stanza.
Jun 30 2021, 12:40 PM · Release-Engineering-Team (Seen), serviceops, MW-on-K8s, Release Pipeline (Blubber)
wkandek added a project to T285806: Document communication expectations around planning a DC switchover: serviceops.
Jun 30 2021, 2:55 AM · serviceops, SRE, Datacenter-Switchover

Jun 24 2021

wkandek updated the task description for T281515: June 2021 Datacenter switchover.
Jun 24 2021, 5:46 PM · serviceops, Performance-Team (Radar), Datacenter-Switchover, SRE

Jun 23 2021

wkandek moved T274462: Logging for GitLab from Incoming 🐫 to Next up 🥌 on the serviceops board.
Jun 23 2021, 1:46 PM · Patch-For-Review, observability, SRE Observability (FY2021/2022-Q1), serviceops, Wikimedia-Logstash, User-brennen, GitLab (Initialization)
wkandek added a project to T274462: Logging for GitLab: serviceops.
Jun 23 2021, 1:46 PM · Patch-For-Review, observability, SRE Observability (FY2021/2022-Q1), serviceops, Wikimedia-Logstash, User-brennen, GitLab (Initialization)
wkandek moved T275170: Define monitoring for gitlab from Incoming 🐫 to Next up 🥌 on the serviceops board.
Jun 23 2021, 1:45 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability
wkandek added a project to T275170: Define monitoring for gitlab: serviceops.
Jun 23 2021, 1:45 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability
wkandek moved T274463: Backups for GitLab from Incoming 🐫 to Doing 😎 on the serviceops board.
Jun 23 2021, 1:44 PM · GitLab, serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen
wkandek added a project to T274463: Backups for GitLab: serviceops.
Jun 23 2021, 1:43 PM · GitLab, serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen

Jun 21 2021

wkandek moved T281515: June 2021 Datacenter switchover from Incoming 🐫 to Doing 😎 on the serviceops board.
Jun 21 2021, 4:40 AM · serviceops, Performance-Team (Radar), Datacenter-Switchover, SRE
wkandek added a project to T281515: June 2021 Datacenter switchover: serviceops.
Jun 21 2021, 4:39 AM · serviceops, Performance-Team (Radar), Datacenter-Switchover, SRE

Jun 16 2021

wkandek added a comment to T285024: OTRS queue "Probably-Spam" does not render on screen.

Notes by Quiddity:

Jun 16 2021, 2:23 AM · serviceops
wkandek created T285024: OTRS queue "Probably-Spam" does not render on screen.
Jun 16 2021, 12:59 AM · serviceops

Jun 14 2021

wkandek added a comment to T284773: Requesting access to mwmaint1002 for mepps.

Let's use the restricted group

Jun 14 2021, 6:53 PM · SRE, SRE-Access-Requests

Jun 9 2021

wkandek added a comment to T274463: Backups for GitLab.

3 days duration confirmed as sufficient.

Jun 9 2021, 8:34 PM · GitLab, serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen
wkandek changed the status of T273137: decommission thumbor100[34].eqiad.wmnet from Open to Stalled.

We are waiting for Thumbor to migrate to k8s before retiring these servers. New servers that were originally been purchased for thumbor have been repurposed for mediawiki. I will check with the dev team on where they are with the k8s migration.

Jun 9 2021, 3:15 PM · serviceops, decommission-hardware
wkandek changed the status of T273141: decommission thumbor200[12].codfw.wmnet from Open to Stalled.

We are waiting for Thumbor to migrate to k8s before retiring these servers. New servers that were originally been purchased for thumbor have been repurposed for mediawiki. I will check with the dev team on where they are with the k8s migration.

Jun 9 2021, 3:14 PM · serviceops, decommission-hardware

Jun 2 2021

wkandek added a comment to T274463: Backups for GitLab.

gitlab is configured to store backups in /srv/gitlab-backup

Jun 2 2021, 6:55 PM · GitLab, serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen

Jun 1 2021

wkandek added a comment to T274463: Backups for GitLab.

I think that is something that Jelto (our new SRE - starts June 7) can handle, i.e. add a second disk with the right dimensions.

Jun 1 2021, 6:51 PM · GitLab, serviceops, Data-Persistence-Backup, Patch-For-Review, User-brennen
wkandek added a comment to T283850: Requesting access to gitlab1001 for Ahmon Dancy (@dancy).

approved

Jun 1 2021, 1:58 PM · SRE, SRE-Access-Requests

May 27 2021

wkandek added a comment to T275170: Define monitoring for gitlab.

On my test machine I see the following:

May 27 2021, 10:11 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability
wkandek added a comment to T275170: Define monitoring for gitlab.

I think we should enable the following exporters:

May 27 2021, 4:53 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability

May 18 2021

wkandek added a comment to T283022: Requesting access to gitlab1001 for Brennen Bearnes (brennen).

Approved.

May 18 2021, 9:32 PM · SRE, SRE-Access-Requests
wkandek updated the task description for T283022: Requesting access to gitlab1001 for Brennen Bearnes (brennen).
May 18 2021, 9:32 PM · SRE, SRE-Access-Requests

May 7 2021

wkandek claimed T88643: Standards and health score for existing and new services.

The SLO project addresses some points: https://wikitech.wikimedia.org/wiki/SLO - we will continue to add applications on wikitech as they get defined together with engineering and product. The SLO project has been going on in FY20/21 and will continue in FY21/22.

May 7 2021, 2:40 PM · serviceops, Services, Service-Architecture
wkandek added a comment to T274462: Logging for GitLab.

Latest state that I am aware of:

May 7 2021, 2:33 PM · Patch-For-Review, observability, SRE Observability (FY2021/2022-Q1), serviceops, Wikimedia-Logstash, User-brennen, GitLab (Initialization)

Apr 20 2021

wkandek updated subscribers of T238909: Proposal: simplify set up of a new load-balanced service on kubernetes.
Apr 20 2021, 8:56 PM · SRE, Prod-Kubernetes, Pybal, Traffic, serviceops

Apr 14 2021

wkandek added a comment to T275170: Define monitoring for gitlab.

For Prometheus/Grafana:

Apr 14 2021, 8:21 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability

Apr 7 2021

wkandek added a subtask for T274458: Remove Speed & Function blockers for GitLab work: T279545: Gitlab Installation Procedure.
Apr 7 2021, 2:44 PM · Release-Engineering-Team (Doing), User-brennen, GitLab (Initialization)
wkandek added a parent task for T279545: Gitlab Installation Procedure: T274458: Remove Speed & Function blockers for GitLab work.
Apr 7 2021, 2:44 PM · Patch-For-Review, GitLab (Initialization)
wkandek updated subscribers of T279545: Gitlab Installation Procedure.
Apr 7 2021, 2:43 PM · Patch-For-Review, GitLab (Initialization)
wkandek created T279545: Gitlab Installation Procedure.
Apr 7 2021, 2:42 PM · Patch-For-Review, GitLab (Initialization)

Mar 23 2021

wkandek assigned T276148: SSH Access of Git data in GitLab to Dzahn.
Mar 23 2021, 8:48 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)
wkandek added a comment to T276148: SSH Access of Git data in GitLab.

To provide as little obstacles for developers as possible access through port 22 is the preferred option.

Mar 23 2021, 8:48 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)
wkandek added a comment to T276148: SSH Access of Git data in GitLab.
Mar 23 2021, 6:34 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)
wkandek closed T274459: Eqiad: 2 VM request for GitLab as Resolved.

Let's close and reopen if a second server becomes necessary.

Mar 23 2021, 5:00 PM · GitLab (Initialization), Patch-For-Review, User-brennen, vm-requests, SRE
wkandek closed T274459: Eqiad: 2 VM request for GitLab, a subtask of T274458: Remove Speed & Function blockers for GitLab work, as Resolved.
Mar 23 2021, 5:00 PM · Release-Engineering-Team (Doing), User-brennen, GitLab (Initialization)
wkandek assigned T276459: Outgoing mail for Gitlab to brennen.
Mar 23 2021, 4:55 PM · Mail, GitLab (Initialization)
wkandek added a comment to T276459: Outgoing mail for Gitlab.

gitlab1001 has the standard exim mail package installed, so local e-mail works already. What does Gitlab need?

Mar 23 2021, 4:54 PM · Mail, GitLab (Initialization)

Mar 17 2021

wkandek renamed T277706: Request increased quota for k8splay Cloud VPS project from Request increased quota for <Replace Me> Cloud VPS project to Request increased quota for k8splay Cloud VPS project.
Mar 17 2021, 9:08 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
wkandek created T277706: Request increased quota for k8splay Cloud VPS project.
Mar 17 2021, 9:08 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Mar 10 2021

wkandek added a comment to T276673: Certificate Management for GitLab.

https://docs.gitlab.com/omnibus/settings/nginx.html#update-the-ssl-certificates has the command to execute when a certificate is updated: gitlab-ctl hup nginx

Mar 10 2021, 10:12 PM · GitLab (Initialization)

Mar 9 2021

wkandek added a comment to T276170: DNS for GitLab.

Let's go with the simpler solution and use the CNAME.

Mar 9 2021, 11:33 PM · SRE, Traffic, DNS, serviceops, Release-Engineering-Team-TODO, User-brennen, GitLab (Initialization)
wkandek added a comment to T276170: DNS for GitLab.

gerrit.wikimedia.org lives on a second IP address on gerrit1001. Should we follow that model here as well?

Mar 9 2021, 2:43 PM · SRE, Traffic, DNS, serviceops, Release-Engineering-Team-TODO, User-brennen, GitLab (Initialization)

Mar 6 2021

wkandek updated the task description for T276673: Certificate Management for GitLab.
Mar 6 2021, 5:03 PM · GitLab (Initialization)
wkandek added a comment to T276144: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed).

I created https://phabricator.wikimedia.org/T276673 to track the decison on how to manage HTTPS certificates.

Mar 6 2021, 4:47 PM · SRE, Traffic, Patch-For-Review, User-brennen, GitLab (Initialization), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3))
wkandek added a subtask for T274458: Remove Speed & Function blockers for GitLab work: T276673: Certificate Management for GitLab.
Mar 6 2021, 4:46 PM · Release-Engineering-Team (Doing), User-brennen, GitLab (Initialization)
wkandek added a parent task for T276673: Certificate Management for GitLab: T274458: Remove Speed & Function blockers for GitLab work.
Mar 6 2021, 4:46 PM · GitLab (Initialization)
wkandek created T276673: Certificate Management for GitLab.
Mar 6 2021, 4:45 PM · GitLab (Initialization)

Mar 4 2021

wkandek moved T273521: Create restricted docker-registry namespace for security patched images from Incoming 🐫 to Doing 😎 on the serviceops board.
Mar 4 2021, 10:39 PM · Patch-For-Review, serviceops, Release-Engineering-Team-TODO, Release Pipeline, MW-on-K8s
wkandek added a comment to T276351: Requesting access to restricted for Daimona.

Approved.

Mar 4 2021, 3:31 PM · SRE, SRE-Access-Requests
wkandek added a subtask for T274458: Remove Speed & Function blockers for GitLab work: T276459: Outgoing mail for Gitlab.
Mar 4 2021, 2:22 PM · Release-Engineering-Team (Doing), User-brennen, GitLab (Initialization)
wkandek added a parent task for T276459: Outgoing mail for Gitlab: T274458: Remove Speed & Function blockers for GitLab work.
Mar 4 2021, 2:22 PM · Mail, GitLab (Initialization)
wkandek created T276459: Outgoing mail for Gitlab.
Mar 4 2021, 2:19 PM · Mail, GitLab (Initialization)

Mar 3 2021

wkandek assigned T274459: Eqiad: 2 VM request for GitLab to Dzahn.
Mar 3 2021, 12:40 AM · GitLab (Initialization), Patch-For-Review, User-brennen, vm-requests, SRE
wkandek reopened T274459: Eqiad: 2 VM request for GitLab as "Open".

Please recreate the 2 VMs in the VLAN that allows for direct external IP addresses.

Mar 3 2021, 12:40 AM · GitLab (Initialization), Patch-For-Review, User-brennen, vm-requests, SRE
wkandek reopened T274459: Eqiad: 2 VM request for GitLab, a subtask of T274458: Remove Speed & Function blockers for GitLab work, as Open.
Mar 3 2021, 12:40 AM · Release-Engineering-Team (Doing), User-brennen, GitLab (Initialization)

Mar 2 2021

wkandek added a comment to T276148: SSH Access of Git data in GitLab.

A second sshd daemon for port 29418 concentrates all configuration for gitlab/ssh access in a separate configuration file and does not touch the "normal" config file. That seems to me a more robust way of configuring, minimizing impact on sshd on port 22. sshd on port 29418 can then simply be limited to user git.

Mar 2 2021, 3:01 PM · Patch-For-Review, Release-Engineering-Team (Doing), SRE, User-brennen, GitLab (Initialization)
wkandek added a comment to T276170: DNS for GitLab.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/667731

Mar 2 2021, 2:41 PM · SRE, Traffic, DNS, serviceops, Release-Engineering-Team-TODO, User-brennen, GitLab (Initialization)
wkandek added a comment to T274171: (Need By: TBD) rack/setup/install (35) mw2377 and upwards.

Yes, we can do all servers in A3 at once.

Mar 2 2021, 1:00 PM · SRE, ops-codfw, DC-Ops

Mar 1 2021

wkandek assigned T276144: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed) to Dzahn.
Mar 1 2021, 8:48 PM · SRE, Traffic, Patch-For-Review, User-brennen, GitLab (Initialization), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3))

Feb 19 2021

Krenair awarded T274953: Access group for Gitlab contractors a Dislike token.
Feb 19 2021, 8:20 PM · GitLab, User-brennen, SRE, SRE-Access-Requests

Feb 18 2021

wkandek added a subtask for T274458: Remove Speed & Function blockers for GitLab work: T275170: Define monitoring for gitlab.
Feb 18 2021, 10:34 PM · Release-Engineering-Team (Doing), User-brennen, GitLab (Initialization)
wkandek added a parent task for T275170: Define monitoring for gitlab: T274458: Remove Speed & Function blockers for GitLab work.
Feb 18 2021, 10:34 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability
wkandek created T275170: Define monitoring for gitlab.
Feb 18 2021, 10:33 PM · Patch-For-Review, serviceops, GitLab (Initialization), observability

Feb 16 2021

wkandek updated subscribers of T274953: Access group for Gitlab contractors.
Feb 16 2021, 10:53 PM · GitLab, User-brennen, SRE, SRE-Access-Requests
wkandek created T274953: Access group for Gitlab contractors.
Feb 16 2021, 10:51 PM · GitLab, User-brennen, SRE, SRE-Access-Requests

Feb 5 2021

wkandek moved T273807: New Service Request - Calculator Service from Incoming 🐫 to Doing 😎 on the serviceops board.
Feb 5 2021, 11:25 PM · SRE, Service-deployment-requests, Services, serviceops
wkandek triaged T273807: New Service Request - Calculator Service as Low priority.
Feb 5 2021, 11:24 PM · SRE, Service-deployment-requests, Services, serviceops

Feb 3 2021

wkandek committed rBECS179e9c72b52e: Minikube files added, new testmode on server (authored by wkandek).
Minikube files added, new testmode on server
Feb 3 2021, 8:34 PM
wkandek created T273807: New Service Request - Calculator Service.
Feb 3 2021, 7:29 PM · SRE, Service-deployment-requests, Services, serviceops

Jan 28 2021

wkandek updated subscribers of T273151: Implement complete cycle for kubernetes production for calculator-service.
Jan 28 2021, 5:12 AM · Patch-For-Review, Prod-Kubernetes
wkandek claimed T273151: Implement complete cycle for kubernetes production for calculator-service.
Jan 28 2021, 5:08 AM · Patch-For-Review, Prod-Kubernetes
wkandek created T273151: Implement complete cycle for kubernetes production for calculator-service.
Jan 28 2021, 5:05 AM · Patch-For-Review, Prod-Kubernetes
wkandek created T273142: decommission francium.eqiad.wmnet.
Jan 28 2021, 1:12 AM · SRE, ops-eqiad, serviceops, decommission-hardware
wkandek renamed T273138: decommission scb200[1234].codfw.wmnet from decommission scb200[1234].eqiad.wmnet to decommission scb200[1234].codfw.wmnet.
Jan 28 2021, 1:09 AM · serviceops, decommission-hardware
wkandek renamed T273140: decommission rdb200[3456].codfw.wmnet from decommission rdb200[34].eqiad.wmnet to decommission rdb200[34].codfw.wmnet.
Jan 28 2021, 1:09 AM · SRE, ops-codfw, DC-Ops, serviceops, decommission-hardware
wkandek updated the task description for T273141: decommission thumbor200[12].codfw.wmnet.
Jan 28 2021, 1:08 AM · serviceops, decommission-hardware
wkandek created T273141: decommission thumbor200[12].codfw.wmnet.
Jan 28 2021, 1:08 AM · serviceops, decommission-hardware
wkandek added a project to T273136: decommission scb100[1234].eqiad.wmnet: serviceops.
Jan 28 2021, 1:06 AM · serviceops, decommission-hardware
wkandek added a project to T273137: decommission thumbor100[34].eqiad.wmnet: serviceops.
Jan 28 2021, 1:06 AM · serviceops, decommission-hardware
wkandek added a project to T273138: decommission scb200[1234].codfw.wmnet: serviceops.
Jan 28 2021, 1:05 AM · serviceops, decommission-hardware
wkandek added a project to T273139: decommission rdb100[56].eqiad.wmnet: serviceops.
Jan 28 2021, 1:05 AM · serviceops, decommission-hardware
wkandek added a project to T273140: decommission rdb200[3456].codfw.wmnet: serviceops.
Jan 28 2021, 1:05 AM · SRE, ops-codfw, DC-Ops, serviceops, decommission-hardware