Page MenuHomePhabricator

Marostegui (Manuel Aróstegui)
Staff Database Administrator

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Sep 1 2016, 6:48 AM (393 w, 4 d)
Availability
Available
IRC Nick
marostegui
LDAP User
Marostegui
MediaWiki User
MArostegui (WMF) [ Global Accounts ]

TZ: UTC +1/+2

Recent Activity

Yesterday

Marostegui reopened T352098: Requesting access to "researchers" and "analytics-privatedata-users" for Xiao Xiao as "Open".

We just had an email letting us know that this key is now being used in WMCS as well

Mon, Mar 18, 9:08 AM · SRE, SRE-Access-Requests

Thu, Mar 14

Marostegui closed T360118: Many jsonTruncated production errors: "Unable to store text to external storage" as Resolved.

It was, es2020 (and only that host) had a mis configured max_allowed_packet (which is strange as this is handled by puppet) which was way smaller than the normal and it is okay on file.
Logstash now seems happy. Thanks a lot for reporting this!

Thu, Mar 14, 3:37 PM · DBA, MediaWiki-Core-Revision-backend, Wikimedia-production-error
Marostegui added a comment to T360118: Many jsonTruncated production errors: "Unable to store text to external storage".

I am checking if it might be related to multi-master, which was enabled in the morning.

Thu, Mar 14, 3:30 PM · DBA, MediaWiki-Core-Revision-backend, Wikimedia-production-error
Marostegui updated subscribers of T360029: Integrate dbctl IP changes as part of VLAN changes. .

We've been discussing with @Ladsgroup that we need to double check if he IPs are really used for dbct /MW or it is just some tech debt. dbctl isn't too old, so if they were added it was probably for a reason, but maybe they're not needed anymore. If that is the case that'd simplify things a lot.

Thu, Mar 14, 1:42 PM · Data-Persistence, SRE, Infrastructure-Foundations
Marostegui closed T358199: Database pre-switchover tasks March 2024 as Resolved.

This is all done

Thu, Mar 14, 9:51 AM · DBA, Datacenter-Switchover
Marostegui closed T358199: Database pre-switchover tasks March 2024, a subtask of T357547: ☂️ Northward Datacentre Switchover (March 2024) , as Resolved.
Thu, Mar 14, 9:50 AM · Patch-For-Review, Datacenter-Switchover, Data-Persistence, SRE Observability (FY2023/2024-Q3), collaboration-services, observability, serviceops, DC-Ops, Traffic
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Thu, Mar 14, 9:50 AM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Thu, Mar 14, 9:36 AM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Thu, Mar 14, 9:26 AM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Thu, Mar 14, 9:21 AM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Thu, Mar 14, 9:16 AM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Thu, Mar 14, 9:09 AM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Thu, Mar 14, 9:06 AM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Thu, Mar 14, 9:01 AM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Thu, Mar 14, 8:50 AM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Thu, Mar 14, 8:36 AM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Thu, Mar 14, 8:10 AM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Thu, Mar 14, 7:05 AM · DBA, Datacenter-Switchover
Marostegui created P58790 (An Untitled Masterwork).
Thu, Mar 14, 6:59 AM

Wed, Mar 13

Marostegui changed the status of T359940: hw troubleshooting: Unidentified for db1246.eqiad.wmnet from In Progress to Open.

So the filesystem was totally corrupted. A full reimage (deleting all the partitions) seems to have fixed it.
I am recloning the host right now and I will leave it replicating for a few days to make sure the storage is stable.

Wed, Mar 13, 4:17 PM · DBA, SRE, ops-eqiad, DC-Ops
Marostegui added a comment to T359940: hw troubleshooting: Unidentified for db1246.eqiad.wmnet.

Rebooting the server resulted in the same XFS errors and the OS doesn't boot past mounting the filesystem. I am going to reimage it in case it was all corrupted after the crash. Hopefully this will fix it, if not...it means the storage is physically broken. Will report back

Wed, Mar 13, 2:32 PM · DBA, SRE, ops-eqiad, DC-Ops
Marostegui updated subscribers of T359940: hw troubleshooting: Unidentified for db1246.eqiad.wmnet.

I've managed to get it boot up past the grub and it looks storage related:

Starting default.target
[85598.425324] XFS (dm-0): Metadata corruption detected at xfs_agi_verify+0x11a/0x170 [xfs], xfs_agi block 0x1e3a5e02
[85598.435932] XFS (dm-0): Unmount and run xfs_repair
[85598.440731] XFS (dm-0): First 128 bytes of corrupted metadata buffer:
[85598.447172] 00000000: 58 41 47 49 00 00 00 01 00 00 00 01 03 c7 4b c0  XAGI..........K.
[85598.455178] 00000010: 00 00 00 40 00 00 00 03 00 00 00 01 00 00 00 3b  ...@...........;
[85598.463176] 00000020: 00 00 00 c0 ff ff ff ff ff ff ff ff 00 00 00 c1  ................
[85598.471176] 00000030: 00 00 00 c2 00 00 00 c3 00 00 00 c4 ff ff ff ff  ................
[85598.479175] 00000040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[85598.487177] 00000050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[85598.495183] 00000060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
[85598.503182] 00000070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
Wed, Mar 13, 2:00 PM · DBA, SRE, ops-eqiad, DC-Ops
Marostegui added a comment to T359940: hw troubleshooting: Unidentified for db1246.eqiad.wmnet.

For what is worth, this error is also present on the HW logs, even though it is a month old, it might be an indication of something else

-------------------------------------------------------------------------------
Record:      26
Date/Time:   02/20/2024 06:53:35
Source:      system
Severity:    Critical
Description: A fatal error was detected on a component at bus 4 device 0 function 0.
-------------------------------------------------------------------------------
Wed, Mar 13, 1:49 PM · DBA, SRE, ops-eqiad, DC-Ops
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Wed, Mar 13, 1:45 PM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Wed, Mar 13, 1:44 PM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Wed, Mar 13, 1:41 PM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Wed, Mar 13, 1:40 PM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Wed, Mar 13, 1:36 PM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Wed, Mar 13, 1:35 PM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Wed, Mar 13, 1:33 PM · DBA, Datacenter-Switchover
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Wed, Mar 13, 1:30 PM · DBA, Datacenter-Switchover
Marostegui created T360029: Integrate dbctl IP changes as part of VLAN changes. .
Wed, Mar 13, 1:30 PM · Data-Persistence, SRE, Infrastructure-Foundations
Marostegui updated the task description for T358199: Database pre-switchover tasks March 2024.
Wed, Mar 13, 8:57 AM · DBA, Datacenter-Switchover
Marostegui moved T358199: Database pre-switchover tasks March 2024 from Blocked to In progress on the DBA board.
Wed, Mar 13, 8:44 AM · DBA, Datacenter-Switchover
Marostegui closed T359490: Requesting access to mwmaint for rkhan / Himejijo as Resolved.

This has been done. Give it 30-45 minutes for the change to get spread across the infra.

Wed, Mar 13, 7:25 AM · SRE, SRE-Access-Requests
Marostegui updated the task description for T359490: Requesting access to mwmaint for rkhan / Himejijo.
Wed, Mar 13, 7:24 AM · SRE, SRE-Access-Requests

Tue, Mar 12

Marostegui closed T359092: Requesting access to kubernetes deployment for tjones as Resolved.

This has been deployed. Please give it 30-45 minutes for puppet to run everywhere and spread the change.

Tue, Mar 12, 1:09 PM · Data-Platform-SRE (2024.03.04 - 2024.03.24), SRE, SRE-Access-Requests
Marostegui moved T359490: Requesting access to mwmaint for rkhan / Himejijo from Awaiting User Input to Manager/NDA Approval/Confirmation on the SRE-Access-Requests board.
Tue, Mar 12, 8:28 AM · SRE, SRE-Access-Requests
Marostegui updated the task description for T359490: Requesting access to mwmaint for rkhan / Himejijo.
Tue, Mar 12, 8:28 AM · SRE, SRE-Access-Requests
Marostegui moved T359092: Requesting access to kubernetes deployment for tjones from Manager/NDA Approval/Confirmation to Patch in Review on the SRE-Access-Requests board.
Tue, Mar 12, 8:25 AM · Data-Platform-SRE (2024.03.04 - 2024.03.24), SRE, SRE-Access-Requests
Marostegui updated the task description for T359092: Requesting access to kubernetes deployment for tjones.
Tue, Mar 12, 7:18 AM · Data-Platform-SRE (2024.03.04 - 2024.03.24), SRE, SRE-Access-Requests

Mon, Mar 11

Marostegui updated the task description for T359490: Requesting access to mwmaint for rkhan / Himejijo.
Mon, Mar 11, 5:46 PM · SRE, SRE-Access-Requests
Marostegui updated subscribers of T359490: Requesting access to mwmaint for rkhan / Himejijo.

@thcipriani would you approve this request to mwmaint?

Mon, Mar 11, 5:40 PM · SRE, SRE-Access-Requests
Marostegui updated the task description for T359490: Requesting access to mwmaint for rkhan / Himejijo.
Mon, Mar 11, 5:35 PM · SRE, SRE-Access-Requests
Marostegui added a project to T359702: Degraded RAID on dumpsdata1007: Data-Engineering.
Mon, Mar 11, 10:54 AM · Data-Engineering, SRE, ops-eqiad

Sun, Mar 10

Marostegui added a comment to T359490: Requesting access to mwmaint for rkhan / Himejijo.

Yes but on the original ticket your username listed is Himejijo, which doesn't match with that email. That's why I believe you just need to confirm that that username is really meant to be rkhan

Sun, Mar 10, 9:47 AM · SRE, SRE-Access-Requests
Marostegui added a comment to T359490: Requesting access to mwmaint for rkhan / Himejijo.

@Marostegui I'm sorry I'm still confused about which is which.

Is it associated with the rkhan-ctr@wikipedia.org address?

Sun, Mar 10, 5:23 AM · SRE, SRE-Access-Requests

Fri, Mar 8

Marostegui moved T359561: Add user fabfur to analytics-privatedata-users from Untriaged to In Discussion on the SRE-Access-Requests board.
Fri, Mar 8, 7:21 AM · SRE, SRE-Access-Requests, Data-Platform-SRE
Marostegui moved T358658: Requesting access to wmf-nda, analytics-private-data, analytics-product for kcvelaga from Untriaged to Awaiting User Input on the SRE-Access-Requests board.
Fri, Mar 8, 7:21 AM · Patch-For-Review, SRE, SRE-Access-Requests
Marostegui closed T359597: db2124 depooled with index corruption as Resolved.

Host recloned and being slowly repooled.
Thanks @RLazarus for addressing this incident!

Fri, Mar 8, 7:21 AM · DBA
Marostegui moved T359597: db2124 depooled with index corruption from Triage to In progress on the DBA board.
Fri, Mar 8, 6:31 AM · DBA
Marostegui claimed T359597: db2124 depooled with index corruption.

I could probably fix it right away, but I think I am going to quickly reclone it instead

Fri, Mar 8, 6:27 AM · DBA

Thu, Mar 7

Marostegui closed T359417: Requesting access to analytics-privatedata-users for bdgreenlee as Resolved.

I have merged and deployed the change. Please give it 30 minutes or so for the change to spread everywhere across the fleet.

Thu, Mar 7, 5:41 PM · SRE, SRE-Access-Requests
Marostegui moved T359417: Requesting access to analytics-privatedata-users for bdgreenlee from Awaiting User Input to Patch in Review on the SRE-Access-Requests board.
Thu, Mar 7, 5:29 PM · SRE, SRE-Access-Requests
Marostegui added a comment to T359561: Add user fabfur to analytics-privatedata-users.

Yeah, I think the whole procedure needs to be followed (using the correct template for this ticket too)

Thu, Mar 7, 5:19 PM · SRE, SRE-Access-Requests, Data-Platform-SRE
Marostegui added a comment to T359561: Add user fabfur to analytics-privatedata-users.

@Fabfur as a SRE I assume you'd self serve?

Thu, Mar 7, 5:15 PM · SRE, SRE-Access-Requests, Data-Platform-SRE
Marostegui assigned T359561: Add user fabfur to analytics-privatedata-users to Fabfur.
Thu, Mar 7, 5:15 PM · SRE, SRE-Access-Requests, Data-Platform-SRE
Marostegui added a comment to T359417: Requesting access to analytics-privatedata-users for bdgreenlee.

Which access you specifically need for analytics-private-users? https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Analytics_shell_(posix)_groups_explained

Thu, Mar 7, 4:26 PM · SRE, SRE-Access-Requests
Marostegui updated the task description for T359417: Requesting access to analytics-privatedata-users for bdgreenlee.
Thu, Mar 7, 4:05 PM · SRE, SRE-Access-Requests
Marostegui closed T354015: DBQueryDisconnectedError upon editing en:Template:COVID-19 pandemic data as Resolved.

Closing this as the change is merged and will be deployed with the next train. Thanks everyone!

Thu, Mar 7, 12:03 PM · MW-1.42-notes (1.42.0-wmf.22; 2024-03-12), Patch-For-Review, DBA, Wikimedia-Slow-DB-Query, MediaWiki-Core-Revision-backend, Wikimedia-production-error
Marostegui updated the task description for T359490: Requesting access to mwmaint for rkhan / Himejijo.
Thu, Mar 7, 6:39 AM · SRE, SRE-Access-Requests
Marostegui added a comment to T359490: Requesting access to mwmaint for rkhan / Himejijo.

The user rkhan does exist but it is not associated to that email. Can you post the email it is associated to that user and if you really meant rkhan instead of Himejijo

Thu, Mar 7, 6:36 AM · SRE, SRE-Access-Requests
Marostegui moved T359490: Requesting access to mwmaint for rkhan / Himejijo from Untriaged to Awaiting User Input on the SRE-Access-Requests board.
Thu, Mar 7, 6:34 AM · SRE, SRE-Access-Requests
Marostegui added a comment to T359490: Requesting access to mwmaint for rkhan / Himejijo.

@Himejijo are you sure that is your wikitech user name and your email? I cannot find anything for any of those two.

Thu, Mar 7, 6:34 AM · SRE, SRE-Access-Requests
Marostegui triaged T359490: Requesting access to mwmaint for rkhan / Himejijo as Medium priority.
Thu, Mar 7, 6:33 AM · SRE, SRE-Access-Requests
Marostegui moved T359417: Requesting access to analytics-privatedata-users for bdgreenlee from Untriaged to Awaiting User Input on the SRE-Access-Requests board.
Thu, Mar 7, 6:29 AM · SRE, SRE-Access-Requests
Marostegui added a comment to T359417: Requesting access to analytics-privatedata-users for bdgreenlee.

Waiting for ssh out of band verification

Thu, Mar 7, 6:29 AM · SRE, SRE-Access-Requests
Marostegui updated the task description for T359417: Requesting access to analytics-privatedata-users for bdgreenlee.
Thu, Mar 7, 6:29 AM · SRE, SRE-Access-Requests

Wed, Mar 6

Marostegui updated the task description for T359417: Requesting access to analytics-privatedata-users for bdgreenlee.
Wed, Mar 6, 6:59 PM · SRE, SRE-Access-Requests
Marostegui updated the task description for T359417: Requesting access to analytics-privatedata-users for bdgreenlee.
Wed, Mar 6, 6:57 PM · SRE, SRE-Access-Requests
Marostegui triaged T359417: Requesting access to analytics-privatedata-users for bdgreenlee as Medium priority.
Wed, Mar 6, 5:23 PM · SRE, SRE-Access-Requests
Marostegui added a comment to T359417: Requesting access to analytics-privatedata-users for bdgreenlee.

@odimitrijevic I assume you are also their manager and hence approving for manager and analytics group?

Wed, Mar 6, 5:23 PM · SRE, SRE-Access-Requests
Marostegui merged T359197: Grant Access to analytics-privatedata-users for bdgreenlee into T359417: Requesting access to analytics-privatedata-users for bdgreenlee.
Wed, Mar 6, 5:23 PM · SRE, SRE-Access-Requests
Marostegui merged task T359197: Grant Access to analytics-privatedata-users for bdgreenlee into T359417: Requesting access to analytics-privatedata-users for bdgreenlee.
Wed, Mar 6, 5:23 PM · SRE, LDAP-Access-Requests
Marostegui closed T359208: Grant Access to ldap/wmf for FebinBellamy as Resolved.

This is all done

Wed, Mar 6, 2:30 PM · SRE, LDAP-Access-Requests
Marostegui added a member for WMF-NDA: FBellamy-WMF.
Wed, Mar 6, 2:27 PM
Marostegui claimed T359208: Grant Access to ldap/wmf for FebinBellamy.
Wed, Mar 6, 2:05 PM · SRE, LDAP-Access-Requests
Marostegui moved T355422: Productionize db2196-db2220 from Ready to In progress on the DBA board.
Wed, Mar 6, 6:37 AM · Patch-For-Review, DBA
Marostegui updated the task description for T358746: Upgrade es5 to MariaDB 10.6.
Wed, Mar 6, 6:27 AM · DBA
Marostegui triaged T359197: Grant Access to analytics-privatedata-users for bdgreenlee as Medium priority.
Wed, Mar 6, 6:16 AM · SRE, LDAP-Access-Requests
Marostegui added a comment to T359197: Grant Access to analytics-privatedata-users for bdgreenlee.

@bdgreenlee please follow the ticket template at https://phabricator.wikimedia.org/maniphest/task/edit/form/8/

Wed, Mar 6, 6:16 AM · SRE, LDAP-Access-Requests
Marostegui triaged T359208: Grant Access to ldap/wmf for FebinBellamy as Medium priority.

@FBellamy-WMF we'd need your manager to approve this.

Wed, Mar 6, 6:13 AM · SRE, LDAP-Access-Requests
Marostegui moved T359208: Grant Access to ldap/wmf for FebinBellamy from Backlog to Manager Approval Pending on the LDAP-Access-Requests board.
Wed, Mar 6, 6:12 AM · SRE, LDAP-Access-Requests
Marostegui closed T359123: Grant Access to ldap/wmf for bdgreenlee as Resolved.

bdgreenlee added to WMF group.

Wed, Mar 6, 6:10 AM · SRE, LDAP-Access-Requests
Marostegui added a member for WMF-NDA: bdgreenlee.
Wed, Mar 6, 6:07 AM
Marostegui updated the task description for T359092: Requesting access to kubernetes deployment for tjones.
Wed, Mar 6, 6:06 AM · Data-Platform-SRE (2024.03.04 - 2024.03.24), SRE, SRE-Access-Requests
Marostegui added a comment to T354015: DBQueryDisconnectedError upon editing en:Template:COVID-19 pandemic data.

Thanks. Forcing a table rebuilt fixed db1186, but if we are adding the index, there's no point in spending time doing this anywhere else.

Wed, Mar 6, 6:06 AM · MW-1.42-notes (1.42.0-wmf.22; 2024-03-12), Patch-For-Review, DBA, Wikimedia-Slow-DB-Query, MediaWiki-Core-Revision-backend, Wikimedia-production-error

Tue, Mar 5

Marostegui added a comment to T355424: Productionize es[2035-2040].
[16:55:08] marostegui@cumin1002:~$ sudo cumin es20[35-40].codfw.wmnet 'lvextend -L+1T /dev/mapper/tank-data ; xfs_growfs /srv/ ; df -hT /srv'
6 hosts will be targeted:
es[2035-2040].codfw.wmnet
OK to proceed on 6 hosts? Enter the number of affected hosts to confirm or "q" to quit: 6
===== NODE GROUP =====
(6) es[2035-2040].codfw.wmnet
----- OUTPUT of 'lvextend -L+1T /...v/ ; df -hT /srv' -----
  Size of logical volume tank/data changed from 9.09 TiB (2384188 extents) to 10.09 TiB (2646332 extents).
  Logical volume tank/data successfully resized.
meta-data=/dev/mapper/tank-data  isize=512    agcount=32, agsize=76294016 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=0
data     =                       bsize=4096   blocks=2441408512, imaxpct=5
         =                       sunit=64     swidth=256 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 2441408512 to 2709843968
Filesystem            Type  Size  Used Avail Use% Mounted on
/dev/mapper/tank-data xfs    11T   73G   11T   1% /srv
Tue, Mar 5, 4:56 PM · DBA
Marostegui changed the status of T355424: Productionize es[2035-2040], a subtask of T355343: Q3:rack/setup/install es[2035-2040], from Stalled to Open.
Tue, Mar 5, 4:55 PM · SRE, Data-Persistence, ops-codfw, DC-Ops
Marostegui changed the status of T355424: Productionize es[2035-2040] from Stalled to Open.
Tue, Mar 5, 4:55 PM · DBA
Marostegui added a comment to T355343: Q3:rack/setup/install es[2035-2040].

Thank you so much!

Tue, Mar 5, 4:53 PM · SRE, Data-Persistence, ops-codfw, DC-Ops
Marostegui triaged T359163: Reclone m5 hosts from production into old hosts to simulate GTID cleaning up as Medium priority.
Tue, Mar 5, 1:23 PM · DBA
Marostegui created T359163: Reclone m5 hosts from production into old hosts to simulate GTID cleaning up.
Tue, Mar 5, 1:23 PM · DBA
Marostegui added a comment to T354015: DBQueryDisconnectedError upon editing en:Template:COVID-19 pandemic data.

Eqiad hosts are done, the following ones didn't change their query plan
db1186
db1218
db1207

Tue, Mar 5, 12:50 PM · MW-1.42-notes (1.42.0-wmf.22; 2024-03-12), Patch-For-Review, DBA, Wikimedia-Slow-DB-Query, MediaWiki-Core-Revision-backend, Wikimedia-production-error
Marostegui added a comment to T359141: decommission db2117.codfw.wmnet.

Thanks! @Jhancock.wm see above, you can proceed whenever you want.

Tue, Mar 5, 9:34 AM · SRE, DC-Ops, ops-codfw, DBA, decommission-hardware
Marostegui updated the task description for T355422: Productionize db2196-db2220.
Tue, Mar 5, 9:19 AM · Patch-For-Review, DBA
Marostegui added a parent task for T359141: decommission db2117.codfw.wmnet: T358741: Decommission db2196-db2220.
Tue, Mar 5, 9:18 AM · SRE, DC-Ops, ops-codfw, DBA, decommission-hardware
Marostegui added a subtask for T358741: Decommission db2196-db2220: T359141: decommission db2117.codfw.wmnet.
Tue, Mar 5, 9:18 AM · DBA
Marostegui updated the task description for T358741: Decommission db2196-db2220.
Tue, Mar 5, 9:18 AM · DBA
Marostegui updated the task description for T355422: Productionize db2196-db2220.
Tue, Mar 5, 9:18 AM · Patch-For-Review, DBA