Page MenuHomePhabricator
Create Task
Maniphest T421484

Reclaim parsoidtest1001 and testreduce1001
Open, In Progress, MediumPublicRequest
Actions

Description

This task will track the RETIREMENT and RECLAIM of server parsoidtest1001.eqiad.wmnet and testreduce1001.eqiad.wmnet

With the launch of updates to the decom cookbook, the majority of these steps can be handled by the service owners directly. The DC Ops team only gets involved once the system has been fully removed from service and powered down by the decommission cookbook.

parsoidtest1001

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag
testreduce1001

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
site.pp: retire parsoidtest and testreduceEffie Mouzelioperations/puppetproduction+5 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

jijiki created this task.Mar 27 2026, 11:57 AM
jijiki added a subtask: T421483: parsoidtest1001.eqiad.wmnet retirement plan.
jijiki added a project: Infrastructure-Foundations.Apr 17 2026, 4:48 PM
jijiki added a subscriber: Muehlenhoff.

@Muehlenhoff following up on our discussion, and unless there are no objections, we could repurpose this machine for the ganeti cluster on eqiad. It was purchased on 2024-06-26.

Jclark-ctr subscribed.May 28 2026, 3:26 PM
MoritzMuehlenhoff subscribed.Jun 9 2026, 2:16 PM
In T421484#11834275, @jijiki wrote:

@Muehlenhoff following up on our discussion, and unless there are no objections, we could repurpose this machine for the ganeti cluster on eqiad. It was purchased on 2024-06-26.

Sounds good to me!

MoritzMuehlenhoff added a subscriber: wiki_willy.Jun 9 2026, 2:21 PM
In T421484#11834275, @jijiki wrote:

@Muehlenhoff following up on our discussion, and unless there are no objections, we could repurpose this machine for the ganeti cluster on eqiad. It was purchased on 2024-06-26.

@wiki_willy Would that be okay? How do we track this?

Procedure-wise Effie could simply complete the decom steps until "remove all remaining puppet references and all host entries in the puppet repo" and we create a task with DC ops to re-label the server and re-provision under the new name?

Jclark-ctr added a comment.Jun 9 2026, 2:58 PM

@MoritzMuehlenhoff if current location location is ok. Could it just be renamed and reimaged by service owner?

wiki_willy added a comment.Jun 9 2026, 4:36 PM

@MoritzMuehlenhoff - sure, no problem. Repurposing it works for us as well. As long as we have all the updates in Phabricator, I think that's all we really need for tracking. Procedural wise (to John's point above), treating it as a rename/reimage would work for us too.

In T421484#12000011, @MoritzMuehlenhoff wrote:
In T421484#11834275, @jijiki wrote:

@Muehlenhoff following up on our discussion, and unless there are no objections, we could repurpose this machine for the ganeti cluster on eqiad. It was purchased on 2024-06-26.

@wiki_willy Would that be okay? How do we track this?

Procedure-wise Effie could simply complete the decom steps until "remove all remaining puppet references and all host entries in the puppet repo" and we create a task with DC ops to re-label the server and re-provision under the new name?

jijiki changed the task status from Open to Stalled.Jun 10 2026, 2:18 PM
jijiki added a subscriber: Jgiannelos.

Hey folks, the last bits are being wrapped up by @Jgiannelos. I am happy to decom the server when we are good to go, and hand it over to DC-ops. Sounds good?

Jgiannelos added a comment.Jun 10 2026, 2:29 PM

I think this train is the last week we run both envs for our weekly testing (both internal parsoidtest and mw-parsoid). I have one more thing to check for our regression testing and then we are good to decommission.

MoritzMuehlenhoff added a comment.Jul 3 2026, 7:27 AM
In T421484#12005000, @Jgiannelos wrote:

I think this train is the last week we run both envs for our weekly testing (both internal parsoidtest and mw-parsoid). I have one more thing to check for our regression testing and then we are good to decommission.

Is this completed?

MLechvien-WMF subscribed.Jul 3 2026, 1:54 PM

Porting here my comment from children task: given this service is part of Bullseye hosts, we'd need an ETA for full decommissioning. Thanks

bd808 mentioned this in T276650: Re-consider setting up a Kubernetes cluster on the Beta cluster.Tue, Jul 14, 10:44 PM
jijiki added a comment.Thu, Jul 16, 12:20 PM
In T421484#12085797, @MLechvien-WMF wrote:

Porting here my comment from children task: given this service is part of Bullseye hosts, we'd need an ETA for full decommissioning. Thanks

Due to T428909: Investigate X-Wikimedia-Debug routing for api.wikimedia.org and T431838: mw-parsoid endpoints are returning 404 for the rest.php endpoints used by rt-testing, folks had to use their old infra last week. We will try to wrap up this week.

MLechvien-WMF added a comment.Tue, Jul 21, 8:04 AM

As discussed with Effie yesterday, ETA is end of this week (Jul 24).

Jgiannelos added a comment.Tue, Jul 21, 9:02 AM

We just finished one more round of rt-testing after switching over to the old stack (T428909) and I've given a heads up to the team to backup any scripts on homedirs, so we should be good to decommision parsoidtestXXXX and testreduceXXXX

jijiki renamed this task from decommission parsoidtest1001.eqiad.wmnet to Reclaim parsoidtest1001.eqiad.wmnet.Tue, Jul 21, 10:32 AM
jijiki changed the task status from Stalled to In Progress.
jijiki updated the task description. (Show Details)
In T421484#12140195, @Jgiannelos wrote:

We just finished one more round of rt-testing after switching over to the old stack (T428909) and I've given a heads up to the team to backup any scripts on homedirs, so we should be good to decommision parsoidtestXXXX and testreduceXXXX

I will decomm both hosts next week, thank you!

jijiki renamed this task from Reclaim parsoidtest1001.eqiad.wmnet to Reclaim parsoidtest1001 and testreduce1001.Tue, Jul 21, 10:49 AM
jijiki closed subtask T421483: parsoidtest1001.eqiad.wmnet retirement plan as Resolved.Mon, Jul 27, 11:30 AM
jijiki created subtask T433239: Clean up puppet repo from parsoidtest* and testreduce*.Mon, Jul 27, 11:46 AM
gerritbot added a comment.Tue, Aug 4, 8:03 AM

Change #1320683 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] site.pp: retire parsoidtest and testreduce

https://gerrit.wikimedia.org/r/1320683

gerritbot added a project: Patch-For-Review.Tue, Aug 4, 8:03 AM
gerritbot added a comment.Tue, Aug 4, 9:54 AM

Change #1320683 merged by Effie Mouzeli:

[operations/puppet@production] site.pp: retire parsoidtest and testreduce

https://gerrit.wikimedia.org/r/1320683

Maintenance_bot removed a project: Patch-For-Review.Tue, Aug 4, 10:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits