Page MenuHomePhabricator
Create Task
Maniphest T383025

decommission dbproxy1020.eqiad.wmnet
Closed, ResolvedPublicRequest
Actions

Description

This task will track the decommission-hardware of server dbproxy1020.eqiad.wmnet.

With the launch of updates to the decom cookbook, the majority of these steps can be handled by the service owners directly. The DC Ops team only gets involved once the system has been fully removed from service and powered down by the decommission cookbook.

dbproxy1020

Steps for service owner:

  • - all system services confirmed offline from production use
  • - Remove it from the report_users whitelist.
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - Remove grants from m3
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
report_users.sh: Remove 10.64.32.179Marosteguioperations/softwaremaster+0 -1
mariadb: Remove dbproxy1020Marosteguioperations/puppetproduction+0 -6
production-m3.sql.erb: Replace dbproxy1020 with dbproxy1028Marosteguioperations/puppetproduction+6 -6
dbproxy1020: Disable notificationsMarosteguioperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedjcrespoT376916 Upgrade backup hosts to Debian Bookworm 12.X
 ResolvedjcrespoT383902 Upgrade backup source or mediabackup database host os to Debian bookworm or decommission them
Unknown Object (Task)
 ResolvedJclark-ctrT365485 Q4:rack/setup/install dbproxy102[89]
Unknown Object (Task)
 ResolvedMarosteguiT373579 Productionize db22[21-40]
 ResolvedMarosteguiT374623 Decommission db21[21-40]
 ResolvedMarosteguiT368874 Productionize dbproxy102[89]
 ResolvedMarosteguiT367380 Productionize dbproxy200[5-8]
 ResolvedMarosteguiT381962 Decommission dbproxy200[1-4]
 ResolvedRequestJhancock.wmT382867 decommission dbproxy2001.codfw.wmnet
 ResolvedRequestJhancock.wmT382868 decommission dbproxy2002.codfw.wmnet
 ResolvedRequestJhancock.wmT382875 decommission dbproxy2003.codfw.wmnet
 ResolvedRequestJhancock.wmT382877 decommission dbproxy2004.codfw.wmnet
 ResolvedRequestJhancock.wmT383388 decommission db2123.codfw.wmnet
 ResolvedRequestJhancock.wmT383395 decommission db2126.codfw.wmnet
 ResolvedRequestJhancock.wmT383426 decommission db2135.codfw.wmnet
 ResolvedRequestJhancock.wmT383572 decommission db2128.codfw.wmnet
 ResolvedRequestJhancock.wmT383697 decommission db2132.codfw.wmnet
 ResolvedRequestJhancock.wmT383766 decommission db2130.codfw.wmnet
 ResolvedJhancock.wmT383971 decommission db2139
 ResolvedRequestJhancock.wmT384001 decommission db2131.codfw.wmnet
 ResolvedRequestJhancock.wmT384343 decommission db2133.codfw.wmnet
 ResolvedRequestJhancock.wmT384476 decommission db2134.codfw.wmnet
 ResolvedRequestJhancock.wmT384479 decommission db2136.codfw.wmnet
 ResolvedRequestJhancock.wmT384480 decommission db2140.codfw.wmnet
Restricted Task
 ResolvedRequestJclark-ctrT383025 decommission dbproxy1020.eqiad.wmnet

Event Timeline

Marostegui created this task.Jan 6 2025, 7:54 AM
Marostegui added a parent task: T368874: Productionize dbproxy102[89].

This is not ready yet.

Marostegui moved this task from Triage to Blocked on the DBA board.Jan 6 2025, 7:55 AM
Marostegui updated the task description. (Show Details)Jan 6 2025, 9:18 AM
Marostegui added a parent task: Restricted Task.Jan 6 2025, 9:28 AM
Marostegui added a comment.Jan 6 2025, 9:46 AM

This host is no longer the master, but let's wait 24h to make sure all the connections have moved smoothly to the other proxy.

gerritbot added a comment.Jan 6 2025, 10:21 AM

Change #1108405 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1020: Disable notifications

https://gerrit.wikimedia.org/r/1108405

gerritbot added a project: Patch-For-Review.Jan 6 2025, 10:21 AM

Change #1108405 merged by Marostegui:

[operations/puppet@production] dbproxy1020: Disable notifications

https://gerrit.wikimedia.org/r/1108405

Marostegui updated the task description. (Show Details)Jan 6 2025, 10:22 AM

Still some long running queries active from this proxy:

32458669	phuser	10.64.32.179:53674	phabricator_repository	Query	2223318	Statistics	SELECT `r`.*, IF(`ft_rank`.termCorpus LIKE '% boxedcommand %', 2, 0) + IF(`ft_rank`.normalCorpus LIK	0.000
72674090	phabricatorphd	10.64.32.179:36740	phabricator_config	Sleep	8		NULL	0.000
72674130	phabricatorphd	10.64.32.179:37084	phabricator_repository	Sleep	0		NULL	0.000
93863437	phuser	10.64.32.179:42424	phabricator_repository	Query	1358041	Statistics	SELECT `commit`.*, COUNT(DISTINCT(COALESCE(IF(FIELD(`edgelogic_ancestor_41`.dst, 'PHID-PROJ-zeogtxwy	0.000
130773906	phuser	10.64.32.179:59882	phabricator_phame	Query	505875	Statistics	SELECT `p`.*, IF(`ft_rank`.termCorpus LIKE '% convert %', 2, 0) + IF(`ft_rank`.normalCorpus LIKE '% 	0.000
136708215	phuser	10.64.32.179:40178	phabricator_phame	Query	354424	Statistics	SELECT `p`.*, IF(`ft_rank`.termCorpus LIKE '% convert %', 2, 0) + IF(`ft_rank`.normalCorpus LIKE '% 	0.000
139914555	phuser	10.64.32.179:47708	phabricator_dashboard	Query	292303	Statistics	SELECT `dashboard`.*, IF(`ft_rank`.termCorpus LIKE '% includes %', 2, 0) + IF(`ft_rank`.normalCorpus	0.000
139920682	phuser	10.64.32.179:54222	phabricator_dashboard	Query	292150	Statistics	SELECT `dashboard`.*, IF(`ft_rank`.termCorpus LIKE '% includes %', 2, 0) + IF(`ft_rank`.normalCorpus	0.000
148490258	phabricatorphd	10.64.32.179:50920	phabricator_daemon	Sleep	5		NULL	0.000
root@cumin1002:~# db-mysql db1213 -e "show processlist" | grep "10.64.32.179"
32458669	phuser	10.64.32.179:53674	phabricator_repository	Query	2223326	Statistics	SELECT `r`.*, IF(`ft_rank`.termCorpus LIKE '% boxedcommand %', 2, 0) + IF(`ft_rank`.normalCorpus LIK	0.000
72674090	phabricatorphd	10.64.32.179:36740	phabricator_config	Sleep	6		NULL	0.000
72674130	phabricatorphd	10.64.32.179:37084	phabricator_repository	Sleep	0		NULL	0.000
93863437	phuser	10.64.32.179:42424	phabricator_repository	Query	1358048	Statistics	SELECT `commit`.*, COUNT(DISTINCT(COALESCE(IF(FIELD(`edgelogic_ancestor_41`.dst, 'PHID-PROJ-zeogtxwy	0.000
130773906	phuser	10.64.32.179:59882	phabricator_phame	Query	505882	Statistics	SELECT `p`.*, IF(`ft_rank`.termCorpus LIKE '% convert %', 2, 0) + IF(`ft_rank`.normalCorpus LIKE '% 	0.000
136708215	phuser	10.64.32.179:40178	phabricator_phame	Query	354432	Statistics	SELECT `p`.*, IF(`ft_rank`.termCorpus LIKE '% convert %', 2, 0) + IF(`ft_rank`.normalCorpus LIKE '% 	0.000
139914555	phuser	10.64.32.179:47708	phabricator_dashboard	Query	292310	Statistics	SELECT `dashboard`.*, IF(`ft_rank`.termCorpus LIKE '% includes %', 2, 0) + IF(`ft_rank`.normalCorpus	0.000
139920682	phuser	10.64.32.179:54222	phabricator_dashboard	Query	292158	Statistics	SELECT `dashboard`.*, IF(`ft_rank`.termCorpus LIKE '% includes %', 2, 0) + IF(`ft_rank`.normalCorpus	0.000
148490258	phabricatorphd	10.64.32.179:50920	phabricator_daemon	Sleep	13		NULL	0.000
Marostegui updated the task description. (Show Details)Jan 6 2025, 10:24 AM
Maintenance_bot removed a project: Patch-For-Review.Jan 6 2025, 10:31 AM
Marostegui added a comment.Jan 7 2025, 6:34 AM
In T383025#10431892, @Marostegui wrote:

Still some long running queries active from this proxy:

32458669	phuser	10.64.32.179:53674	phabricator_repository	Query	2223318	Statistics	SELECT `r`.*, IF(`ft_rank`.termCorpus LIKE '% boxedcommand %', 2, 0) + IF(`ft_rank`.normalCorpus LIK	0.000
72674090	phabricatorphd	10.64.32.179:36740	phabricator_config	Sleep	8		NULL	0.000
72674130	phabricatorphd	10.64.32.179:37084	phabricator_repository	Sleep	0		NULL	0.000
93863437	phuser	10.64.32.179:42424	phabricator_repository	Query	1358041	Statistics	SELECT `commit`.*, COUNT(DISTINCT(COALESCE(IF(FIELD(`edgelogic_ancestor_41`.dst, 'PHID-PROJ-zeogtxwy	0.000
130773906	phuser	10.64.32.179:59882	phabricator_phame	Query	505875	Statistics	SELECT `p`.*, IF(`ft_rank`.termCorpus LIKE '% convert %', 2, 0) + IF(`ft_rank`.normalCorpus LIKE '% 	0.000
136708215	phuser	10.64.32.179:40178	phabricator_phame	Query	354424	Statistics	SELECT `p`.*, IF(`ft_rank`.termCorpus LIKE '% convert %', 2, 0) + IF(`ft_rank`.normalCorpus LIKE '% 	0.000
139914555	phuser	10.64.32.179:47708	phabricator_dashboard	Query	292303	Statistics	SELECT `dashboard`.*, IF(`ft_rank`.termCorpus LIKE '% includes %', 2, 0) + IF(`ft_rank`.normalCorpus	0.000
139920682	phuser	10.64.32.179:54222	phabricator_dashboard	Query	292150	Statistics	SELECT `dashboard`.*, IF(`ft_rank`.termCorpus LIKE '% includes %', 2, 0) + IF(`ft_rank`.normalCorpus	0.000
148490258	phabricatorphd	10.64.32.179:50920	phabricator_daemon	Sleep	5		NULL	0.000
root@cumin1002:~# db-mysql db1213 -e "show processlist" | grep "10.64.32.179"
32458669	phuser	10.64.32.179:53674	phabricator_repository	Query	2223326	Statistics	SELECT `r`.*, IF(`ft_rank`.termCorpus LIKE '% boxedcommand %', 2, 0) + IF(`ft_rank`.normalCorpus LIK	0.000
72674090	phabricatorphd	10.64.32.179:36740	phabricator_config	Sleep	6		NULL	0.000
72674130	phabricatorphd	10.64.32.179:37084	phabricator_repository	Sleep	0		NULL	0.000
93863437	phuser	10.64.32.179:42424	phabricator_repository	Query	1358048	Statistics	SELECT `commit`.*, COUNT(DISTINCT(COALESCE(IF(FIELD(`edgelogic_ancestor_41`.dst, 'PHID-PROJ-zeogtxwy	0.000
130773906	phuser	10.64.32.179:59882	phabricator_phame	Query	505882	Statistics	SELECT `p`.*, IF(`ft_rank`.termCorpus LIKE '% convert %', 2, 0) + IF(`ft_rank`.normalCorpus LIKE '% 	0.000
136708215	phuser	10.64.32.179:40178	phabricator_phame	Query	354432	Statistics	SELECT `p`.*, IF(`ft_rank`.termCorpus LIKE '% convert %', 2, 0) + IF(`ft_rank`.normalCorpus LIKE '% 	0.000
139914555	phuser	10.64.32.179:47708	phabricator_dashboard	Query	292310	Statistics	SELECT `dashboard`.*, IF(`ft_rank`.termCorpus LIKE '% includes %', 2, 0) + IF(`ft_rank`.normalCorpus	0.000
139920682	phuser	10.64.32.179:54222	phabricator_dashboard	Query	292158	Statistics	SELECT `dashboard`.*, IF(`ft_rank`.termCorpus LIKE '% includes %', 2, 0) + IF(`ft_rank`.normalCorpus	0.000
148490258	phabricatorphd	10.64.32.179:50920	phabricator_daemon	Sleep	13		NULL	0.000

I've drained these connections which are coming from T383068

gerritbot added a comment.Jan 7 2025, 7:57 AM

Change #1108706 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] production-m3.sql.erb: Replace dbproxy1020 with dbproxy1028

https://gerrit.wikimedia.org/r/1108706

gerritbot added a project: Patch-For-Review.Jan 7 2025, 7:57 AM
gerritbot added a comment.Jan 7 2025, 7:59 AM

Change #1108706 merged by Marostegui:

[operations/puppet@production] production-m3.sql.erb: Replace dbproxy1020 with dbproxy1028

https://gerrit.wikimedia.org/r/1108706

Maintenance_bot removed a project: Patch-For-Review.Jan 7 2025, 8:30 AM
gerritbot added a comment.Jan 7 2025, 11:52 AM

Change #1108730 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Remove dbproxy1020

https://gerrit.wikimedia.org/r/1108730

gerritbot added a project: Patch-For-Review.Jan 7 2025, 11:52 AM

Change #1108730 merged by Marostegui:

[operations/puppet@production] mariadb: Remove dbproxy1020

https://gerrit.wikimedia.org/r/1108730

Marostegui updated the task description. (Show Details)Jan 7 2025, 11:55 AM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Jan 7 2025, 11:56 AM

Change #1108731 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] report_users.sh: Remove 10.64.32.179

https://gerrit.wikimedia.org/r/1108731

gerritbot added a comment.Jan 7 2025, 11:57 AM

Change #1108731 merged by Marostegui:

[operations/software@master] report_users.sh: Remove 10.64.32.179

https://gerrit.wikimedia.org/r/1108731

Marostegui updated the task description. (Show Details)Jan 7 2025, 11:58 AM
ops-monitoring-bot added a comment.Jan 7 2025, 12:03 PM

cookbooks.sre.hosts.decommission executed by marostegui@cumin1002 for hosts: dbproxy1020.eqiad.wmnet

  • dbproxy1020.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Marostegui removed Marostegui as the assignee of this task.Jan 7 2025, 12:03 PM
Marostegui updated the task description. (Show Details)
Marostegui edited projects, added: ops-eqiad; removed: Patch-For-Review.
Restricted Application added a project: DC-Ops. · View Herald TranscriptJan 7 2025, 12:03 PM
Marostegui moved this task from Blocked to Done on the DBA board.Jan 7 2025, 12:03 PM

This is ready for DC-Ops

Marostegui moved this task from Backlog to pending onsite steps (eqiad) on the decommission-hardware board.Jan 7 2025, 12:04 PM
Maintenance_bot added a project: SRE.Jan 7 2025, 12:29 PM
Jclark-ctr moved this task from Backlog to Decommission on the ops-eqiad board.Jan 8 2025, 3:57 PM
Jclark-ctr closed this task as Resolved.Jan 8 2025, 4:36 PM
Jclark-ctr claimed this task.
Jclark-ctr updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits