Page MenuHomePhabricator
Create Task
Maniphest T277780

decom 8 codfw appservers purchased on 2016-06-02
Closed, ResolvedPublic
Actions

Description

After T277119 this is the next decom ticket for servers purchased on 2016-06-02 in T134272.

https://netbox.wikimedia.org/dcim/devices/?q=mw2&mac_address=&has_primary_ip=&local_context_data=&virtual_chassis_member=&console_ports=&console_server_ports=&power_ports=&power_outlets=&interfaces=&pass_through_ports=&cf_purchase_date=2016-06-02

  • mw2243 jobrunner (new jobrunner: mw2379)
  • mw2244 canary API replaced with mw2251
  • mw2245 canary API replaced with mw2252
  • mw2246 jobrunner (new jobrunner: mw2380)
  • mw2247 jobrunner (new jobrunner: mw2381)
  • mw2248 jobrunner (new jobrunner: mw2382)
  • mw2249 canary jobrunner (new canary: mw2278)
  • mw2250 canary jobrunner (new canary: mw2279)

Details

SubjectRepoBranchLines +/-
site/conftool-data: decom jobrunners mw2243,mw2246,mw2247,mw2248operations/puppetproduction+0 -9
site/conftool-data: turn 4 more new servers into jobrunnersoperations/puppetproduction+5 -5
site/conftool-data: turn new servers mw2377,mw2378 into jobrunnersoperations/puppetproduction+5 -2
site/conftool-data: decom mw2249,mw2250 jobrunner canariesoperations/puppetproduction+0 -7
site/conftool-data: turn mw2278,mw2279 into canary jobrunnersoperations/puppetproduction+11 -6
site/conftool-data: decom mw2244,mw2245, former canary serversoperations/puppetproduction+0 -8
site/conftool-data: turn mw2251,mw2252 into canariesoperations/puppetproduction+8 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

Dzahn created this task.Mar 18 2021, 5:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 18 2021, 5:08 PM
Dzahn renamed this task from decom codfw appservers purchased on 2016-06-02 to decom 7 codfw appservers purchased on 2016-06-02 .Mar 18 2021, 5:12 PM
Dzahn updated the task description. (Show Details)
Dzahn added a project: ops-codfw.
Maintenance_bot added a project: SRE.Mar 18 2021, 5:45 PM
gerritbot added a comment.Mar 18 2021, 10:00 PM

Change 673367 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/conftool-data: turn mw2251,mw2252 into canaries

https://gerrit.wikimedia.org/r/673367

gerritbot added a project: Patch-For-Review.Mar 18 2021, 10:00 PM

Change 673368 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/conftool-data: decom mw2244,mw2245, former canary servers

https://gerrit.wikimedia.org/r/673368

Dzahn renamed this task from decom 7 codfw appservers purchased on 2016-06-02 to decom 8 codfw appservers purchased on 2016-06-02 .Mar 18 2021, 10:52 PM
Dzahn updated the task description. (Show Details)

6 out of 8 are jobrunners. Maybe best to wait for T274171 to have started and turn some new servers in A3 into jobrunners, then remove these in A4 afterwards.

Dzahn changed the task status from Open to Stalled.Mar 18 2021, 10:54 PM
Dzahn triaged this task as High priority.
Dzahn added a subtask: T274171: (Need By: TBD) rack/setup/install (35) mw2377 and upwards.
gerritbot added a comment.Mar 19 2021, 6:59 PM

Change 673367 merged by Dzahn:
[operations/puppet@production] site/conftool-data: turn mw2251,mw2252 into canaries

https://gerrit.wikimedia.org/r/673367

ops-monitoring-bot added a comment.Mar 19 2021, 7:53 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2244.codfw.wmnet

  • mw2244.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Dzahn updated the task description. (Show Details)Mar 19 2021, 8:03 PM
ops-monitoring-bot added a comment.Mar 19 2021, 8:11 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2245.codfw.wmnet

  • mw2245.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Mar 19 2021, 9:12 PM

Change 673368 merged by Dzahn:
[operations/puppet@production] site/conftool-data: decom mw2244,mw2245, former canary servers

https://gerrit.wikimedia.org/r/673368

Maintenance_bot removed a project: Patch-For-Review.Mar 19 2021, 10:10 PM
gerritbot added a comment.Mar 20 2021, 12:36 AM

Change 673630 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/conftool-data: turn mw2278,mw2279 into canary jobrunners

https://gerrit.wikimedia.org/r/673630

gerritbot added a project: Patch-For-Review.Mar 20 2021, 12:36 AM
Dzahn added a subscriber: Papaul.Mar 22 2021, 7:56 PM

@Papaul fyi, this one is separate from T277119. I had to somehow separate them and instead by rack this is by purchase date. You will see though that this is just A4 and A3 is already covered.

gerritbot added a comment.Mar 22 2021, 7:58 PM

Change 673630 merged by Dzahn:
[operations/puppet@production] site/conftool-data: turn mw2278,mw2279 into canary jobrunners

https://gerrit.wikimedia.org/r/673630

Maintenance_bot removed a project: Patch-For-Review.Mar 22 2021, 8:10 PM
gerritbot added a comment.Mar 22 2021, 8:23 PM

Change 674137 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/conftool-data: decom mw2249,mw2250 jobrunner canaries

https://gerrit.wikimedia.org/r/674137

gerritbot added a project: Patch-For-Review.Mar 22 2021, 8:23 PM
Papaul added a comment.Mar 22 2021, 9:13 PM

@Dzahn thanks for the update. I am planning on racking mw2401 to mw2411 in A5 and not in A4 since A4 is a 10G rack , i will like to keep this rack only for 10G servers and not put in a 1G server in a 10G rack. Let me know if this is a problem.

Thanks

ops-monitoring-bot added a comment.Mar 22 2021, 11:01 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2249.codfw.wmnet

  • mw2249.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Dzahn added a comment.Mar 22 2021, 11:02 PM

@Papaul We can do that, it isn't a problem. We can use A3 and A5. Thank you

ops-monitoring-bot added a comment.Mar 22 2021, 11:34 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2250.codfw.wmnet

  • mw2250.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Mar 22 2021, 11:49 PM

Change 674137 merged by Dzahn:
[operations/puppet@production] site/conftool-data: decom mw2249,mw2250 jobrunner canaries

https://gerrit.wikimedia.org/r/674137

Dzahn updated the task description. (Show Details)Mar 22 2021, 11:51 PM
gerritbot added a comment.Mar 24 2021, 11:31 PM

Change 674727 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] site/conftool-data: turn new servers mw2377,mw2378 into jobrunners

https://gerrit.wikimedia.org/r/674727

gerritbot added a comment.Mar 24 2021, 11:55 PM

Change 674727 merged by Dzahn:
[operations/puppet@production] site/conftool-data: turn new servers mw2377,mw2378 into jobrunners

https://gerrit.wikimedia.org/r/674727

gerritbot added a comment.Mar 25 2021, 12:12 AM

Change 674736 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] site/conftool-data: decom jobrunners mw2243,mw2246,mw2247,mw2248

https://gerrit.wikimedia.org/r/674736

Dzahn mentioned this in T278396: bring 25 new mediawiki appserver in codfw into production, rack A3 (mw2377 - mw2401).Mar 31 2021, 8:19 PM
gerritbot added a comment.Apr 1 2021, 6:42 PM

Change 676437 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool-data: turn 4 more new servers into jobrunners

https://gerrit.wikimedia.org/r/676437

gerritbot added a comment.Apr 1 2021, 6:55 PM

Change 676437 merged by Dzahn:

[operations/puppet@production] site/conftool-data: turn 4 more new servers into jobrunners

https://gerrit.wikimedia.org/r/676437

Dzahn changed the task status from Stalled to Open.Apr 1 2021, 8:40 PM

4 new jobrunners have been created. This can now continue.

Stashbot added a comment.Apr 1 2021, 8:42 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-01T20:42:39Z] <mutante> mw2243, mw2246, mw2247, mw2248 - depooled - replaced by mw2379, mw2380, mw2381, mw2382 ( T277780)

ops-monitoring-bot added a comment.Apr 1 2021, 9:58 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2243.codfw.wmnet

  • mw2243.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Apr 1 2021, 10:12 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2246.codfw.wmnet

  • mw2246.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Apr 1 2021, 10:33 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2247.codfw.wmnet

  • mw2247.codfw.wmnet (FAIL)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Failed to power off, manual intervention required: Remote IPMI for mw2247.mgmt.codfw.wmnet failed (exit=1): b''
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Apr 1 2021, 10:53 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2248.codfw.wmnet

  • mw2248.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Apr 1 2021, 11:04 PM

Change 674736 merged by Dzahn:

[operations/puppet@production] site/conftool-data: decom jobrunners mw2243,mw2246,mw2247,mw2248

https://gerrit.wikimedia.org/r/674736

Dzahn closed this task as Resolved.Apr 1 2021, 11:28 PM
Dzahn reopened this task as Open.
Dzahn updated the task description. (Show Details)

@Papaul These were old servers in rack A4. They are also ready to go now.

Dzahn reassigned this task from Dzahn to Papaul.Apr 1 2021, 11:30 PM
Dzahn lowered the priority of this task from High to Medium.

@Papaul This is about these: https://netbox.wikimedia.org/dcim/devices/?q=mw2&mac_address=&has_primary_ip=&local_context_data=&virtual_chassis_member=&console_ports=&console_server_ports=&power_ports=&power_outlets=&interfaces=&pass_through_ports=&cf_purchase_date=2016-06-02

Legoktm subscribed.Apr 2 2021, 5:30 AM
In T277780#6966775, @ops-monitoring-bot wrote:

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2247.codfw.wmnet

  • mw2247.codfw.wmnet (FAIL)
    • Downtimed host on Icinga

Not sure why, but the icinga downtime on this actually failed. I just set it manually.

ops-monitoring-bot added a comment.Apr 2 2021, 6:28 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2247.codfw.wmnet

  • mw2247.codfw.wmnet (FAIL)
    • Downtimed host on Icinga
    • Host steps raised exception: Invalid management FQDN mw2247.mgmt.codfw.wmnet for mw2247.codfw.wmnet

ERROR: some step on some host failed, check the bolded items above

Dzahn added a subscriber: RLazarus.Apr 2 2021, 7:14 PM
In T277780#6967287, @Legoktm wrote:

Not sure why, but the icinga downtime on this actually failed. I just set it manually.

Thank you! The original reason was:

Failed to power off, manual intervention required: Remote IPMI for mw2247.mgmt.codfw.wmnet failed (exit=1): b''

and then it got into a zombie state where it and its mgmt was gone from DNS but the host was still in PuppeDB, which meant it was also still in Icinga.

This was surprising because the logs for the first cookbook run explictly include the "removed from PuppetDB" step and how it ran 'puppet node clean/deactivate'.

Repeating the decom cookbook did not fix that situation.

Then I manually ran puppet node deactivate mw2247.codfw.wmnet on puppetmaster1001 followed by running puppet agent on alert1001 and this solved it. The Icinga config snippets go removed and its gone from the web UI now.

Another side-effect of this was that running mcrouter_generate_certs failed when trying to add new hosts. Because it looks up hosts from PuppetDB and found mw2247 but then failed to lookup its host name.

Thanks to @RLazarus for adding a debug line that told us which host name lookup actually fails. Originally you could not tell which of the hosts is actually the issue.

Dzahn added a comment.Apr 5 2021, 6:12 PM

https://wikitech.wikimedia.org/w/index.php?title=Server_Lifecycle&type=revision&diff=1906888&oldid=1904157

Stashbot added a comment.Apr 6 2021, 11:43 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-06T11:43:51Z] <moritzm> removed mw2247 from debmonitor T277780

Papaul closed this task as Resolved.Apr 6 2021, 3:18 PM

complete

Papaul closed subtask T274171: (Need By: TBD) rack/setup/install (35) mw2377 and upwards as Resolved.Apr 7 2021, 2:45 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits