Page MenuHomePhabricator
Create Task
Maniphest T172487

decom iridium
Closed, ResolvedPublic
Actions

Description

iridium was phabricator, has been replaced by phab1001.

after a grace period, start decom'ing with all the steps.. not yet though

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update racktables with result & added to gsheet tracking.
  • - switch port configration removed from switch once system is unracked.
  • - mgmt dns entries removed.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Removing mgmt dns for iridiumCmjohnsonoperations/dnsmaster+1 -4
decom iridiumRobHoperations/puppetproduction+1 -13
decom iridiumRobHoperations/dnsmaster+1 -4
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• mmodellT152129 reinstall iridium (phabricator) as phab1001 with jessie
 ResolvedRobHT172487 decom iridium

Event Timeline

Dzahn created this task.Aug 4 2017, 7:43 AM
Dzahn mentioned this in T163938: setup/install phab1001.eqiad.wmnet.
Dzahn changed the task status from Open to Stalled.Aug 4 2017, 7:49 AM
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)Aug 4 2017, 9:31 PM

please keep stalled for a few more days like this and don't shut down. we are a spare but don't want the disk wiped just yet, and phab-admins still have shell.. just in case

Paladox subscribed.Aug 4 2017, 9:35 PM
Paladox added a subscriber: • mmodell.Aug 4 2017, 10:00 PM
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Aug 11 2017, 8:42 PM
Dzahn changed the task status from Stalled to Open.EditedAug 17 2017, 7:30 PM
Dzahn added a subscriber: • demon.

@20after4 Last call before we are actually killing iridium and wiping the disk?

Dzahn added a subscriber: 20after4.Aug 17 2017, 7:34 PM
Dzahn removed Dzahn as the assignee of this task.Aug 21 2017, 9:16 PM

assigning it from me to pool. it can now be finalized and iridium can be shutdown and wiped. I can't do all the non-interruptable steps myself due to lack of switch access.

Dzahn added a comment.Aug 21 2017, 9:17 PM

status is: in puppet as role::spare, in DHCP and DNS. remnants in mysql grants (https://gerrit.wikimedia.org/r/369832) , all else is removed

• mmodell awarded a token.Aug 30 2017, 7:32 PM
Dzahn triaged this task as Medium priority.Sep 6 2017, 5:34 PM
MoritzMuehlenhoff subscribed.Jan 16 2018, 1:01 PM

I rebooted this spare host for completeless wrt Meltdown kernel update and while it's now running the fixed kernel, sshd came up running the /etc/ssh/sshd_config.phabricator instead of the regular /etc/ssh/sshd_config. iridium can still be reached via mgmt and is up for decom, so no point in debugging/fixing this IMO.

Dzahn added a comment.Jan 16 2018, 2:08 PM

No, no point i debugging indeed. Instead it would be really nice if it could be shutdown after running such a long time doing nothing.

Dzahn added a project: ops-eqiad.Jan 16 2018, 2:09 PM
• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Jan 26 2018, 7:10 PM
MoritzMuehlenhoff added a comment.Feb 2 2018, 1:09 PM
In T172487#3902655, @Dzahn wrote:

No, no point i debugging indeed. Instead it would be really nice if it could be shutdown after running such a long time doing nothing.

Agreed, it would be good to perform the "non-interruptible steps" soon. Currently this adds some noise to monitoring etc.

Dzahn added a comment.Feb 2 2018, 1:53 PM

Yes, i can't do them though because i don't have the access to disable switch ports.

RobH claimed this task.EditedFeb 2 2018, 5:19 PM

so this host ssh is down, so i cannot disable puppet on the host. I'll do the remainder of the uninterruptible steps now.

its port is asw-c-eqiad:ge-4/0/21

RobH updated the task description. (Show Details)Feb 2 2018, 5:23 PM
gerritbot subscribed.Feb 2 2018, 5:25 PM

Change 407683 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom iridium

https://gerrit.wikimedia.org/r/407683

gerritbot added a project: Patch-For-Review.Feb 2 2018, 5:25 PM

Change 407684 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom iridium

https://gerrit.wikimedia.org/r/407684

gerritbot added a comment.Feb 2 2018, 5:26 PM

Change 407683 merged by RobH:
[operations/dns@master] decom iridium

https://gerrit.wikimedia.org/r/407683

gerritbot added a comment.Feb 2 2018, 5:27 PM

Change 407684 merged by RobH:
[operations/puppet@production] decom iridium

https://gerrit.wikimedia.org/r/407684

RobH reassigned this task from RobH to • Cmjohnson.Feb 2 2018, 5:28 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH subscribed.

Ok, this is now ready for onsite wipe.

MoritzMuehlenhoff added a comment.Feb 2 2018, 5:48 PM

That host has a broken sshd config (coming from Phabricator), but it's possible to login via mgmt and the root password.

RobH added a comment.Feb 2 2018, 6:29 PM
In T172487#3941433, @MoritzMuehlenhoff wrote:

That host has a broken sshd config (coming from Phabricator), but it's possible to login via mgmt and the root password.

Done!

power off ready for wipe

RobH updated the task description. (Show Details)Feb 2 2018, 6:29 PM
Dzahn added a parent task: T152129: reinstall iridium (phabricator) as phab1001 with jessie.Feb 8 2018, 8:43 PM
RobH mentioned this in T186288: replace all Ubuntu (trusty) hosts in production with Debian.Feb 8 2018, 8:51 PM
gerritbot added a comment.Mar 23 2018, 4:57 PM

Change 421571 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing mgmt dns for iridium

https://gerrit.wikimedia.org/r/421571

gerritbot added a project: Patch-For-Review.Mar 23 2018, 4:57 PM
gerritbot added a comment.Mar 23 2018, 4:59 PM

Change 421571 abandoned by Cmjohnson:
Removing mgmt dns for iridium

Reason:
This server is meant for spares not full decom.

https://gerrit.wikimedia.org/r/421571

RobH removed a project: Patch-For-Review.Mar 23 2018, 6:13 PM
RobH updated the task description. (Show Details)

So this had a different port listed and disabled than the actual system port. I just fixed it, so it'll stop calling into puppet and monitoring. Task description updated, ready for decom/wipe.

MoritzMuehlenhoff reassigned this task from • Cmjohnson to RobH.Mar 26 2018, 10:01 AM
MoritzMuehlenhoff added a subscriber: • Cmjohnson.

The server is still visible in Cumin:

jmm@sarin:~$ sudo cumin irid*
1 hosts will be targeted:
iridium.eqiad.wmnet
DRY-RUN mode enabled, aborting
jmm@sarin:~$
• Cmjohnson moved this task from Decommission to High Priority Task on the ops-eqiad board.Mar 28 2018, 5:49 PM
• Cmjohnson updated the task description. (Show Details)Mar 28 2018, 6:03 PM
RobH closed this task as Resolved.Mar 29 2018, 3:31 PM
In T172487#4079943, @MoritzMuehlenhoff wrote:

The server is still visible in Cumin:

jmm@sarin:~$ sudo cumin irid*
1 hosts will be targeted:
iridium.eqiad.wmnet
DRY-RUN mode enabled, aborting
jmm@sarin:~$

When it powered on and called into puppet, it put itself back into everything. Sicne I cleared it on the 23rd, I don't see it in cumin:

robh@neodymium:~$ sudo cumin irid*
No hosts found that matches the query

• hashar mentioned this in T216714: gmail considers all Phabricator email to be spam due to missing SPF record.Feb 21 2019, 2:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits