Page MenuHomePhabricator
Create Task
Maniphest T175830

decommission logstash100[1-3]
Closed, ResolvedPublic
Actions

Description

The new logstash servers are in place, we can start thinking about decommissioning the old ones. Note that we first need to ensure all log producers are migrated to logstash.svc.eqiad.wmnet.

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)

logstash1001=>asw-a-eqiad:ge-4/0/13
logstash1002=>asw-c-eqiad:ge-4/0/2
logstash1003=>asw2-d-eqiad:ge-3/0/16

  • - remove production dns entries
  • - puppet node clean, puppet node deactivate

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update racktables with result
  • - switch port configration removed from switch once system is unracked.
  • - mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
Removing mgmt dns logstash1001-1003operations/dnsmaster+1 -12
decom logstash100[1-3] prod dnsoperations/dnsmaster+3 -6
logstash100[1-3] decommissionoperations/puppetproduction+0 -26
logstash: decommission logstash100[1-3]operations/puppetproduction+2 -12
logstash: move eventlogging collection to logstash1007operations/puppetproduction+6 -8
logstash: remove references to old logstash servers for decommissioningoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Gehel created this task.Sep 13 2017, 2:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 13 2017, 2:35 PM
Gehel added a subtask: T175242: all log producers need to use the logstash LVS endpoint.Sep 13 2017, 2:35 PM
debt triaged this task as Medium priority.Sep 14 2017, 5:33 PM
gerritbot subscribed.Oct 9 2017, 10:14 AM

Change 383096 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] logstash: remove references to old logstash servers for decommissioning

https://gerrit.wikimedia.org/r/383096

gerritbot added a project: Patch-For-Review.Oct 9 2017, 10:14 AM
gerritbot added a comment.Oct 9 2017, 3:12 PM

Change 383096 merged by Muehlenhoff:
[operations/puppet@production] logstash: remove references to old logstash servers for decommissioning

https://gerrit.wikimedia.org/r/383096

Stashbot subscribed.Nov 2 2017, 10:53 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-02T10:53:31Z] <gehel> depooling logstash100[123] in preparation for decommission - T175830

gerritbot added a comment.Nov 2 2017, 11:16 AM

Change 388026 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: cleanup JVM options for blazegraph

https://gerrit.wikimedia.org/r/388026

Gehel moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.Dec 5 2017, 4:08 PM
Gehel added a project: hardware-requests.Dec 5 2017, 4:11 PM
Gehel updated the task description. (Show Details)Dec 5 2017, 4:14 PM
gerritbot added a comment.Dec 5 2017, 4:21 PM

Change 395564 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] logstash: move eventlogging collection to logstash1007

https://gerrit.wikimedia.org/r/395564

gerritbot added a comment.Dec 5 2017, 4:21 PM

Change 395565 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] logstash: decommission logstash100[1-3]

https://gerrit.wikimedia.org/r/395565

gerritbot added a comment.Dec 5 2017, 7:18 PM

Change 395564 merged by Gehel:
[operations/puppet@production] logstash: move eventlogging collection to logstash1007

https://gerrit.wikimedia.org/r/395564

Stashbot added a comment.Dec 5 2017, 7:20 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-05T19:20:18Z] <gehel> moving eventlogging collection by logstash from logstash1003 to logstash1007, no messages should be lost - T175830

Stashbot added a comment.Dec 6 2017, 9:34 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-06T09:34:41Z] <gehel> shuttting down logstash / elasticsearch on logstash100[123] in preparation for decommission -T175830

gerritbot added a comment.Dec 6 2017, 9:42 AM

Change 395565 merged by Gehel:
[operations/puppet@production] logstash: decommission logstash100[1-3]

https://gerrit.wikimedia.org/r/395565

Gehel reassigned this task from Gehel to RobH.Dec 6 2017, 9:55 AM
Gehel updated the task description. (Show Details)
Gehel moved this task from not in use - please delete to Needs Reporting on the Discovery-Search (Current work) board.
Gehel added a subscriber: RobH.

My steps for decommissioning are done (see checklist in the task description). Assigning to @RobH to continue.

Gehel removed projects: Patch-For-Review, Discovery-Search (Current work).Dec 6 2017, 5:23 PM
debt closed subtask T175242: all log producers need to use the logstash LVS endpoint as Resolved.Dec 8 2017, 8:45 PM
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Dec 11 2017, 5:11 PM
RobH updated the task description. (Show Details)Feb 9 2018, 8:08 PM
gerritbot added a comment.Feb 9 2018, 8:13 PM

Change 409433 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] logstash100[1-3] decommission

https://gerrit.wikimedia.org/r/409433

gerritbot added a project: Patch-For-Review.Feb 9 2018, 8:13 PM

Change 409433 merged by RobH:
[operations/puppet@production] logstash100[1-3] decommission

https://gerrit.wikimedia.org/r/409433

gerritbot added a comment.Feb 9 2018, 8:16 PM

Change 409434 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom logstash100[1-3] prod dns

https://gerrit.wikimedia.org/r/409434

gerritbot added a comment.Feb 9 2018, 8:17 PM

Change 409434 merged by RobH:
[operations/dns@master] decom logstash100[1-3] prod dns

https://gerrit.wikimedia.org/r/409434

RobH reassigned this task from RobH to • Cmjohnson.Feb 9 2018, 8:20 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH added a project: ops-eqiad.
RobH moved this task from Backlog to Decommission on the ops-eqiad board.

ready for on-site wipe and unracking steps

gerritbot added a comment.Mar 23 2018, 4:49 PM

Change 421570 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing mgmt dns logstash1001-1003

https://gerrit.wikimedia.org/r/421570

gerritbot added a project: Patch-For-Review.Mar 23 2018, 4:49 PM

Change 421570 merged by Cmjohnson:
[operations/dns@master] Removing mgmt dns logstash1001-1003

https://gerrit.wikimedia.org/r/421570

• Cmjohnson closed this task as Resolved.Mar 23 2018, 4:54 PM
• Cmjohnson updated the task description. (Show Details)
fgiunchedi added a project: observability.Aug 19 2019, 2:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits