Page MenuHomePhabricator

Decommission lsearchd
Closed, ResolvedPublic

Description

Elasticsearch is running happily on all the wikis. It's time to say goodbye to lsearchd.

Use "kill-lsearchd" as your topic in Gerrit.

Event Timeline

demon created this task.Dec 19 2014, 5:34 PM
demon raised the priority of this task from to Needs Triage.
demon updated the task description. (Show Details)
demon added a project: acl*sre-team.
demon changed Security from none to None.
demon updated the task description. (Show Details)
demon added subscribers: demon, BBlack, Manybubbles and 2 others.
Joe added a subscriber: Joe.Dec 22 2014, 8:26 AM

Can we also re-commission servers? we do have a few applications where this could be useful.

fgiunchedi triaged this task as Medium priority.Dec 22 2014, 9:46 AM

next steps:

  • drain all pools from traffic in MW
  • drain backends from LVS search IPs
  • repurpose search machines in the spare pool or to other uses
  • remove puppet search-related manifests, including:
    • search.pp
    • role::lucene
    • lvs config
    • udp2log config
Joe added a comment.Dec 22 2014, 2:43 PM

Well, the last part is probably not needed (some pools are still going to be there)

I was under the impression that we are deprecating all pools and thus all of lsearchd?

demon added a comment.Dec 30 2014, 4:16 AM

Eventually. Just no ETA on pool 1 and 3 yet.

ok, I'll take care of this

what's a likely eta for decommissioning all the pools? if it isn't that far out I'd prefer to do this piecemeal

demon added a comment.Jan 6 2015, 7:43 PM

Pools drained on the application side. Can start on the remaining steps now.

faidon added a subscriber: faidon.Jan 6 2015, 7:44 PM
MaxSem added a subscriber: MaxSem.Jan 6 2015, 9:52 PM

all backend hosts commented in pybal config for search_pool1-5 and search_prefix and disabled notification in icinga

Change 183462 had a related patch set uploaded (by Filippo Giunchedi):
lsearchd: remove lvs configuration

https://gerrit.wikimedia.org/r/183462

Patch-For-Review

Change 183469 had a related patch set uploaded (by Filippo Giunchedi):
lsearchd: remove udp2log configuration

https://gerrit.wikimedia.org/r/183469

Patch-For-Review

Change 183462 merged by Filippo Giunchedi:
lsearchd: remove lvs configuration

https://gerrit.wikimedia.org/r/183462

Change 184611 had a related patch set uploaded (by Filippo Giunchedi):
lsearchd: remove all lvs references

https://gerrit.wikimedia.org/r/184611

Patch-For-Review

Change 184611 merged by Filippo Giunchedi:
lsearchd: remove all lvs references

https://gerrit.wikimedia.org/r/184611

Change 183469 merged by Filippo Giunchedi:
lsearchd: remove udp2log configuration

https://gerrit.wikimedia.org/r/183469

Change 184624 had a related patch set uploaded (by Filippo Giunchedi):
remove service endpoints for lsearchd

https://gerrit.wikimedia.org/r/184624

Patch-For-Review

see https://gerrit.wikimedia.org/r/#/c/184620/ for puppet decom and https://gerrit.wikimedia.org/r/#/c/184624/ for dns

pending machine deprovisioning, @mark did we have an use for those already or back to the spare pool?

mark added a comment.Jan 13 2015, 1:02 PM

see https://gerrit.wikimedia.org/r/#/c/184620/ for puppet decom and https://gerrit.wikimedia.org/r/#/c/184624/ for dns

pending machine deprovisioning, @mark did we have an use for those already or back to the spare pool?

Depends on the age of the machines as well. IIRC some were older/out of warranty, some fairly recent. Has anyone looked at that yet?

mark renamed this task from Decomission lsearchd to Decommission lsearchd.Jan 13 2015, 1:02 PM

according to the dell website warranty expired on 2014-02-02 for all machines, I couldn't find any machine with a longer warranty

curiously enough I couldn't find the asset tag for search1014 via dmidecode, racktables had it though

mark added a comment.Jan 13 2015, 3:56 PM

Then we shouldn't repurpose it for any critical roles. If anyone has any purpose for them where machine failures/lack of support isn't a problem we can keep them, but keep in mind that these won't live much longer.

Change 184624 merged by Filippo Giunchedi:
remove service endpoints for lsearchd

https://gerrit.wikimedia.org/r/184624

understood re: machines, I can't think of a use case right now where we'd be happy with out of warranty machines. specs are 48G ram, 2x12 cores and 300GB SSD, I'll let @RobH comment on putting them back in the spare pool/shutdown/reuse etc

fgiunchedi mentioned this in Unknown Object (Diffusion Commit).Jan 14 2015, 4:52 PM

moving to ops-eqiad

fgiunchedi reassigned this task from fgiunchedi to RobH.Feb 2 2015, 10:47 AM
RobH added a comment.Feb 2 2015, 10:47 PM

I'll keep this ticket assgined to me, but put in sub-tasks for the clearing of data and such.

I'm not sure if we want to keep these at all, but they certainly need the disks wiped.

I think we want to keep them as spares unless we have a good reason not to (e.g. lack of rack space, cleanup, etc) since they are fairly powerful but out of warranty

Ottomata removed a subscriber: Ottomata.Mar 10 2015, 3:27 PM
RobH closed this task as Resolved.Mar 16 2015, 9:02 PM
RobH closed subtask T86149: reclaim lsearchd hosts as Resolved.

with adding the old systems back to spares, I think that ends the decom steps for lsearch. Resolving.