Page MenuHomePhabricator

Create maintenance script to clean up stale indexes
Open, HighPublic

Description

When index creation or reindex fails or is interrupted, there might be stale aliases left sitting around, which block further reindexing and in general waste space. We should have a maintenance script that allows to clean them up.

The procedure for removing the indexes is already documented at https://wikitech.wikimedia.org/wiki/Search#Removing_Duplicate_Indices but the script should:

  • Handle both clusters via --cluster option
  • Allow single-wiki cleanup via --wiki
  • Allow also all-wikis cleanup via suitable option

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 16 2017, 7:09 PM
debt triaged this task as Medium priority.Oct 19 2017, 5:06 PM
debt moved this task from needs triage to Up Next on the Discovery-Search board.
debt added a subscriber: dcausse.
Smalyshev lowered the priority of this task from Medium to Low.May 3 2018, 3:10 AM
dcausse raised the priority of this task from Low to Medium.Apr 6 2020, 7:58 AM
dcausse added a subscriber: Mstyles.

Raising prio as it happened on cloudelastic for wikidatawiki_content, it'd be great if we could detect these stale indices and clean them up more easily.

Ping @Mstyles as this is perhaps a task you might be interested in.

Mstyles claimed this task.Apr 6 2020, 5:01 PM
Mstyles removed Mstyles as the assignee of this task.Apr 13 2020, 10:37 PM

Discussed and this might be a task better served by SRE tooling and possibly for a future Search Platform SRE person

@Mstyles: Which project tag(s) should this task have, so it could be found by someone?

Restricted Application added a project: Discovery-Search. · View Herald TranscriptApr 16 2020, 12:45 PM
Gehel added a subscriber: Gehel.Aug 18 2020, 7:10 PM

A few things to take into account:

  • there are some indices not related to CirrusSearch (phabricator, translate extension, ...) that don't follow the same logic
  • during reindex, there are inflight indices that don't yet have an alias, but are not in error
Gehel raised the priority of this task from Medium to High.Tue, Sep 1, 7:08 PM