Page MenuHomePhabricator

Clean up failed reindexing indexes
Closed, ResolvedPublic

Description

User story: As a search engineer I would like there to be only one copy of a given index in production so I know which one is the live index, and to know that reindexing was succcessful.

From mwmaint1002 you can run this janky command line code to find duplicate indexes:

for cluster in search.svc.eqiad.wmnet search.svc.codfw.wmnet cloudelastic.wikimedia.org; do for port in 9243 9443 9643; do echo "$cluster:$port"; curl -s https://$cluster:$port/_cat/indices | perl -pe 's/^\S+\s+\S+\s+(\S+)\s+.*/$1/; s/_(\d+|first)//;' | sort | uniq -c | sort -n | grep -vP "^\s+1\s"; done; done

There are 46 indexes with duplicates across 8 of the nine cluster/port combos, including glent on eqiad and codfw. Most have 2, but many have 3, especially on cloudelastic:9243. It was unclear in the Wednesday Meeting today whether there should be multiple glent indexes or not, so those may be okay. One duplicate at a time during reindexing is probably valid, too.

When the current round of reindexing is done, we can clean up duplicates.

Acceptance Criteria:

  • There are no unexpected (glent?) duplicate indexes in the eqiad, codfw, or cloudelastic clusters.

Bonus result:

  • A better way to find duplicate indexes than the command line abomination above.

Event Timeline

Gehel added a subscriber: Gehel.

Note that T279009 also has some indices to cleanup, it might make sense to address both at the same time.

Change 682189 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] Reconcile configured indices with live state

https://gerrit.wikimedia.org/r/682189

Change 682189 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Reconcile configured indices with live state

https://gerrit.wikimedia.org/r/682189

Gehel claimed this task.