Page MenuHomePhabricator

Clean up failed reindexing indexes
Closed, ResolvedPublic

Description

User story: As a search engineer I would like there to be only one copy of a given index in production so I know which one is the live index, and to know that reindexing was succcessful.

From mwmaint1002 you can run this janky command line code to find duplicate indexes:

for cluster in search.svc.eqiad.wmnet search.svc.codfw.wmnet cloudelastic.wikimedia.org; do for port in 9243 9443 9643; do echo "$cluster:$port"; curl -s https://$cluster:$port/_cat/indices | perl -pe 's/^\S+\s+\S+\s+(\S+)\s+.*/$1/; s/_(\d+|first)//;' | sort | uniq -c | sort -n | grep -vP "^\s+1\s"; done; done

There are 46 indexes with duplicates across 8 of the nine cluster/port combos, including glent on eqiad and codfw. Most have 2, but many have 3, especially on cloudelastic:9243. It was unclear in the Wednesday Meeting today whether there should be multiple glent indexes or not, so those may be okay. One duplicate at a time during reindexing is probably valid, too.

When the current round of reindexing is done, we can clean up duplicates.

Acceptance Criteria:

  • There are no unexpected (glent?) duplicate indexes in the eqiad, codfw, or cloudelastic clusters.

Bonus result:

  • A better way to find duplicate indexes than the command line abomination above.

Event Timeline

Gehel subscribed.

Note that T279009 also has some indices to cleanup, it might make sense to address both at the same time.

Change 682189 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] Reconcile configured indices with live state

https://gerrit.wikimedia.org/r/682189

Change 682189 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Reconcile configured indices with live state

https://gerrit.wikimedia.org/r/682189

Gehel claimed this task.