Page MenuHomePhabricator

Disable ElasticSearch for empty Wikibases
Open, MediumPublic

Description

We currently see 149 Wikibases sitting empty. Overall, the share of empty Wikibases is increasing, from 15% in Q1 2023 to over 20% nowadays. This is a sizeable opportunity to reduce the load on the ElasticSearch cluster.
We continuously see challenges with scaling our ElasticSearch cluster, most recently: https://docs.google.com/document/d/1K5ekk52UL_WBYAjp1zga5biPqgDq_8DTNDL3J2vov2g/edit?usp=sharing

Empty Wikibase = no data
For reference: https://phabricator.wikimedia.org/T344689

AC:

  • Job that runs every 24h to determine whether a Wikibase is empty or not (be mindful that a main page is created by default now, so we don't have new completely 'empty' instances anymore)
  • If a Wikibase is empty, disable CirrusSearch, close the ElasticSearch indices
  • If a Wikibase no longer empty, open the indices in ElasticSearch, re-enable CirrusSearch, re-index the last 24 hours
  • To avoid blocking new Wikibases: Start running this job 30 days after creation