Page MenuHomePhabricator

Develop recovery/reindex procedures for new Search Update Pipeline
Closed, ResolvedPublic

Description

As mentioned in the parent ticket, the following operational procedures need to be created for the new search update pipeline:

  • Recovering from an Elasticsearch outage/interruption in updates. Current process documented here
  • In place reindex Documented here
  • Full reindex Documented here

Event Timeline

Per today's 1x1 with @RKemper , @EBernhardson and myself, we talked about what a new outage recovery process might look like.

Today's process uses a mediawiki extension script, ForceSearchIndex.php to read from the database and re-queue the writes.

We could potentially use another instance of the Cirrus Streaming Updater itself to do this, but it would be a one-off job...the type of operation we typically do with cookbooks. That type of job doesn't seem to fit the helm paradigm.

Per IRC conversation with @RLazarus , he's done a lot of work towards this exact use case (see T341553 ).

Gehel triaged this task as High priority.Feb 9 2024, 1:31 PM
Gehel moved this task from Incoming to 2024.02.12 - 2024.03.03 on the Data-Platform-SRE board.
Gehel claimed this task.