CirrusSearch job are the biggest chunk of jobs still on the Redis queue. Here's the list:
- cirrusSearchCheckerJob - basically idempotent. It verifies data in elasticsearch matches mysql, creates new jobs if they don't match. Uses delayed execution. Tricky. It runs from a cron script scheduling bulk jobs with a set of pageIds and uses delay 1,2,3,4... to scatter the jobs in time. Really this is abusing the delayed job functionality, and what it really needs is a job scheduler that can insert jobs in the future.
- cirrusSearchDeleteArchive - idempotent - checks database to verify archive indexing is still appropriate when run.
- cirrusSearchDeletePages - idempotent
- cirrusSearchElasticaWrite - idempotent. Issued to retry failed write requests to elasticsearch. uses delayed execution
- cirrusSearchIncomingLinkCount - idempotent. expensive, high volume duplicates
- cirrusSearchLinksUpdate - idempotent, expensive
- cirrusSearchLinksUpdatePrioritized - idempotent, expensive,
- cirrusSearchMassIndex - idempotent, expensive, low volume
- cirrusSearchOtherIndex - cant use versioning, so out of order updates could be problematic
I think by now we're fairly confident that switching simpler jobs will be pretty straightforward, but we need to coordinate the with @EBernhardson so that we choose a fairly quiet period for Elasticsearch and also to help with verifying the correctness.
I think we have to use a similar approach that we've used for other jobs and for step 0 switch all the jobs for test wikis and mediawiki and then ask the discovery team to verify correctness on the Elasticsearch side.