We have a documented procedure to reindex after an outage, which is quite simple as it is. Still, it would be nice to create a cookbook for it, so that it is available centrally with the other operational procedures
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T203943 Spicerack cookbooks TODO list | |||
| Resolved | Gehel | T251149 [epic] Ryan's onboarding to the Search Platform team | |||
| Declined | bking | T219507 Create cookbook to reindex into elasticsearch / cirrus |
Event Timeline
Comment Actions
Some potential nice-to-have features, from recent discussions about reindexing problems:
- Catching reindexing failures in general and issuing an alert/warning. Right now the best/only/horrible way is something like grep -ilP "Reindex task was not successful|fail|error|warn" *log.
- Check for multiple indexes: Alert, possibly before doing anything else, if there are multiple indexes for a given wiki. Alert if, after reindexing, there are multiple indexes.
- Alert If the index didn't change from before reindexing to after (the index name is formatted as <wiki>_<timestamp> so it should change).
Comment Actions
I've moved this ticket back to "needs triage" so we can discuss it again in light of the recent problems with T274200, and decide if we should make it more of a priority, and maybe consider a different approach than cookbooks (so non-SREs can reindex).
If this ticket should only be for outage recovery, I can create a new ticket to cover the more general reindexing case.
Comment Actions
Moving this out of the current work, but this is still a priority for us. Will revisit next quarter.
Comment Actions
Per IRC conversation with @dcausse , we now have an alternate way of reindexing that does not involve cookbooks . As such, we can close out this ticket.