List of steps to reproduce (step by step, including full links if applicable):
What happens?:
"An error has occurred while searching: We could not complete your search due to a temporary problem. Please try again later. "
What should have happened instead?:
Query returns files
Software version (if not a Wikimedia wiki), browser information, screenshots, other information, etc:
Change 737938 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):
[operations/mediawiki-config@master] Move CirrusSearch read traffic to codfw
Change 737938 merged by jenkins-bot:
[operations/mediawiki-config@master] Move CirrusSearch read traffic to codfw
Mentioned in SAL (#wikimedia-operations) [2021-11-10T15:52:47Z] <ebernhardson@deploy1002> Synchronized wmf-config/InitialiseSettings.php: T295480: Move all cirrussearch traffic to codfw (duration: 00m 56s)
Change 737945 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):
[operations/mediawiki-config@master] Actually move CirrusSearch read traffic to codfw
Change 737945 merged by jenkins-bot:
[operations/mediawiki-config@master] Actually move CirrusSearch read traffic to codfw
Mentioned in SAL (#wikimedia-operations) [2021-11-10T16:32:37Z] <ebernhardson@deploy1002> Synchronized wmf-config/InitialiseSettings.php: T295480: Move all cirrussearch traffic to codfw (duration: 00m 55s)
Incident report will be coming up, but the short answer is the commonswiki_file index on eqiad cluster went missing. Traffic has been moved to a cluster that still has the index, working up processes now to copy the good index between clusters and restore the eqiad cluster.