Page MenuHomePhabricator

Eqiad Geosearch API queries return errors on Commons
Closed, ResolvedPublic3 Estimated Story PointsBUG REPORT

Description

List of steps to reproduce (step by step, including full links if applicable):

What happens?:
First one returns cirrussearch-parse-error and the second query returns cirrussearch-backend-error

What should have happened instead?:
The search completes without errors

Software version (if not a Wikimedia wiki), browser information, screenshots, other information, etc:
On the other hand, if you use X-Wikimedia-Debug: backend=mwdebug2001.codfw.wmnet in the request header to pass the request to the codfw cluster, the request returns without errors.
Perhaps this is related to the commonswiki file index outage (T295705, T295478), the CirrusSearch traffic was recently moved back to eqiad in T295705#7535301
Noting that this also breaks Special:Nearby on Commons.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
dcausse triaged this task as High priority.Dec 6 2021, 8:56 AM
dcausse added a project: CirrusSearch.
dcausse added subscribers: EBernhardson, dcausse.

shipped a bandaid (manually recreated the coord field on affected indices) to fix the errors. The situation is still not ideal as it probably means that these indices (the ones serving everything except files on commons) are not totally up to date and will have to be restored from the spare DC.

MPhamWMF set the point value for this task to 3.Dec 6 2021, 4:46 PM

Change 744070 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] Move cirrus traffic to codfw

https://gerrit.wikimedia.org/r/744070

Change 744070 merged by jenkins-bot:

[operations/mediawiki-config@master] Move cirrus traffic to codfw

https://gerrit.wikimedia.org/r/744070

Mentioned in SAL (#wikimedia-operations) [2021-12-06T17:27:02Z] <ebernhardson@deploy1002> Synchronized wmf-config/InitialiseSettings.php: T296897 Move cirrus traffic to codfw (duration: 00m 56s)

Change 744857 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] Revert \"Move cirrus traffic to codfw\"

https://gerrit.wikimedia.org/r/744857

Change 744857 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert \"Move cirrus traffic to codfw\"

https://gerrit.wikimedia.org/r/744857

Mentioned in SAL (#wikimedia-operations) [2021-12-08T00:09:33Z] <ebernhardson@deploy1002> Synchronized wmf-config/InitialiseSettings.php: T296897 Move cirrus traffic back to eqiad (duration: 01m 08s)

The other (archive, content and general) commonswiki indices have been snapshotted from codfw and then restored on eqiad, same as we previously did for the file index. Traffic is back on eqiad and the queries above now work as expected.