Page MenuHomePhabricator
Create Task
Maniphest T297221

Search backend error during sending 1 documents to the commonswiki_content_1617495209 index(s): primary shard is not active
Closed, ResolvedPublic
Actions

Description

More than 25,000 instances of an error message like this have occurred over the last hour.
https://logstash.wikimedia.org/app/dashboards#/view/mediawiki-errors
Note that they're logged under the jsonTruncated channel. I filed T297219 for that issue.

Related Objects

Event Timeline

• dancy created this task.Dec 7 2021, 5:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 7 2021, 5:22 PM
brennen added a project: Discovery-ARCHIVED.Dec 7 2021, 5:29 PM
brennen edited projects, added Discovery-Search; removed Discovery-ARCHIVED.
• dancy added a parent task: T293953: 1.38.0-wmf.12 deployment blockers.Dec 7 2021, 6:09 PM

I don't want to roll the train forward under these conditions so I added this task as a train blocker.

EBernhardson subscribed.EditedDec 7 2021, 6:28 PM

These errors are unrelated, it is an expected outage (user traffic is on another cluster). The errors come out of a snapshot restore process that's currently running re T296897 and T295705. Ideally these would be a bit quieter, but we didn't have a process already set for snapshot/restore and just kinda making it work.

The process itself is on the last index and should finish in the next hour or two.

EBernhardson closed this task as Resolved.Dec 7 2021, 6:47 PM
EBernhardson claimed this task.

These are no longer being emitted, the cluster now has primaries available for the given indices.

• dancy added a comment.Dec 7 2021, 7:27 PM
In T297221#7553892, @EBernhardson wrote:

These are no longer being emitted, the cluster now has primaries available for the given indices.

Fantastic. Thanks for looking into it.

• dancy removed a parent task: T293953: 1.38.0-wmf.12 deployment blockers.Dec 7 2021, 7:27 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits