Page MenuHomePhabricator

High error rate in WDQS since 2022-12-02 15:00 UTC
Closed, ResolvedPublic

Event Timeline

Bugreporter triaged this task as Unbreak Now! priority.Dec 2 2022, 4:45 PM

Just a short comment to acknowledge the issue. There are a number of people working on this at the moment. We'll post an update and an incident report once we know more.

Email sent to Wikidata Users list:

We are aware of an ongoing issue with WDQS that begun around 1500 UTC today. To stabilize the service, we have introduced drastic measures including banning cloud provider IPs and certain user agents. Unfortunately, this will likely affect legitimate users of the service. We are currently working on less disruptive measures and will update when we have more information to share. Thanks for your patience, and we sincerely apologize for the inconvenience.

Follow-up email sent to Wikidata Users list:

At this point, we believe we have mitigated the stability issues, and legitimate users should be able to access the WDQS service once again. If you are still having issues, please respond here or on the Phabricator ticket linked below.

Thanks once again for your patience and have a great weekend!

Adding a reminder to myself to capture dashboard links, log-diving commands and other troubleshooting info from today's IRC log and add to the WDQS runbook .

Gehel lowered the priority of this task from Unbreak Now! to High.Dec 5 2022, 10:48 AM
Gehel moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.

Changing priority from UBN to High as the outage is over.

Aklapper renamed this task from High error rate in WDQS since 15:00 UTC to High error rate in WDQS since 2022-12-02 15:00 UTC.Dec 5 2022, 12:23 PM
bking moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.

Closing in favor of T325021 , which contains the post-outage action items.