Page MenuHomePhabricator
Create Task
Maniphest T395201

Quarry down - web service unreachable
Closed, ResolvedPublicBUG REPORT
Actions

Description

Steps to replicate the issue (include links if applicable):

  • I try to run my queries on Quarry and get an error message. It's been like this for several hours. I posted about it on En.wiki WP:VPT but no response there.

What happens?:
I get this message:

Error
This web service cannot be reached. Please contact a maintainer of this project.

Maintainers can find troubleshooting instructions from our documentation on Wikitech.

proxy-5.project-proxy.eqiad1.wikimedia.cloud

What should have happened instead?:
Queries should have run normally

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):
Chrome, PC, ordinary system

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedBUG REPORTtaaviT395201 Quarry down - web service unreachable
 ResolvedBUG REPORTtaaviT395226 quarry: Set a timeout on Redis operations
 ResolvedBUG REPORTtaaviT395237 quarry is leaking tmp files
 ResolvedtaaviT396785 Fix Quarry's Redis pod exiting causing frequent outages

Event Timeline

Liz created this task.May 25 2025, 6:36 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 25 2025, 6:36 AM
DreamRimmer added a project: Quarry.May 25 2025, 7:02 AM
Polygnotus subscribed.May 25 2025, 7:07 AM
RhinosF1 renamed this task from Is Quarry down? to Quarry down - web service unreachable.May 25 2025, 7:09 AM
Restricted Application added a subscriber: RhinosF1. · View Herald TranscriptMay 25 2025, 7:09 AM
RhinosF1 added a comment.May 25 2025, 7:11 AM

04:32:28 <wmcs-alerts> FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
04:32:39 <wmcs-alerts> FIRING: QuarryDown: Quarry application is unreachable  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown

So it's been down just over 3.5 hours

taavi claimed this task.May 25 2025, 7:56 AM
taavi subscribed.

One of the workers froze and took down the Redis instance with it, rebooting that worker solved it. Leaving this open to figure out on Monday why that happened.

Stashbot added a comment.May 25 2025, 7:56 AM

Mentioned in SAL (#wikimedia-cloud) [2025-05-25T07:56:28Z] <taavi> reboot quarry-127b-3lqizumia4xn-node-1 T395201

taavi closed this task as Resolved.May 26 2025, 9:47 AM

Filed subtasks for the issues I've found. Closing.

taavi closed subtask T395226: quarry: Set a timeout on Redis operations as Resolved.May 26 2025, 9:50 AM
taavi closed subtask T395237: quarry is leaking tmp files as Resolved.May 26 2025, 6:15 PM
taavi reopened subtask T395237: quarry is leaking tmp files as Open.Jun 2 2025, 10:29 AM
Polygnotus unsubscribed.Jun 2 2025, 1:04 PM
Liz added a comment.EditedJun 8 2025, 5:13 AM

Looks like this is resolved. Many thanks!

Liz reopened this task as Open.Jun 9 2025, 6:06 AM

It's happening again! The exact same thing is happening again.

SD0001 merged a task: T396295: [bug] Quarry down?.Jun 9 2025, 6:14 AM
SD0001 added subscribers: komla, dcaro.
Liz added a comment.EditedJun 10 2025, 5:38 AM

Okay, this glitch got fixed late last night but now it's down again! What is the problem here?

Edit: @taavi, can you look at this?

dcaro added a comment.Jun 10 2025, 7:13 AM
In T395201#10898038, @Liz wrote:

Okay, this glitch got fixed late last night but now it's down again! What is the problem here?

Edit: @taavi, can you look at this?

Just manually restarted redis (that was in a 'completed' state) and the stuck web pods (one was complaining of exceeding tmp usage, the onther complaining of redis being down). Things seem to be back up for now.

Liz added a comment.Jun 10 2025, 7:21 AM

Okay, it's working again. Let's hope this doesn't happen again tomorrow. Thanks!

taavi closed this task as Resolved.Jun 10 2025, 7:23 AM
taavi closed subtask T395237: quarry is leaking tmp files as Resolved.Jun 13 2025, 10:27 AM
taavi added a subtask: T396785: Fix Quarry's Redis pod exiting causing frequent outages.Jun 13 2025, 11:01 AM
taavi closed subtask T396785: Fix Quarry's Redis pod exiting causing frequent outages as Resolved.Jun 23 2025, 11:01 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits