Page MenuHomePhabricator
Create Task
Maniphest T191108

https://stream.wikimedia.org/v2/stream/recentchange 502 Bad Gateway
Closed, ResolvedPublic
Actions

Description

$ curl https://stream.wikimedia.org/v2/stream/recentchange
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.13.6</center>
</body>
</html>

Details

SubjectRepoBranchLines +/-
Config: Increase memory and heartbeat limitsmediawiki/services/eventstreams/deploymaster+2 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

Xqt created this task.Mar 30 2018, 9:54 AM
Restricted Application added subscribers: pywikibot-bugs-list, Kizule, Aklapper. · View Herald TranscriptMar 30 2018, 9:54 AM
Xqt triaged this task as High priority.Mar 30 2018, 9:54 AM
Xqt added a subtask: T179986: Investigate why current es2r daemon is randomly hanging.
Xqt updated the task description. (Show Details)Mar 30 2018, 10:02 AM
Xqt raised the priority of this task from High to Needs Triage.Mar 30 2018, 10:08 AM
Xqt added projects: Event-Platform, EventStreams.

Looks like an upstream issue

Restricted Application added a project: Analytics. · View Herald TranscriptMar 30 2018, 10:08 AM
Xqt closed this task as Resolved.Mar 30 2018, 10:26 AM
Xqt claimed this task.

Working again. Thanks.

Xqt reopened this task as Open.Mar 30 2018, 1:33 PM

Again the same problem as described above.

Xqt removed Xqt as the assignee of this task.Mar 30 2018, 1:36 PM
zhuyifei1999 renamed this task from EventStreams fails due to Bad Gateway for url: https://stream.wikimedia.org/v2/stream/recentchange to https://stream.wikimedia.org/v2/stream/recentchange 502 Bad Gateway.Mar 31 2018, 4:56 PM
zhuyifei1999 triaged this task as High priority.
zhuyifei1999 updated the task description. (Show Details)
zhuyifei1999 subscribed.

Raising priority to High. A production service is broken here.

• mobrovac subscribed.Apr 1 2018, 1:37 PM

The service is back up and working. From the logs, it seems that some clients fell behind, causing the service the accumulate messages and exceed the set memory limit. As a first-course remedy, we can increase the limits a bit as there is headroom for that on the hosts.

gerritbot subscribed.Apr 1 2018, 1:39 PM

Change 423341 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/eventstreams/deploy@master] Config: Increase memory and heartbeat limits

https://gerrit.wikimedia.org/r/423341

gerritbot added a project: Patch-For-Review.Apr 1 2018, 1:39 PM
Milimetric moved this task from Incoming to Radar on the Analytics board.Apr 2 2018, 3:31 PM
Dalba moved this task from Backlog to Upstream issues on the Pywikibot board.Apr 4 2018, 9:07 AM
gerritbot added a comment.Jul 18 2018, 2:47 PM

Change 423341 abandoned by Mobrovac:
Config: Increase memory and heartbeat limits

Reason:
Not viable because of T199813

https://gerrit.wikimedia.org/r/423341

• Pchelolo closed this task as Resolved.Sep 7 2018, 10:14 PM
• Pchelolo claimed this task.
• Pchelolo removed projects: Patch-For-Review, Upstream.
• Pchelolo subscribed.

I believe after T199813 was fixed this should not happen again.

Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:44 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL