Page MenuHomePhabricator
Create Task
Maniphest T300525

Beta cluster down: Error: 502, Next Hop Connection Failed
Closed, ResolvedPublicBUG REPORT
Actions

Description

List of steps to reproduce (step by step, including full links if applicable):

What happens?:
Commons:

Error

Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes.

See the error message at the bottom of this page for more information.

If you report this error to the Wikimedia System Administrators, please include the details below.

Request from - via deployment-cache-text06.deployment-prep.eqiad.wmflabs, ATS/8.0.8
Error: 502, Next Hop Connection Failed at 2022-01-31 14:58:54 GMT

Bert:

Error

Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes.

See the error message at the bottom of this page for more information.

If you report this error to the Wikimedia System Administrators, please include the details below.

Request from - via deployment-cache-text06.deployment-prep.eqiad.wmflabs, ATS/8.0.8
Error: 502, Next Hop Connection Failed at 2022-01-31 15:00:44 GMT

What should have happened instead?:
200

Related Objects
Search...

StatusSubtypeAssignedTask
 InvalidBUG REPORTNoneT312253 502 errors on beta cluster
 ResolvedBUG REPORTTheresNoTimeT300525 Beta cluster down: Error: 502, Next Hop Connection Failed

Event Timeline

AlexisJazz created this task.Jan 31 2022, 3:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 31 2022, 3:06 PM
AlexisJazz updated the task description. (Show Details)Jan 31 2022, 3:06 PM
RhinosF1 added a project: Release-Engineering-Team.Jan 31 2022, 3:17 PM
Restricted Application added a subscriber: RhinosF1. · View Herald TranscriptJan 31 2022, 3:17 PM
taavi removed a project: Traffic.Jan 31 2022, 3:18 PM
Vgutierrez subscribed.Jan 31 2022, 3:20 PM

hmm both URLs are currently working for me

• ema triaged this task as Medium priority.Jan 31 2022, 3:22 PM
• ema subscribed.

Apparently deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud is gone and has been replaced by deployment-mediawiki12.

I have updated the prefix puppet config for the prefix deployment-cache-text, using the new hostname in profile::trafficserver::backend::mapping_rules. Both links mentioned in this bug report look good to me now.

RhinosF1 added a comment.Jan 31 2022, 3:26 PM

Change was https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/35f8c97bd92ad240f2d4a52f6d37d916c911f977%5E%21/#F0 for the record

AlexisJazz added a comment.Jan 31 2022, 3:37 PM

Seems to be working again, thanks. :-)

Tgr mentioned this in T300591: Beta cluster MediaWiki code not updating.Jan 31 2022, 11:54 PM
AlexisJazz added a comment.EditedFeb 1 2022, 7:18 AM

Broken again.

Error

Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes.

See the error message at the bottom of this page for more information.

If you report this error to the Wikimedia System Administrators, please include the details below.

Request from - via deployment-cache-text06.deployment-prep.eqiad.wmflabs, ATS/8.0.8
Error: 502, Next Hop Connection Failed at 2022-02-01 07:15:32 GMT

Edit: works again

AlexisJazz added a comment.EditedFeb 1 2022, 2:50 PM

And down again.

Request from - via deployment-cache-text06.deployment-prep.eqiad.wmflabs, ATS/8.0.8
Error: 502, Next Hop Connection Failed at 2022-02-01 14:50:28 GMT
thcipriani removed a project: Release-Engineering-Team.Feb 9 2022, 5:51 PM
AlexisJazz added a comment.EditedFeb 28 2022, 3:58 AM

Edit: it's probably unrelated I'll just make a new task: T302699.

JMeybohm mentioned this in T302699: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022).Mar 2 2022, 11:21 AM
AlexisJazz mentioned this in T307311: API on Betacommons extremely slow (>20 seconds).May 1 2022, 8:47 PM
AlexisJazz mentioned this in T295427: Problem with delay caused by intake-analytics.wikimedia.org.May 13 2022, 11:25 AM
AlexisJazz added a parent task: T312253: 502 errors on beta cluster.Jul 6 2022, 5:35 PM
TheresNoTime closed this task as Resolved.Aug 16 2022, 5:13 PM
TheresNoTime claimed this task.
TheresNoTime subscribed.

This resolved

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL