Page MenuHomePhabricator
Create Task
Maniphest T244586

Restbase routing down on beta, 2020-02-07
Closed, ResolvedPublic
Actions

Description

It looks like Restbase might be down on beta [1]

Expected behavior

  1. Go to https://en.wikipedia.beta.wmflabs.org/wiki/Talk:Cats
  2. Click a "Reply" link
  3. Draft a reply
  4. Click "Reply"

✅ 5. Comment is successfully published to https://en.wikipedia.beta.wmflabs.org/wiki/Talk:Cats

Actual behavior

  1. Go to https://en.wikipedia.beta.wmflabs.org/wiki/Talk:Cats
  2. Click a "Reply" link
  3. Draft a reply
  4. Click "Reply"

❌ 5. The following warning appears above the reply text input area Invalid response from server.

  1. https://en.wikipedia.beta.wmflabs.org/api/rest_v1/page/html/Talk%3AChat/412808?redirect=false&stash=true

Related Objects

Event Timeline

ppelberg created this task.Feb 7 2020, 5:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 7 2020, 5:20 PM
Jdforrester-WMF added a project: Beta-Cluster-Infrastructure.Feb 7 2020, 5:57 PM
• Mholloway subscribed.Feb 7 2020, 5:58 PM
• Mholloway added a comment.Feb 7 2020, 6:25 PM

Looks like a routing problem. The deployment-restbase0* instances are working fine, as is restbase-beta.wmflabs.org (try it), but the BC no longer appears to know how to handle /api/rest_v1/ paths.

DLynch merged a task: T244598: [Regression] Cannot reply to any comment on a Talk page, showing error "Invalid response from server.".Feb 7 2020, 7:50 PM
DLynch added subscribers: Ryasmeen, Esanders.
Ryasmeen triaged this task as Unbreak Now! priority.Feb 7 2020, 8:59 PM
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptFeb 7 2020, 8:59 PM
Ryasmeen added a comment.EditedFeb 7 2020, 9:00 PM

I am raising it as UBN, since all the testing activities are stalled because of this.

Jdforrester-WMF renamed this task from Restbase [might be] down on beta to Restbase down on beta, 2020-02-07.Feb 7 2020, 9:24 PM
Jdforrester-WMF renamed this task from Restbase down on beta, 2020-02-07 to Restbase routing down on beta, 2020-02-07.
• Mholloway added a comment.Feb 7 2020, 9:44 PM

To help isolate possible culprits, when was this last working?

• Pchelolo added a project: Traffic.Feb 7 2020, 9:58 PM
• Pchelolo subscribed.

RESTBase itself seem to be working correctly. Something is wrong with routing before RESTBase. If you look at https://en.wikipedia.beta.wmflabs.org/api/rest_v1/ the default error page my mediawiki is returned.

Restricted Application added a project: SRE. · View Herald TranscriptFeb 7 2020, 9:58 PM
Jdforrester-WMF subscribed.Feb 7 2020, 11:16 PM
In T244586#5860963, @Mholloway wrote:

To help isolate possible culprits, when was this last working?

Yesterday, but not sure when.

Nothing obvious recently merged in puppet.

Krenair subscribed.Feb 7 2020, 11:20 PM

Did fix puppet on cache-text05 yesterday, it did a lot of stuff to replace
some nginx/varnish stuff with ATS. May be related?

Jdforrester-WMF added a comment.Feb 7 2020, 11:21 PM
In T244586#5861250, @Krenair wrote:

Did fix puppet on cache-text05 yesterday, it did a lot of stuff to replace
some nginx/varnish stuff with ATS. May be related?

That sounds very plausible.

• Mholloway added a comment.Feb 7 2020, 11:22 PM

It looks like routing for /api/rest_v1/ in Beta is set up in the prefix puppet settings for deployment-cache-text (as seen here). Not sure what could be breaking it.

MarcoAurelio subscribed.EditedFeb 7 2020, 11:22 PM

Not sure T243226 might be related. Posting just in case.

Jdforrester-WMF added a comment.Feb 7 2020, 11:24 PM

deployment-restbase01.deployment-prep.eqiad.wmflabs reports The last Puppet run was at Mon Jan 20 10:54:08 UTC 2020 (26669 minutes ago). but as it's still working I guess it's fine?

• Pchelolo added a comment.Feb 7 2020, 11:25 PM
In T244586#5861273, @Jdforrester-WMF wrote:

deployment-restbase01.deployment-prep.eqiad.wmflabs reports The last Puppet run was at Mon Jan 20 10:54:08 UTC 2020 (26669 minutes ago). but as it's still working I guess it's fine?

Puppet run fails on RESTBase node with

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Resource type not found: Wmflib::Service at /etc/puppet/modules/wmflib/functions/service/fetch.pp:4:51 on node deployment-restbase01.deployment-prep.eqiad.wmflabs
Jdforrester-WMF added a comment.Feb 7 2020, 11:25 PM

Ah, right, that's the please-upgrade-puppet task that @MarcoAurelio linked above.

Jdforrester-WMF mentioned this in T244629: Popups on beta server display "There was an issue displaying this preview" message.Feb 8 2020, 2:27 AM
Jdforrester-WMF merged a task: T244629: Popups on beta server display "There was an issue displaying this preview" message.
Jdforrester-WMF added subscribers: • nray, Jdlrobson, Masumrezarock100.
Krenair claimed this task.Feb 8 2020, 11:44 AM

looks like profile::trafficserver::backend::mapping_rules in hieradata/labs.yaml only has support for mediawiki and upload - it's missing the restbase section that appears in hieradata/common/profile/trafficserver/backend.yaml

Krenair closed this task as Resolved.Feb 8 2020, 11:57 AM
Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptFeb 8 2020, 11:57 AM
Krenair added a comment.Feb 8 2020, 12:02 PM

(I added some corrected hieradata to cache-text05 in horizon)

Jdforrester-WMF added a comment.Feb 10 2020, 11:02 PM

Thank you!

ppelberg added a comment.Feb 11 2020, 5:15 AM
In T244586#5870304, @Jdforrester-WMF wrote:

Thank you!

+1 – thank you for your quick help on this, @Krenair.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL