Page MenuHomePhabricator
Create Task
Maniphest T113542

Getting 404s for BetaCluster domains
Closed, ResolvedPublic
Actions

Description

The Mobile content service seems to disregard domains from the Beta Cluster:

$ curl -v localhost:6927/en.wikipedia.org/v1/page/mobile-html/Foobar
> GET /en.wikipedia.org/v1/page/mobile-html/Foobar HTTP/1.1
> User-Agent: curl/7.38.0
> Host: localhost:6927
> Accept: */*
> 
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: Accept, X-Requested-With, Content-Type
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< Content-Security-Policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
< X-Content-Security-Policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
< X-WebKit-CSP: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
< Content-Type: text/html; charset=utf-8
< Vary: Accept-Encoding
< Date: Wed, 23 Sep 2015 22:04:00 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked

$ curl -v localhost:6927/en.wikipedia.beta.wmflabs.org/v1/page/mobile-html/Foobar
> GET /en.wikipedia.beta.wmflabs.org/v1/page/mobile-html/Foobar HTTP/1.1
> User-Agent: curl/7.38.0
> Host: localhost:6927
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: Accept, X-Requested-With, Content-Type
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< Content-Security-Policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
< X-Content-Security-Policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
< X-WebKit-CSP: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
< Content-Type: application/json; charset=utf-8
< Content-Length: 222
< Vary: Accept-Encoding
< Date: Wed, 23 Sep 2015 22:04:24 GMT
< Connection: keep-alive

It needs to work well for *.wikipedia.beta.wmflabs.org domains as well, since the Beta Cluster is where we test the integration of various services.

Details

SubjectRepoBranchLines +/-
Prod config: fix the MW API URImediawiki/services/mobileappsmaster+1 -1
MobileApps: Fix the MW API URI, revert c848ca4fd and resurect restbase_urioperations/puppetproduction+5 -3
MobileApps: Specify the content type in the MW API calloperations/puppetproduction+1 -0
MobileApps: Put in place proper request templates for RB and MWoperations/puppetproduction+23 -4
Update mobileapps to 5be9a79mediawiki/services/mobileapps/deploymaster+120 K -68 K
Build request URIs with templating from swagger-routermediawiki/services/mobileappsmaster+134 -108
Customize query in gerrit

Related Objects

Event Timeline

• mobrovac created this task.Sep 23 2015, 10:09 PM
• mobrovac raised the priority of this task from to Needs Triage.
• mobrovac updated the task description. (Show Details)
• mobrovac added projects: Services, Mobile-Content-Service.
• mobrovac subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 23 2015, 10:09 PM
Krenair subscribed.Sep 23 2015, 10:13 PM
• bearND triaged this task as Medium priority.Nov 6 2015, 5:00 AM
• bearND set Security to None.
• bearND moved this task from Incoming to Backlog on the Mobile-Content-Service board.
• Mholloway subscribed.Feb 1 2016, 9:59 PM
• mobrovac raised the priority of this task from Medium to High.Feb 18 2016, 12:49 AM

Ping? We cannot test new features such as caching and purging because of this.

• Mholloway claimed this task.Feb 22 2016, 5:45 PM
• Mholloway moved this task from Backlog to Doing on the Mobile-Content-Service board.
• Mholloway added a comment.Feb 22 2016, 11:35 PM

I'm getting a 504 Gateway Timeout error when I curl the Beta Cluster instead of a 404 as shown above. But no headway yet.

Putting this up for grabs in case Bernd wants to look at it. I'm pretty focused on an app-side task now and picked it up prematurely.

• Mholloway removed • Mholloway as the assignee of this task.Feb 22 2016, 11:35 PM
• Mholloway moved this task from Doing to Backlog on the Mobile-Content-Service board.
• bearND subscribed.Feb 23 2016, 1:01 AM

I am also getting a 504. BTW, the URIs need to be updated. There is no mobile-html endpoint. It's called mobile-sections.

curl -sI localhost:6927/en.wikipedia.org/v1/page/mobile-sections/Foobar
HTTP/1.1 200 OK
access-control-allow-origin: *
access-control-allow-headers: accept, x-requested-with, content-type
access-control-expose-headers: etag
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
content-security-policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
x-content-security-policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
x-webkit-csp: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
etag: 702883545/4fbbb565-d9c3-11e5-8ddf-ab3933346382
Content-Type: application/json; charset=utf-8
Content-Length: 23141
Vary: Accept-Encoding
Date: Tue, 23 Feb 2016 00:20:58 GMT
Connection: keep-alive

curl -sI localhost:6927/en.wikipedia.beta.wmflabs.org/v1/page/mobile-sections/Foobar
HTTP/1.1 504 Gateway Timeout
access-control-allow-origin: *
access-control-allow-headers: accept, x-requested-with, content-type
access-control-expose-headers: etag
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
content-security-policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
x-content-security-policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
x-webkit-csp: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
Content-Type: application/json; charset=utf-8
Content-Length: 151
Vary: Accept-Encoding
Date: Tue, 23 Feb 2016 00:21:32 GMT
Connection: keep-alive

It may have something to do with Parsoid or SSL being misconfigured on beta labs:

This is what I get from the log output of the service when it tries to get the Parsoid content:

"name":"service-mobileapp-node","hostname":"***","pid":2857,"level":50,"message":"504: internal_http_error","status":504,"type":"internal_http_error","detail":"504: internal_http_error","levelPath":"error/504","request_id":"12bfe9e1-d9c5-11e5-9455-6898d536970e","msg":"504: internal_http_error","time":"2016-02-23T00:33:35.417Z","v":0}

The code is here: https://github.com/wikimedia/mediawiki-services-mobileapps/blob/master/lib/parsoid-access.js#L102-L103.
It calls preq.get on the Parsoid URI https://en.wikipedia.beta.wmflabs.org/api/rest_v1/page/html/Foobar.

curl https://en.wikipedia.beta.wmflabs.org/api/rest_v1/page/html/Foobar

complains about SSL config. I suspect that preq.get doesn't like it. Guidance from the services team is appreciated.

• bearND added a comment.Feb 23 2016, 7:21 PM

@mobrovac We need your help with that.

• Mholloway added a comment.Feb 24 2016, 4:21 PM

The 504 is happening because we're hard-coding our protocols in our request URIs to https, but non-production environments like the beta cluster don't have https correctly configured. Changing the following to http fixes the issue:

https://github.com/wikimedia/mediawiki-services-mobileapps/blob/master/lib/mwapi.js#L82
https://github.com/wikimedia/mediawiki-services-mobileapps/blob/master/config.dev.yaml#L62
https://github.com/wikimedia/mediawiki-services-mobileapps/blob/master/config.prod.yaml#L36

Probably better, though, to test for a production URI and use https only in that case. Patch coming shortly.

• Mholloway claimed this task.Feb 24 2016, 6:06 PM
• Mholloway moved this task from Backlog to Doing on the Mobile-Content-Service board.
gerritbot subscribed.Feb 24 2016, 7:25 PM

Change 273026 had a related patch set uploaded (by Mholloway):
Use HTTPS only for requests to production sites

https://gerrit.wikimedia.org/r/273026

gerritbot added a project: Patch-For-Review.Feb 24 2016, 7:25 PM
• Mholloway moved this task from Doing to Code Review on the Mobile-Content-Service board.Feb 24 2016, 7:25 PM
• mobrovac added a comment.Feb 25 2016, 4:01 PM

I think having proper URI template handling in the code would very easily solve this problem. Such functionality is already available in the [swagger-router package](https://github.com/wikimedia/swagger-router), in the Template object. This would allow us to have proper request objects compiled for each environment for calling both RESTBase and the MW API.

See:

gerritbot added a comment.Mar 2 2016, 8:31 AM

Change 273026 merged by Mobrovac:
Build request URIs with templating from swagger-router

https://gerrit.wikimedia.org/r/273026

gerritbot added a comment.Mar 2 2016, 8:54 AM

Change 274348 had a related patch set uploaded (by Mobrovac):
MobileApps: Put in place proper request templates for RB and MW

https://gerrit.wikimedia.org/r/274348

gerritbot added a comment.Mar 2 2016, 9:12 AM

Change 274352 had a related patch set uploaded (by Mobrovac):
Update mobileapps to 5be9a79

https://gerrit.wikimedia.org/r/274352

gerritbot added a comment.Mar 2 2016, 11:34 AM

Change 274352 merged by Mobrovac:
Update mobileapps to 5be9a79

https://gerrit.wikimedia.org/r/274352

gerritbot added a comment.Mar 2 2016, 11:50 AM

Change 274348 merged by Giuseppe Lavagetto:
MobileApps: Put in place proper request templates for RB and MW

https://gerrit.wikimedia.org/r/274348

Stashbot subscribed.Mar 2 2016, 12:00 PM

Mentioned in SAL [2016-03-02T12:00:25Z] <mobrovac> mobileapps stopping the service on scb1001 for debug purposes, T113542

gerritbot added a comment.Mar 2 2016, 12:15 PM

Change 274373 had a related patch set uploaded (by Mobrovac):
MobileApps: Specify the content type in the MW API call

https://gerrit.wikimedia.org/r/274373

gerritbot added a comment.Mar 2 2016, 12:16 PM

Change 274373 merged by Giuseppe Lavagetto:
MobileApps: Specify the content type in the MW API call

https://gerrit.wikimedia.org/r/274373

Stashbot added a comment.Mar 2 2016, 12:25 PM

Mentioned in SAL [2016-03-02T12:25:24Z] <mobrovac> mobileapps rolling back to 68e38ec7, problems found in the latest deploy for T113542

Stashbot added a comment.Mar 2 2016, 12:32 PM

Mentioned in SAL [2016-03-02T12:32:42Z] <mobrovac> mobileapps stopping (again) the service on scb1001 for debugging, T113542

gerritbot added a comment.Mar 2 2016, 12:59 PM

Change 274381 had a related patch set uploaded (by Mobrovac):
MobileApps: Fix the MW API URI, revert c848ca4fd and resurect restbase_uri

https://gerrit.wikimedia.org/r/274381

gerritbot added a comment.Mar 2 2016, 1:06 PM

Change 274381 merged by Giuseppe Lavagetto:
MobileApps: Fix the MW API URI, revert c848ca4fd and resurect restbase_uri

https://gerrit.wikimedia.org/r/274381

Stashbot added a comment.Mar 2 2016, 1:10 PM

Mentioned in SAL [2016-03-02T13:10:02Z] <mobrovac> mobileapps re-deploying d384f1ba for T113542

gerritbot added a comment.Mar 2 2016, 1:16 PM

Change 274384 had a related patch set uploaded (by Mobrovac):
Prod config: fix the MW API URI

https://gerrit.wikimedia.org/r/274384

gerritbot added a comment.Mar 2 2016, 1:21 PM

Change 274384 merged by Mobrovac:
Prod config: fix the MW API URI

https://gerrit.wikimedia.org/r/274384

• mobrovac closed this task as Resolved.Mar 2 2016, 1:23 PM
• mobrovac removed a project: Patch-For-Review.
• mobrovac removed a subscriber: gerritbot.

This is now done. Thank you @Mholloway for the code patch!

• bearND moved this task from Code Review to To Deploy on the Mobile-Content-Service board.Mar 18 2016, 8:51 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL