Page MenuHomePhabricator

Getting 404s for BetaCluster domains
Closed, ResolvedPublic

Description

The Mobile content service seems to disregard domains from the Beta Cluster:

$ curl -v localhost:6927/en.wikipedia.org/v1/page/mobile-html/Foobar
> GET /en.wikipedia.org/v1/page/mobile-html/Foobar HTTP/1.1
> User-Agent: curl/7.38.0
> Host: localhost:6927
> Accept: */*
> 
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: Accept, X-Requested-With, Content-Type
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< Content-Security-Policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
< X-Content-Security-Policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
< X-WebKit-CSP: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
< Content-Type: text/html; charset=utf-8
< Vary: Accept-Encoding
< Date: Wed, 23 Sep 2015 22:04:00 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked

$ curl -v localhost:6927/en.wikipedia.beta.wmflabs.org/v1/page/mobile-html/Foobar
> GET /en.wikipedia.beta.wmflabs.org/v1/page/mobile-html/Foobar HTTP/1.1
> User-Agent: curl/7.38.0
> Host: localhost:6927
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: Accept, X-Requested-With, Content-Type
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< Content-Security-Policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
< X-Content-Security-Policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
< X-WebKit-CSP: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
< Content-Type: application/json; charset=utf-8
< Content-Length: 222
< Vary: Accept-Encoding
< Date: Wed, 23 Sep 2015 22:04:24 GMT
< Connection: keep-alive

It needs to work well for *.wikipedia.beta.wmflabs.org domains as well, since the Beta Cluster is where we test the integration of various services.

Details

Related Gerrit Patches:
mediawiki/services/mobileapps : masterProd config: fix the MW API URI
operations/puppet : productionMobileApps: Fix the MW API URI, revert c848ca4fd and resurect restbase_uri
operations/puppet : productionMobileApps: Specify the content type in the MW API call
operations/puppet : productionMobileApps: Put in place proper request templates for RB and MW
mediawiki/services/mobileapps/deploy : masterUpdate mobileapps to 5be9a79
mediawiki/services/mobileapps : masterBuild request URIs with templating from swagger-router

Event Timeline

mobrovac raised the priority of this task from to Needs Triage.
mobrovac updated the task description. (Show Details)
mobrovac added a subscriber: mobrovac.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 23 2015, 10:09 PM
bearND triaged this task as Medium priority.Nov 6 2015, 5:00 AM
bearND set Security to None.
bearND moved this task from Incoming to Backlog on the Mobile-Content-Service board.
mobrovac raised the priority of this task from Medium to High.Feb 18 2016, 12:49 AM

Ping? We cannot test new features such as caching and purging because of this.

Mholloway moved this task from Backlog to Doing on the Mobile-Content-Service board.

I'm getting a 504 Gateway Timeout error when I curl the Beta Cluster instead of a 404 as shown above. But no headway yet.

Putting this up for grabs in case Bernd wants to look at it. I'm pretty focused on an app-side task now and picked it up prematurely.

Mholloway removed Mholloway as the assignee of this task.Feb 22 2016, 11:35 PM
Mholloway moved this task from Doing to Backlog on the Mobile-Content-Service board.
bearND added a subscriber: bearND.Feb 23 2016, 1:01 AM

I am also getting a 504. BTW, the URIs need to be updated. There is no mobile-html endpoint. It's called mobile-sections.

curl -sI localhost:6927/en.wikipedia.org/v1/page/mobile-sections/Foobar
HTTP/1.1 200 OK
access-control-allow-origin: *
access-control-allow-headers: accept, x-requested-with, content-type
access-control-expose-headers: etag
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
content-security-policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
x-content-security-policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
x-webkit-csp: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
etag: 702883545/4fbbb565-d9c3-11e5-8ddf-ab3933346382
Content-Type: application/json; charset=utf-8
Content-Length: 23141
Vary: Accept-Encoding
Date: Tue, 23 Feb 2016 00:20:58 GMT
Connection: keep-alive

curl -sI localhost:6927/en.wikipedia.beta.wmflabs.org/v1/page/mobile-sections/Foobar
HTTP/1.1 504 Gateway Timeout
access-control-allow-origin: *
access-control-allow-headers: accept, x-requested-with, content-type
access-control-expose-headers: etag
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
content-security-policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
x-content-security-policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
x-webkit-csp: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
Content-Type: application/json; charset=utf-8
Content-Length: 151
Vary: Accept-Encoding
Date: Tue, 23 Feb 2016 00:21:32 GMT
Connection: keep-alive

It may have something to do with Parsoid or SSL being misconfigured on beta labs:

This is what I get from the log output of the service when it tries to get the Parsoid content:

"name":"service-mobileapp-node","hostname":"***","pid":2857,"level":50,"message":"504: internal_http_error","status":504,"type":"internal_http_error","detail":"504: internal_http_error","levelPath":"error/504","request_id":"12bfe9e1-d9c5-11e5-9455-6898d536970e","msg":"504: internal_http_error","time":"2016-02-23T00:33:35.417Z","v":0}

The code is here: https://github.com/wikimedia/mediawiki-services-mobileapps/blob/master/lib/parsoid-access.js#L102-L103.
It calls preq.get on the Parsoid URI https://en.wikipedia.beta.wmflabs.org/api/rest_v1/page/html/Foobar.

curl https://en.wikipedia.beta.wmflabs.org/api/rest_v1/page/html/Foobar

complains about SSL config. I suspect that preq.get doesn't like it. Guidance from the services team is appreciated.

@mobrovac We need your help with that.

The 504 is happening because we're hard-coding our protocols in our request URIs to https, but non-production environments like the beta cluster don't have https correctly configured. Changing the following to http fixes the issue:

https://github.com/wikimedia/mediawiki-services-mobileapps/blob/master/lib/mwapi.js#L82
https://github.com/wikimedia/mediawiki-services-mobileapps/blob/master/config.dev.yaml#L62
https://github.com/wikimedia/mediawiki-services-mobileapps/blob/master/config.prod.yaml#L36

Probably better, though, to test for a production URI and use https only in that case. Patch coming shortly.

Mholloway moved this task from Backlog to Doing on the Mobile-Content-Service board.

Change 273026 had a related patch set uploaded (by Mholloway):
Use HTTPS only for requests to production sites

https://gerrit.wikimedia.org/r/273026

I think having proper URI template handling in the code would very easily solve this problem. Such functionality is already available in the swagger-router package, in the Template object. This would allow us to have proper request objects compiled for each environment for calling both RESTBase and the MW API.

See:

Change 273026 merged by Mobrovac:
Build request URIs with templating from swagger-router

https://gerrit.wikimedia.org/r/273026

Change 274348 had a related patch set uploaded (by Mobrovac):
MobileApps: Put in place proper request templates for RB and MW

https://gerrit.wikimedia.org/r/274348

Change 274352 had a related patch set uploaded (by Mobrovac):
Update mobileapps to 5be9a79

https://gerrit.wikimedia.org/r/274352

Change 274352 merged by Mobrovac:
Update mobileapps to 5be9a79

https://gerrit.wikimedia.org/r/274352

Change 274348 merged by Giuseppe Lavagetto:
MobileApps: Put in place proper request templates for RB and MW

https://gerrit.wikimedia.org/r/274348

Mentioned in SAL [2016-03-02T12:00:25Z] <mobrovac> mobileapps stopping the service on scb1001 for debug purposes, T113542

Change 274373 had a related patch set uploaded (by Mobrovac):
MobileApps: Specify the content type in the MW API call

https://gerrit.wikimedia.org/r/274373

Change 274373 merged by Giuseppe Lavagetto:
MobileApps: Specify the content type in the MW API call

https://gerrit.wikimedia.org/r/274373

Mentioned in SAL [2016-03-02T12:25:24Z] <mobrovac> mobileapps rolling back to 68e38ec7, problems found in the latest deploy for T113542

Mentioned in SAL [2016-03-02T12:32:42Z] <mobrovac> mobileapps stopping (again) the service on scb1001 for debugging, T113542

Change 274381 had a related patch set uploaded (by Mobrovac):
MobileApps: Fix the MW API URI, revert c848ca4fd and resurect restbase_uri

https://gerrit.wikimedia.org/r/274381

Change 274381 merged by Giuseppe Lavagetto:
MobileApps: Fix the MW API URI, revert c848ca4fd and resurect restbase_uri

https://gerrit.wikimedia.org/r/274381

Mentioned in SAL [2016-03-02T13:10:02Z] <mobrovac> mobileapps re-deploying d384f1ba for T113542

Change 274384 had a related patch set uploaded (by Mobrovac):
Prod config: fix the MW API URI

https://gerrit.wikimedia.org/r/274384

Change 274384 merged by Mobrovac:
Prod config: fix the MW API URI

https://gerrit.wikimedia.org/r/274384

mobrovac closed this task as Resolved.Mar 2 2016, 1:23 PM
mobrovac removed a project: Patch-For-Review.
mobrovac removed a subscriber: gerritbot.

This is now done. Thank you @Mholloway for the code patch!