Page MenuHomePhabricator

Extract the feed endpoints from PCS into a new wikifeeds service
Open, HighPublic

Description

We'd like to move some functionality currently in MCS into different deployables so that the various services can be scaled independently.

Current plan

  • Move feed endpoints to new wikifeeds service
  • Keep /page/definitions in mobileapps
  • Remove mobile-sections from mobileapps when the apps switch to mobile-html
  • mobileapps is now PCS!

Todo

  • Setup new Gerrit repos
  • Decide on a new port number for the service. Ideally this time we'd use the same number in all the different environments (local, beta cluster, prod, ...).
  • Move the code that pertains to PCS to the new repo. If there is a lot of overlap we might need to create a common library in npm.
  • New labs instance (with Services?)
  • New beta cluster setup (with Services, RelEng)
  • Deploy wikifeeds to production
  • RB config files would need a few more entries: one per deployable service instead of one options.mobileapps (with Services)
  • Update RB to request feed content from wikifeeds, not mobileapps
  • Remove feed-related code from mobileapps

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Mholloway updated the task description. (Show Details)Jun 11 2019, 10:12 PM

I created a new beta cluster instance, deployment-wikifeeds01, proxied to https://wikifeeds-beta.wmflabs.org; however, most endpoints are not working due to ca-certificates not being present in the production image variant as discussed on https://github.com/wikimedia/service-template-node/pull/120.

I created a new beta cluster instance, deployment-wikifeeds01, proxied to https://wikifeeds-beta.wmflabs.org; however, most endpoints are not working due to ca-certificates not being present in the production image variant as discussed on https://github.com/wikimedia/service-template-node/pull/120.

Have you configured it to work with beta domain or production ones? In the case of the former, I'd be surprised it doesn't work, in the case of the latter I wouldn't.

@mobrovac The Beta Cluster sites also require HTTPS and throw 403 otherwise -- try curl -d "action=query&meta=siteinfo" http://en.wikipedia.beta.wmflabs.org/w/api.php, for example.

@mobrovac The Beta Cluster sites also require HTTPS and throw 403 otherwise -- try curl -d "action=query&meta=siteinfo" http://en.wikipedia.beta.wmflabs.org/w/api.php, for example.

That's a configuration issue with the service and how it issues requests to the MW API, then. Because if they are both in the same (beta) environment, no SSL should be needed. If, however, the service makes requests to MW's public IP then it's needed.

@mobrovac Of course you are correct—I needed to provide an override command in order to use the correct configuration. All endpoints are functional now (or at least as functional as can be expected when they're not necessarily expecting to handle beta cluster domains).

Mholloway renamed this task from Split Page Content Service and Mobile Content Service into separate repositories to Extract the feed endpoints from PCS into a new wikifeeds service.Jun 17 2019, 7:38 PM
Mholloway raised the priority of this task from Normal to High.Jul 30 2019, 4:35 PM

Let's bump this to high priority given the struggles under the added load as of late.

MSantos moved this task from Backlog to Epics on the Page Content Service board.Sep 2 2019, 4:47 PM

Change 535522 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/dns@master] Introduce wikifees LVS IPs

https://gerrit.wikimedia.org/r/535522

Change 535523 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/dns@master] Add discovery RRs for wikifeeds

https://gerrit.wikimedia.org/r/535523

Change 535525 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Add kubernetes wikifeeds stanzas

https://gerrit.wikimedia.org/r/535525

Change 535531 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] Add wikifeeds namespace

https://gerrit.wikimedia.org/r/535531

Change 535533 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] Add wikifeeds helmfile stanzas

https://gerrit.wikimedia.org/r/535533

Change 535525 merged by Alexandros Kosiaris:
[operations/puppet@production] Add kubernetes wikifeeds stanzas

https://gerrit.wikimedia.org/r/535525

Change 535531 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Add wikifeeds namespace

https://gerrit.wikimedia.org/r/535531

Change 535601 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] Add wikifeeds to admin environments

https://gerrit.wikimedia.org/r/535601

Change 535601 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Add wikifeeds to admin environments

https://gerrit.wikimedia.org/r/535601

Change 535533 merged by Mholloway:
[operations/deployment-charts@master] Add wikifeeds helmfile stanzas

https://gerrit.wikimedia.org/r/535533

Change 535522 merged by Alexandros Kosiaris:
[operations/dns@master] Introduce wikifeeds LVS IPs

https://gerrit.wikimedia.org/r/535522

Mholloway updated the task description. (Show Details)Mon, Sep 16, 2:40 PM

Change 537128 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] wikifeeds: Define nodePorts in all envs

https://gerrit.wikimedia.org/r/537128

Change 537134 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Add LVS for wikifeeds

https://gerrit.wikimedia.org/r/537134

Change 537128 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] wikifeeds: Define nodePorts in all envs

https://gerrit.wikimedia.org/r/537128

Change 537135 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] wikifeeds: Enabling paging

https://gerrit.wikimedia.org/r/537135

Change 537134 merged by Alexandros Kosiaris:
[operations/puppet@production] Add LVS for wikifeeds

https://gerrit.wikimedia.org/r/537134

Wikifeeds is up and running, but appears to be using the config.yaml from the service repo root (symlinked from config.dev.yaml) rather than the config defined in the chart.

Otherwise, I'm not sure how or why it could be trying to use a public Wikipedia domain to access the MW API:

mholloway-shell@deploy1001:~$ curl wikifeeds.svc.eqiad.wmnet:8889/en.wikipedia.org/v1/page/most-read/2018/05/05
{"status":504,"type":"internal_http_error","detail":"Error: ETIMEDOUT","method":"post","uri":"https://en.wikipedia.org/w/api.php"}

Wikifeeds is up and running, but appears to be using the config.yaml from the service repo root (symlinked from config.dev.yaml) rather than the config defined in the chart.

This has been filed as T233076: wikifeeds does not honor mwapi_req config.

Change 535523 merged by Alexandros Kosiaris:
[operations/dns@master] Add discovery RRs for wikifeeds

https://gerrit.wikimedia.org/r/535523

akosiaris closed this task as Resolved.Wed, Sep 18, 2:04 PM
akosiaris added a subscriber: akosiaris.

https://grafana.wikimedia.org/d/35vIuGpZk/wikifeeds?refresh=1m&orgId=1 has a tentative dashboard. Feel free to augment

akosiaris reopened this task as Open.Wed, Sep 18, 2:05 PM

Change 537750 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[operations/puppet@production] RESTBase: Configure wikifeeds_uri

https://gerrit.wikimedia.org/r/537750

Change 537754 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/services/restbase/deploy@master] Update mobileapps_uri and use wikifeeds

https://gerrit.wikimedia.org/r/537754

Change 537750 merged by Alexandros Kosiaris:
[operations/puppet@production] RESTBase: Configure wikifeeds_uri

https://gerrit.wikimedia.org/r/537750

Change 538238 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/deployment-charts@master] RESTRouter: Clean up the config && add the wikifeeds URI

https://gerrit.wikimedia.org/r/538238

Change 538238 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] RESTRouter: Clean up the config && add the wikifeeds URI

https://gerrit.wikimedia.org/r/538238

Change 538242 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] Release restrouter chart version 0.0.3

https://gerrit.wikimedia.org/r/538242

Change 538242 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Release restrouter chart version 0.0.3

https://gerrit.wikimedia.org/r/538242

Change 537754 merged by Mobrovac:
[mediawiki/services/restbase/deploy@master] Update mobileapps_uri and use wikifeeds

https://gerrit.wikimedia.org/r/537754

Mentioned in SAL (#wikimedia-operations) [2019-09-24T11:45:22Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@87eea26]: Start using the wikifeeds service for v1/feed - T170455

Mentioned in SAL (#wikimedia-operations) [2019-09-24T11:47:57Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@87eea26]: Start using the wikifeeds service for v1/feed - T170455 (duration: 02m 35s)

Mentioned in SAL (#wikimedia-operations) [2019-09-24T12:16:20Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@19d0f44]: REVERT (due to wikifeeds problems): Start using the wikifeeds service for v1/feed - T170455

Mentioned in SAL (#wikimedia-operations) [2019-09-24T12:18:55Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@19d0f44]: REVERT (due to wikifeeds problems): Start using the wikifeeds service for v1/feed - T170455 (duration: 02m 35s)

@Mholloway I tried to deploy the RESTBase side of things, but Wikifeeds times out when sent a request for feed/featured. We have identified the culprit to be the requestFeaturedFeedSpecialItem() method. Concretely, the method uses directly the URL https://${domain}/${path}/${formattedDateString}/${lang}?useskin=apioutput, however, it should follow the templating principle just like the mwApiGet() method does. That way, we can transform the template to be compatible with the way communication with the MW cluster is done inside production.

To clarify, MCS contains the same code, but works because calling directly the domain from SCB works. Wikifeeds does not work because k8s policies explicitly forbid public IPs to be resolved.

@mobrovac Thanks for attempting the deploy earlier. I hadn't realized we had a stray call to a public URL lurking in this codebase.

I guess we have a few different options from here:

  • We could update that code to get the content it needs directly from the RSS feed provided by the existing action=featuredfeed MW API module. This is the quickest and easiest solution, and I actually have a patch ready to go for it. The downside is that it removes our ability to get the featured content for an arbitrary date; we'd only have access to what happens to be in the current RSS feed at any given time.
  • We could (as suggested) add a new request template for a MW page request, and convert the existing request to use the new template (with an appserver service discovery URI in production).
  • We could add another MW API module to FeaturedFeeds to provide the featured content for an arbitrary date.

I'm most inclined to do the second option, even though it probably won't happen until next week, after we're back from the offsite. I am really looking forward to getting this into production, but it's not so desperately urgent that we need to resort to stopgap measures that will most likely end up needing to be updated later (which is what I imagine option #1 would turn out to be).

Mholloway updated the task description. (Show Details)Wed, Sep 25, 3:44 AM

I'd be for the second option as well, it's the correct way to go about this. Proton does this, so you can get inspiration there (cf. config, setup and usage).

In the meantime, RESTBase will continue using MCS.

Change 539642 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/services/wikifeeds@master] Fix: Use templated request for MediaWiki page requests

https://gerrit.wikimedia.org/r/539642

Change 539642 merged by jenkins-bot:
[mediawiki/services/wikifeeds@master] Fix: Use templated request for MediaWiki page requests

https://gerrit.wikimedia.org/r/539642

Change 540967 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[operations/deployment-charts@master] [WiP] Update wikifeeds

https://gerrit.wikimedia.org/r/540967

Change 540967 merged by Mholloway:
[operations/deployment-charts@master] Update wikifeeds chart to 0.0.4

https://gerrit.wikimedia.org/r/540967

Change 541548 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[operations/deployment-charts@master] Update charts/index.yaml to add wikifeeds v0.0.4 chart

https://gerrit.wikimedia.org/r/541548

Change 541548 merged by Mholloway:
[operations/deployment-charts@master] Update charts/index.yaml to add wikifeeds v0.0.4 chart

https://gerrit.wikimedia.org/r/541548

@mobrovac The wikifeeds config update has been deployed, along with an update to the latest production image. If you could retry the switchover, I'd appreciate it!

Mentioned in SAL (#wikimedia-operations) [2019-10-09T12:20:41Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@068d2ed]: Feed: Use Wikifeeds; Parsoid: Use the ETag revid for stashing and use the same ETag for stashing and response - T170455 T234928

Mentioned in SAL (#wikimedia-operations) [2019-10-09T12:30:21Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@068d2ed]: Feed: Use Wikifeeds; Parsoid: Use the ETag revid for stashing and use the same ETag for stashing and response - T170455 T234928 (duration: 09m 40s)

mobrovac updated the task description. (Show Details)Wed, Oct 9, 1:00 PM

RESTBase is now using wikifeeds for the feed content \o/

Change 541827 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/services/mobileapps@master] Remove feed-related code

https://gerrit.wikimedia.org/r/541827

Mholloway moved this task from Backlog to Code Review on the Wikifeeds board.Wed, Oct 9, 3:14 PM

Change 541827 merged by jenkins-bot:
[mediawiki/services/mobileapps@master] Remove feed-related code

https://gerrit.wikimedia.org/r/541827

Change 542061 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/deployment-charts@master] RESTRouter: Use image v1.1.3

https://gerrit.wikimedia.org/r/542061

Change 542061 merged by jenkins-bot:
[operations/deployment-charts@master] RESTRouter: Use image v1.1.3

https://gerrit.wikimedia.org/r/542061