Page MenuHomePhabricator

Extract the feed endpoints from PCS into a new wikifeeds service
Open, HighPublic

Description

We'd like to move some functionality currently in MCS into different deployables so that the various services can be scaled independently.

Current plan

  • Move feed endpoints to new wikifeeds service
  • Keep /page/definitions in mobileapps
  • Remove mobile-sections from mobileapps when the apps switch to mobile-html
  • mobileapps is now PCS!

Todo

  • Setup new Gerrit repos
  • Decide on a new port number for the service. Ideally this time we'd use the same number in all the different environments (local, beta cluster, prod, ...).
  • Move the code that pertains to PCS to the new repo. If there is a lot of overlap we might need to create a common library in npm.
  • New labs instance (with Services?)
  • New beta cluster setup (with Services, RelEng)
  • Deploy wikifeeds to production
  • RB config files would need a few more entries: one per deployable service instead of one options.mobileapps (with Services)
  • Update RB to request feed content from wikifeeds, not mobileapps
  • Remove feed-related code from mobileapps

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 12 2017, 6:30 PM
Jhernandez triaged this task as Normal priority.Jul 11 2018, 3:58 PM
Jhernandez added subscribers: bearND, Jhernandez.

@bearND Would you mind fleshing out the description here with the steps we need to do?

bearND updated the task description. (Show Details)Jul 11 2018, 5:36 PM

Some thoughts after poking around at this a bit:

The PCS endpoints are extremely tightly coupled with mobile-sections and its related library files and utils, which form the core of the original service. Rather than thinking about this task as extracting PCS from the mobileapps repo, I think we're better off thinking about this in the opposite way: as extracting things from mobileapps, until what remains is PCS.

Here's what is currently contained within the mobileapps repo other than the PCS endpoints:

  1. the Explore feed endpoints;
  2. the legacy, English-only Wiktionary definitions endpoint;
  3. mobile-sections.

The Explore feed plumbing is relatively self-contained, and I'm working on extracting it now. The definitions endpoint, frankly, I'd be happy to see deprecated and removed; I don't see us dedicating the resources to move it beyond an "experimental" English-only state in any foreseeable future. (If we must keep it alive, perhaps we can just grandfather it into PCS.) Mobile-sections we can deprecate, and then remove when the apps have switched over to mobile-html.

@Dbrant How sad would the Android app's English-language users be to see the "define" button go away?
@bearND Does that sound like a good way forward to you?

The result of extracting the feed endpoints from MCS is at https://github.com/mdholloway/wikifeeds. There are some utility methods copied over, mostly in lib/util.js, that we could look at putting into a shared node module somewhere down the line.

bearND added a comment.May 7 2019, 9:24 PM

Moving out the feed stuff sounds like a reasonable approach to me. I'm just not sure if we really should be starting with a brand new repo. Have you considered using a clone so we can keep the Git history?
Some of the libraries are shared between mobile-sections and the PCS endpoints. We may consider creating an npm library for sharing those if it make sense to do so.
Before we move code to other repos I think it might be beneficial to make sure the OpenAPI spec is merged and some other code cleanup is done. That could be updating of dependencies, esp. eslint configs, a convention for unit test file names.
We could start out with moving easily identifiable files and subfolders into different buckets, at least for the lib and test. Probably the routes could be done as well.

bearND updated the task description. (Show Details)May 7 2019, 9:25 PM
Mholloway added a comment.EditedMay 8 2019, 2:11 PM

Moving out the feed stuff sounds like a reasonable approach to me. I'm just not sure if we really should be starting with a brand new repo. Have you considered using a clone so we can keep the Git history?

OK. I'll update or redo this to preserve the commit history for included files.

Some of the libraries are shared between mobile-sections and the PCS endpoints. We may consider creating an npm library for sharing those if it make sense to do so.

Just to be clear, the path I am proposing here is for mobile-sections to remain in the mobileapps repo with the PCS endpoints until the Android app switches to mobile-html (at which point we remove mobile-sections, and the mobileapps repo becomes just PCS). That means the only code we'd have to split off into a shared library is what the feed endpoints are also using (which is much less than the amount shared between mobile-sections and the various PCS endpoints).

In the Kartotherian maps service repos we were using a bunch of npm modules unnecessarily and maintenance was pretty much a nightmare. That's the other extreme, but the point is that it'd be better to minimize the amount of code split off into standalone modules IMO.

Before we move code to other repos I think it might be beneficial to make sure the OpenAPI spec is merged and some other code cleanup is done. That could be updating of dependencies, esp. eslint configs, a convention for unit test file names.

Waiting on the OpenAPI spec change sort of makes sense, although doing it after the split would make each change easier to review (I note that Mateus's current patch changes over 1000 LOC). To be honest I don't see the point of blocking this on other stuff like lint config updates.

https://github.com/mdholloway/wikifeeds is updated to preserve the earlier commit history from mobileapps.

Keeping mobile-sections with the rest of PCS makes sense to me. Thank you for preserving the git history. The rest is nice to have and not necessarily blocking.

OK, great. I'll request a new Gerrit repo and keep pushing forward on this.

Mholloway renamed this task from Extract Page Content Service into new repository separate from the Mobile Content Service to Split Page Content Service and Mobile Content Service into separate repositories.May 15 2019, 7:36 PM
Mholloway updated the task description. (Show Details)
Mholloway updated the task description. (Show Details)May 15 2019, 7:42 PM

Change 510581 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[integration/config@master] Run service pipeline tests for mediawiki/services/wikifeeds

https://gerrit.wikimedia.org/r/510581

Any objection to port 8889 for the new wikifeeds service? (That's the next port after the production mobileapps port, 8888.)

bearND added a comment.EditedMay 16 2019, 7:05 PM

No objection, as long as we can have the same port in all environments (local dev, beta, production, mwvagrant, ...) if there is such a thing as an exposed port in Kubernetes.
(I think it adds more burden if the ports of the same service are different per environment. Example: In MCS we use 6927 for local development, but 8888 on the production machines.)

Mholloway added a comment.EditedMay 16 2019, 7:09 PM

Cool. I think we should be able to use 8889 for all environments. Let's plan on that, then.

I'm not sure how it came to be that mobileapps uses 6927 for local development and 8888 for production, except that 6927 is the just the services-template default. Actually, there's probably nothing stopping us from updating config.dev.yaml in mobileapps to use 8888 for development also.

Mholloway updated the task description. (Show Details)May 16 2019, 8:08 PM

Change 510581 merged by jenkins-bot:
[integration/config@master] Run service pipeline tests for mediawiki/services/wikifeeds

https://gerrit.wikimedia.org/r/510581

Mholloway updated the task description. (Show Details)May 24 2019, 2:49 PM

Change 514489 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[operations/puppet@production] Add nagios contact group for wikifeeds service

https://gerrit.wikimedia.org/r/514489

Change 514490 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[operations/puppet@production] Add role/profile for wikifeeds service

https://gerrit.wikimedia.org/r/514490

Change 514490 abandoned by Mholloway:
Add role/profile for wikifeeds service

https://gerrit.wikimedia.org/r/514490

Change 514489 abandoned by Mholloway:
Add nagios contact group for wikifeeds service

https://gerrit.wikimedia.org/r/514489

Change 514489 restored by Alexandros Kosiaris:
Add nagios contact group for wikifeeds service

https://gerrit.wikimedia.org/r/514489

Change 514489 merged by Alexandros Kosiaris:
[operations/puppet@production] Add nagios contact group for wikifeeds service

https://gerrit.wikimedia.org/r/514489

Change 516529 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/services/wikifeeds@master] Update configs to use port 8889

https://gerrit.wikimedia.org/r/516529

Change 516529 merged by jenkins-bot:
[mediawiki/services/wikifeeds@master] Update configs to use port 8889

https://gerrit.wikimedia.org/r/516529

Mholloway updated the task description. (Show Details)Jun 11 2019, 10:12 PM

I created a new beta cluster instance, deployment-wikifeeds01, proxied to https://wikifeeds-beta.wmflabs.org; however, most endpoints are not working due to ca-certificates not being present in the production image variant as discussed on https://github.com/wikimedia/service-template-node/pull/120.

I created a new beta cluster instance, deployment-wikifeeds01, proxied to https://wikifeeds-beta.wmflabs.org; however, most endpoints are not working due to ca-certificates not being present in the production image variant as discussed on https://github.com/wikimedia/service-template-node/pull/120.

Have you configured it to work with beta domain or production ones? In the case of the former, I'd be surprised it doesn't work, in the case of the latter I wouldn't.

@mobrovac The Beta Cluster sites also require HTTPS and throw 403 otherwise -- try curl -d "action=query&meta=siteinfo" http://en.wikipedia.beta.wmflabs.org/w/api.php, for example.

@mobrovac The Beta Cluster sites also require HTTPS and throw 403 otherwise -- try curl -d "action=query&meta=siteinfo" http://en.wikipedia.beta.wmflabs.org/w/api.php, for example.

That's a configuration issue with the service and how it issues requests to the MW API, then. Because if they are both in the same (beta) environment, no SSL should be needed. If, however, the service makes requests to MW's public IP then it's needed.

@mobrovac Of course you are correct—I needed to provide an override command in order to use the correct configuration. All endpoints are functional now (or at least as functional as can be expected when they're not necessarily expecting to handle beta cluster domains).

Mholloway renamed this task from Split Page Content Service and Mobile Content Service into separate repositories to Extract the feed endpoints from PCS into a new wikifeeds service.Jun 17 2019, 7:38 PM
Mholloway raised the priority of this task from Normal to High.Tue, Jul 30, 4:35 PM

Let's bump this to high priority given the struggles under the added load as of late.