Page MenuHomePhabricator
Create Task
Maniphest T339119

Switchover plan from restbase to api gateway for wikifeeds
Closed, ResolvedPublic
Actions

Description

Mapping of production URLs to be routed to wikifeeds:

Service location: wikifeeds.discovery.wmnet
Service port: 4101
Use TLS: yes

  • Announcements endpoint
    • Production endpoint:
      • <domain>/api/rest_v1/feed/announcements
    • RESTBase endpoint:
      • <domain>/v1/feed/announcements
    • Wikifeeds endpoint:
      • <domain>/v1/feed/announcements
  • Onthisday endpoint
    • Production endpoint:
      • <domain>/api/rest_v1/feed/onthisday/<type>/<mm>/<dd>
    • RESTBase endpoint:
      • <domain>/v1/feed/onthisday/<type>/<mm>/<dd>
    • Wikifeeds endpoint:
      • <domain>/v1/aggregated/onthisday/<type>/<mm>/<dd>
  • Featured endpoint
    • Production endpoint:
      • <domain>/api/rest_v1/feed/featured/<year>/<mm>/<dd>
    • RESTBase endpoint:
      • <domain>/v1/feed/featured/<year>/<mm>/<dd>
    • Wikifeeds endpoint:
      • <domain>/v1/aggregated/featured/<year>/<mm>/<dd>
  • Random endpoint
    • Production endpoint:
      • <domain>/api/rest_v1/page/random/<format>
    • RESTBase endpoint:
      • <domain>/v1/page/random/<format>
    • Wikifeeds endpoint:
      • <domain>/v1/page/random/redirect/<format>

Only for wikimedia.org level (used by android apps):

  • RESTBase
    • {domain}/v1/feed/availability
  • Wikifeeds
    • {domain}/v1/feed/availability

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
trafficserver: route to wikifeeds via rest-gatewayoperations/puppetproduction+2 -0
trafficserver: route wikifeedsoperations/puppetproduction+2 -0
Add missing cache-control headersmediawiki/services/wikifeedsmaster+9 -0
trafficserver: route wikifeeds requests via the rest-gatewayoperations/puppetproduction+2 -0
rest-gateway: add availability routeoperations/deployment-chartsmaster+3 -0
Bump onthisday spec version to match restbasemediawiki/services/wikifeedsmaster+3 -3
rest-gateway: Add route for wikifeeds availabilityoperations/deployment-chartsmaster+3 -0
rest-gateway: Fix onthisday feed routingoperations/deployment-chartsmaster+4 -1
rest-gateway: add citoid and wikifeeds egressoperations/deployment-chartsmaster+17 -0
rest-gateway: add routes for wikifeedsoperations/deployment-chartsmaster+19 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Jgiannelos created this task.Jun 14 2023, 11:47 AM
Jgiannelos updated the task description. (Show Details)Jul 3 2023, 2:31 PM
Jgiannelos updated the task description. (Show Details)Jul 3 2023, 2:36 PM
Jgiannelos added a project: serviceops-deprecated.Jul 13 2023, 2:11 PM
Jgiannelos added a subscriber: hnowlan.
hnowlan renamed this task from Switchover plan from restbase to api gateway to Switchover plan from restbase to api gateway for wikifeeds.Jul 14 2023, 3:10 PM
hnowlan updated the task description. (Show Details)
hnowlan added a comment.Jul 14 2023, 3:19 PM

Just to note that wikifeeds sends a different access-control-allow-headers header - restbase overrides this so it's not a major concern for the rest gateway.

< access-control-allow-headers: accept, x-requested-with, content-type
gerritbot added a comment.Jul 14 2023, 3:33 PM

Change 938265 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] rest-gateway: add routes for wikifeeds

https://gerrit.wikimedia.org/r/938265

gerritbot added a project: Patch-For-Review.Jul 14 2023, 3:33 PM
Jgiannelos moved this task from Backlog to In Progress on the Content-Transform-Team-WIP board.Jul 17 2023, 3:21 PM
hnowlan added a comment.Jul 27 2023, 11:38 AM

Looking at the spec, it appears there are a few more endpoints not covered: https://github.com/wikimedia/mediawiki-services-wikifeeds/blob/master/spec.yaml Should these also be configured? it also appears that aggregated is an alternate path for most (all?) endpoints - should all cases of this be catered for?

Jgiannelos added a comment.EditedJul 27 2023, 12:00 PM

These were never exposed in public traffic so I don't think they need to be exposed now (also unnecessary risk)

Jgiannelos added a comment.Jul 27 2023, 12:24 PM

Regarding aggregated are you referring to the query aggregated = true for example here? https://github.com/wikimedia/mediawiki-services-wikifeeds/blob/master/spec.yaml#L366
I think that wikifeeds has (some) aggregate implementation but for parity i just implemented what restbase is doing.

hnowlan added a comment.Jul 27 2023, 3:56 PM

As regards aggregated stuff, I mean the difference between https://wikifeeds.discovery.wmnet:4101/en.wikipedia.org/v1/page/featured/2023/01/01 (2607 chars) and curl https://wikifeeds.discovery.wmnet:4101/en.wikipedia.org/v1/aggregated/featured/2023/01/01 (321309 chars). Just wanted to make sure as I typoed my tests and discovered that during testing - Happy to follow your lead on the above being the canonical list

Jgiannelos added a comment.Jul 27 2023, 3:59 PM

The /aggregated path is to be exposed for the restbase replacement, the other shouldn't be exposed.

Jgiannelos updated the task description. (Show Details)Jul 28 2023, 12:08 PM
gerritbot added a comment.Jul 28 2023, 2:50 PM

Change 938265 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: add routes for wikifeeds

https://gerrit.wikimedia.org/r/938265

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2023, 3:10 PM
MSantos moved this task from In Progress to To Deploy on the Content-Transform-Team-WIP board.Jul 31 2023, 3:16 PM
gerritbot added a comment.Jul 31 2023, 4:14 PM

Change 943609 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] rest-gateway: add citoid and wikifeeds egress

https://gerrit.wikimedia.org/r/943609

gerritbot added a project: Patch-For-Review.Jul 31 2023, 4:14 PM
gerritbot added a comment.Aug 1 2023, 10:28 AM

Change 943609 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: add citoid and wikifeeds egress

https://gerrit.wikimedia.org/r/943609

Maintenance_bot removed a project: Patch-For-Review.Aug 1 2023, 10:30 AM
Jgiannelos added a comment.Aug 3 2023, 8:30 AM

Not specific to this specific ticket but while checking the difftests between restbase and rest-gateway it looks like we have a missing header that is applied globally on restbase:

"content-location": "https://fr.wikipedia.org/api/rest_v1/feed/onthisday/all/11/23"

This is injected by the restbase codebase:
https://github.com/wikimedia/restbase/blob/50152c49dc8b8e7219bf00562ca4b37e7f057c8a/lib/content_location_filter.js#L70

I am not sure if its needed to be ported though.

Jgiannelos added a comment.EditedAug 3 2023, 9:14 AM

From difftests:

  • Random endpoint redirects as expected
  • Announcements is return the same responses between restbase/gateway (although empty)
  • Onthisday mostly returns the same responses (there was a 10% responses with differences but it looked like it was a content change between the requests, structure was fine)
Jgiannelos added a comment.Aug 3 2023, 9:15 AM

There were some missing headers but it should be fixed after this deployment chart bump:
https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/945539

Jgiannelos added a comment.Aug 3 2023, 10:22 AM

I rerun the tests once again after the patch and it looks like we are good to go:

  • Failures for onthisday were transient or internal metadata/tids changed
  • Cache/content-type headers look OK
gerritbot added a comment.Aug 3 2023, 10:38 AM

Change 945558 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: route wikifeeds requests via the rest-gateway

https://gerrit.wikimedia.org/r/945558

gerritbot added a project: Patch-For-Review.Aug 3 2023, 10:38 AM
Jgiannelos added a comment.Aug 3 2023, 10:49 AM

Lets hold on this because it looks like wikifeeds doesn't handle onthisday URL routing right. It only handles type=all currently working on a patch to fix the URL matching.

gerritbot added a comment.Aug 3 2023, 11:14 AM

Change 945567 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[operations/deployment-charts@master] rest-gateway: Fix onthisday feed routing

https://gerrit.wikimedia.org/r/945567

gerritbot added a comment.Aug 3 2023, 11:23 AM

Change 945567 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Fix onthisday feed routing

https://gerrit.wikimedia.org/r/945567

Jgiannelos added a comment.Aug 3 2023, 11:44 AM

Just tested the fix on staging and looks OK.

Jgiannelos added a comment.Aug 3 2023, 12:05 PM

Something that we've not captured is /feed/availability which is only served under wikimedia.org RESTBase endpoint:

https://wikimedia.org/api/rest_v1/feed/availability

This brings the following 2 topics:

  • We should add a route for /feed/availability (its just passthrough to wikifeeds so its just a matter of rest-gateway routing) ?
  • How are we handling the namespacing pattern RESTBase implements (different configurations per group of wikis) ?
Jgiannelos updated the task description. (Show Details)Aug 3 2023, 12:08 PM
gerritbot added a comment.Aug 3 2023, 12:34 PM

Change 945591 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[operations/deployment-charts@master] rest-gateway: Add route for wikifeeds availability

https://gerrit.wikimedia.org/r/945591

gerritbot added a comment.Aug 3 2023, 1:15 PM

Change 945599 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[mediawiki/services/wikifeeds@master] Bump onthisday spec version to match restbase

https://gerrit.wikimedia.org/r/945599

gerritbot added a comment.Aug 3 2023, 2:08 PM

Change 945591 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Add route for wikifeeds availability

https://gerrit.wikimedia.org/r/945591

gerritbot added a comment.Aug 3 2023, 3:06 PM

Change 945599 merged by jenkins-bot:

[mediawiki/services/wikifeeds@master] Bump onthisday spec version to match restbase

https://gerrit.wikimedia.org/r/945599

Jgiannelos mentioned this in rWSWF5b0370b8f84c: Bump onthisday spec version to match restbase.Aug 3 2023, 4:46 PM
gerritbot added a comment.Aug 4 2023, 1:55 PM

Change 945784 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] rest-gateway: add availability route

https://gerrit.wikimedia.org/r/945784

gerritbot added a comment.Aug 8 2023, 10:11 AM

Change 945784 abandoned by Hnowlan:

[operations/deployment-charts@master] rest-gateway: add availability route

Reason:

Already done

https://gerrit.wikimedia.org/r/945784

gerritbot added a comment.Aug 9 2023, 9:32 AM

Change 945558 merged by Hnowlan:

[operations/puppet@production] trafficserver: route wikifeeds requests via the rest-gateway

https://gerrit.wikimedia.org/r/945558

Maintenance_bot removed a project: Patch-For-Review.Aug 9 2023, 10:10 AM
gerritbot added a comment.Aug 9 2023, 10:10 AM

Change 947327 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[mediawiki/services/wikifeeds@master] Add missing cache-control headers

https://gerrit.wikimedia.org/r/947327

gerritbot added a project: Patch-For-Review.Aug 9 2023, 10:10 AM
gerritbot added a comment.Aug 9 2023, 10:47 AM

Change 947327 merged by jenkins-bot:

[mediawiki/services/wikifeeds@master] Add missing cache-control headers

https://gerrit.wikimedia.org/r/947327

Jgiannelos mentioned this in rWSWF91f5c801bd2b: Add missing cache-control headers.Aug 9 2023, 11:06 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 9 2023, 11:10 AM
gerritbot added a comment.Aug 9 2023, 3:19 PM

Change 947372 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: route wikifeeds

https://gerrit.wikimedia.org/r/947372

gerritbot added a project: Patch-For-Review.Aug 9 2023, 3:19 PM
gerritbot added a comment.Aug 9 2023, 3:28 PM

Change 947372 merged by Hnowlan:

[operations/puppet@production] trafficserver: route wikifeeds

https://gerrit.wikimedia.org/r/947372

Maintenance_bot removed a project: Patch-For-Review.Aug 9 2023, 3:30 PM
gerritbot added a comment.Sep 12 2023, 1:57 PM

Change 956895 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: route to wikifeeds via rest-gateway

https://gerrit.wikimedia.org/r/956895

gerritbot added a project: Patch-For-Review.Sep 12 2023, 1:57 PM
MSantos moved this task from Unsorted to Doing on the RESTBase Sunsetting board.Sep 13 2023, 12:01 PM
gerritbot added a comment.Sep 13 2023, 2:54 PM

Change 956895 merged by Hnowlan:

[operations/puppet@production] trafficserver: route to wikifeeds via rest-gateway

https://gerrit.wikimedia.org/r/956895

Maintenance_bot removed a project: Patch-For-Review.Sep 13 2023, 3:10 PM
MSantos assigned this task to Jgiannelos.Sep 18 2023, 9:20 AM
MSantos moved this task from To Deploy to To Verify on the Content-Transform-Team-WIP board.
Clement_Goubert moved this task from Incoming 🐫 to 🛎 Services & Oids on the serviceops-deprecated board.Sep 18 2023, 11:31 AM
Jgiannelos closed this task as Resolved.Sep 21 2023, 11:27 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits