Page MenuHomePhabricator

<CORE TECHNOLOGY> API Migration & RESTbase Sunset
Open, In Progress, HighPublic1 Estimated Story Points

Description

Request Status: Existing Request
Request Type: Project Support + deprecate

Request Title: Deprecation of RESTbase

  • Request Description: Currently blocking Parsoid initiative. Aim is to unblock and improve performance by deprecating RestBase.
  • Indicate Priority Level: High
  • Main Requestors: Content Transform
  • Ideal Delivery Date:add target date
  • Stakeholders:
Stakeholder Name (Team)Related DocumentDependency
Subbu S. (Content TransformTBDCritical for Parsoid deployment
Ryan B. (Enterprise)T294798: Allow-Listing for Enterprise IPsCritical for Enterprise go-live

Request Documentation

Document TypeRequired?Document/Link
Related PHAB TicketsYes<add links here>
Product One PagerYes<add link here>
Product Requirements Document (PRD)Yes<add link here>
Product RoadmapYes<add link here>
Product Planning/Business CaseNo<add link here>
Product BriefNo<add link here>
Other LinksNoVision Table: https://www.mediawiki.org/wiki/Core_Platform_Team/Initiatives/Sunsetting_RESTBase

Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusSubtypeAssignedTask
StalledNone
In ProgressNone
ResolvedJgiannelos
In ProgressNone
DuplicateNone
OpenNone
In ProgressNone
DeclinedNone
DeclinedNone
ResolvedBPirkle
Openlbowmaker
Openlbowmaker
OpenNone
OpenNone
ResolvedDAlangi_WMF
ResolvedJgiannelos
OpenNone
OpenNone
In ProgressBPirkle
OpenNone
Resolveddaniel
Resolveddaniel
Resolvedhnowlan
Resolveddaniel
OpenNone
OpenNone
In Progressdaniel
ResolvedBPirkle

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@WDoranWMF it seems that the progress here is somehow stalled. Is this a project that is actively pursued? I would like to make some progress and get a restbaseless version of the Math extension bundled into mediawiki version 1.37. I had the impression that might work under the restbase sunset umbrella, but maybe this was to optimistic. I have the feeling I am somehow out of the loop.

sdkim renamed this task from RestBase Sunset to API Migration & RESTbase Sunset.Jun 8 2021, 8:36 PM
sdkim claimed this task.
sdkim removed a subscriber: eprodromou.
DAbad renamed this task from API Migration & RESTbase Sunset to <CORE TECHNOLOGY> API Migration & RESTbase Sunset.Dec 15 2021, 4:42 PM
DAbad updated the task description. (Show Details)

@WDoranWMF is this is still a goal? I am happy to contribute in the math dimension, but time to plan ahead would be appreciated.

December 21, 2021 Parsoid & RESTbase Deprecation Discussion:
Notes doc: https://docs.google.com/document/d/1b8ylJELbtCb7yszL4sUzo5Wa-zkNJh-7xGyKc869-9A/edit

Discussion Notes:

    • Stages for Migrating all clients to parser cache:
      1. Build out VE capabilities into core (https://phabricator.wikimedia.org/T264669)
      2. Parsoid read-view rollout
        • would need to add ticket to duplicate Parsoid content in both parser cache and RESTbase
  • We can individually migrate each client:
    • Parsoid html - clear path is parser cache
    • Mobile html - may not need to store it - Content Transform team would conduct study to see if feasible
    • Mobile Sections - mobile html was supposed to replace this at some point, but there may still be usage. can run a query in Hive to check
  • Benefits of the RESTbase deprecation:
    • Bring up to industry standards - this was originally built before the concept of API gateways were mainstream and thus was custom built. Now there are many third party resources with strong industry standards supporting them that we can leverage.
    • Mitigate technical debt - sits on a large codebase that is not actively maintained and not compatible with industry standards making it more difficult to maintain
    • CapEx Upsides - uses about 30 machines and lots of disk. These could be re-pooled and once migration is complete would reduce the number of times the same article is stored from 4 times to a single version

Next Steps:

  • Review and estimate effort. determine roadmap and execution plan
  • review mobile html and mobile sections for determination on next steps
  • add ticket to path to replicate parsoid content for migration purposes
  • consider submitting to TDMP for wide-spread commentary
  • review steps to complete migration of feature feed off of RESTbase to api gateway (aka gateway calls service directly)

@DAbad: Could that GDoc be made public? See Physikerwelt's last comment.

Removing inactive task assignee.

2022/01/19 Steering Committee Discussion:

  • WD: quite a significant amount of work remains and the potential for breakage is high due to the complexity
  • Data persistence will not be an issue.
  • Resource would include: 3-4 people from PET + parsing team + visual editor
  • would take a month for Petr & Daniel + someone from Parsing
DAbad changed the task status from Open to In Progress.Feb 25 2022, 4:49 PM
DAbad triaged this task as High priority.

@WDoranWMF and @DAbad it seems that the chromium-render service (proton or PDF renderer) doesn't have a task yet, but it is part of restbase (see code here)

I believe the work needed is similar to mobileapps: move it behind API Gateway.

Background Information about this service:

  • chromium-render is a stateless service and has already been deployed to k8s
  • It's essentially a nodejs wrapper around puppeteer to generate PDF files
  • From restbase code and production configuration it seems that there's no storage configuration for this service

Additionally, I don't see tasks for wikifeeds and recommendations-api, both services are used by the Apps Teams. For those I believe there are some product decisions that need to be made, since they are services that are wrappers around mwapi and a maintenance burden, we could do a better job moving some of it to the MW core or deprecating functionality.

DAbad changed the status of subtask T263489: AQS 2.0 from Open to In Progress.Apr 26 2022, 2:22 PM

There has been some questions in the community lately about what this means for the public rest api endpoints (e.g. https://wikimedia.org/api/rest_v1/ ). My assumption is that this is about implementation technology & data storage, but the public apis are not changing in any user facing way. Is that correct?

There has been some questions in the community lately about what this means for the public rest api endpoints (e.g. https://wikimedia.org/api/rest_v1/ ). My assumption is that this is about implementation technology & data storage, but the public apis are not changing in any user facing way. Is that correct?

That's correct. The only outlier AFIK for this statement is Mobile-Content-Service that has been tagged for deprecation many years ago and the RESTBase sunsetting gave the last push needed.