Page MenuHomePhabricator

Split the RESTBase execution paths
Closed, ResolvedPublic0 Story Points

Description

Problem

RESTBase has two main hierarchies for each domain: /v1 and /sys. The former contains the public end points and, to a certain extent, some business logic for them, while the latter contains modules with business logic and storage access. We need to separate the execution paths in such a way that two independent services can be run in production: the front-end service exposing the public end points and containing all the business logic needed; and the back-end service which allows the front-end service (and others in the future) to store and retrieve values.

Solution

Taking advantage of RESTBase's modularity, the most natural way to split the execution paths is to have two configurations - one for each service (with some additional proxy modules).

Back-end Service

The back-end service should contain all of the modules that directly use the storage, as well as the higher-level key-value bucket abstraction for each domain that RESTBase is enabled on. The lower-level storage module should not be exposed to internal WMF clients, only the higher-level key-value module. The proposed hierarchy is:

Back-end
paths:
  /{api:v1}:
    x-modules:
      - spec:
          paths:
            /key_value:
              x-modules:
                - path: sys/key_value.js
            /page_revisions:
              x-modules:
                - path: sys/page_revisions.js
            /post_data:
              x-modules:
                - path: sys/post_data.js
        options: '{{options}}'
  /{api:sys}:
    x-modules:
      - spec:
          paths:
            /table:
              x-modules:
                - path: sys/table.js
                  options:
                    conf: '{{options.table}}'
        options: '{{options}}'

Front-end Service

The front-end service would largely have the same config as RESTBase currently does in the /v1 hierarchy. For /sys, business-logic modules would remain the same, while the ones that have been moved to the back-end service would either be removed (in the case of /sys/table) or the routes themselves would continue to exist, but new modules would be mounted under them that proxy requests to the back-end service (in the case of /sys/key_value, /sys/page_revisions and /sys/post_data).

Event Timeline

mobrovac triaged this task as Normal priority.Apr 12 2019, 9:19 PM
mobrovac created this task.
Restricted Application added subscribers: Liuxinyu970226, Aklapper. · View Herald TranscriptApr 12 2019, 9:19 PM

PR #1103 is the current work-in-progress patch for the back-end service's config.

Change 518871 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/restbase/deploy@master] Update config to use new projects and the proxy.

https://gerrit.wikimedia.org/r/518871

Change 518871 merged by Ppchelko:
[mediawiki/services/restbase/deploy@master] Update config to use new projects and the proxy.

https://gerrit.wikimedia.org/r/518871

Change 519150 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/restbase/deploy@master] Remove reference for enwiki project for beta.

https://gerrit.wikimedia.org/r/519150

Change 519150 merged by Ppchelko:
[mediawiki/services/restbase/deploy@master] Remove reference for enwiki project for beta.

https://gerrit.wikimedia.org/r/519150

Mentioned in SAL (#wikimedia-operations) [2019-06-26T15:46:55Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@995bc9d]: Use new projects and new config layout T220855, canaries only

Mentioned in SAL (#wikimedia-operations) [2019-06-26T15:50:26Z] <ppchelko@deploy1001> deploy aborted: Use new projects and new config layout T220855, canaries only (duration: 03m 31s)

Mentioned in SAL (#wikimedia-operations) [2019-06-27T17:23:35Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@da50001]: Use new projects and new config layout T220855, rb2009 only

Mentioned in SAL (#wikimedia-operations) [2019-06-27T17:26:13Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@da50001]: Use new projects and new config layout T220855, rb2009 only (duration: 02m 38s)

Change 519465 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/restbase/deploy@master] Config: Fix wm.org options indentation

https://gerrit.wikimedia.org/r/519465

Change 519465 merged by Ppchelko:
[mediawiki/services/restbase/deploy@master] Config: Fix wm.org sys hierarchy

https://gerrit.wikimedia.org/r/519465

Change 519469 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/restbase/deploy@master] Config: Add the skip_updates config to wm.org's sys hierarchy

https://gerrit.wikimedia.org/r/519469

Change 519469 merged by Ppchelko:
[mediawiki/services/restbase/deploy@master] Config: Add the skip_updates config to wm.org's sys hierarchy

https://gerrit.wikimedia.org/r/519469

Mentioned in SAL (#wikimedia-operations) [2019-06-27T17:59:11Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@ff6f302]: Use new projects and new config layout T220855, rb2009 only, fixed mathoid config

Mentioned in SAL (#wikimedia-operations) [2019-06-27T18:01:30Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@ff6f302]: Use new projects and new config layout T220855, rb2009 only, fixed mathoid config (duration: 02m 19s)

Mentioned in SAL (#wikimedia-operations) [2019-06-27T18:03:24Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@ff6f302]: Use new projects and new config layout T220855, restbase1016

Mentioned in SAL (#wikimedia-operations) [2019-06-27T18:03:32Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@ff6f302]: Use new projects and new config layout T220855, restbase1016 (duration: 00m 08s)

Mentioned in SAL (#wikimedia-operations) [2019-06-27T18:05:04Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@ff6f302]: Use new projects and new config layout T220855, restbase1016

Mentioned in SAL (#wikimedia-operations) [2019-06-27T18:06:42Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@ff6f302]: Use new projects and new config layout T220855, restbase1016 (duration: 01m 41s)

Mentioned in SAL (#wikimedia-operations) [2019-06-27T19:04:51Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@ff6f302]: Use new projects and new config layout T220855

Mentioned in SAL (#wikimedia-operations) [2019-06-27T19:16:12Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@ff6f302]: Use new projects and new config layout T220855 (duration: 11m 21s)