| MSantos | |
| Oct 16 2023, 2:38 PM |
| F41521967: image.png | |
| Nov 20 2023, 2:20 PM |
| Restricted File | |
| Nov 3 2023, 7:20 PM |
In order to remove PCS from restbase, PCS needs to manage its own cache.
This is not about pre-generation, this will be covered in a separate task
The cache layer should connect to the same Cassandra cluster used by restbase
What's the work needed in the infrastructure side?
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Introduce server side caching | mediawiki/services/mobileapps | master | +137 -3 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Stalled | None | T324931 Clean up open RESTBase related tickets | |||
| In Progress | None | T262315 <CORE TECHNOLOGY> API Migration & RESTbase Sunset | |||
| Stalled | Dbrant | T328943 Replace PCS lazy-loading logic with standard "loading=lazy" attribute | |||
| Open | None | T314025 [EPIC] Migrate PCS service away from restbase | |||
| Open | Jgiannelos | T319365 PCS caching and pregeneration when restbase is decommissioned | |||
| Resolved | Jgiannelos | T348995 Introduce PCS cache management layer | |||
| Open | hnowlan | T350507 Update mobileapps k8s deployment chart for Cassandra credentials |
Initial patch here after bootstrapping the nodejs env and gitlab CI:
https://gitlab.wikimedia.org/repos/content-transform/nodejs-cassandra-storage/-/merge_requests/1
@Eevans will you be available to take a look at this patch, mostly for the cassandra specific changes?
I added an item to the acceptance criteria about having a strategy for migration that works within the capacity constraints of the cluster. Hopefully this is the right issue for this, let me know if that's not the case and I can move it elsewhere (including to a separate ticket).
I'm going to be taking a closer look at utilization, as I have some reason to think it might be artificially high, but it's really doubtful that we'll have a enough to store all of PCS's cache twice (once for the old system, and once for the new), and maintain a responsible amount of overhead. I don't know if that means that we migrate by group (wikipedia, enwiki, etc), or by table (mobile_html, page_summary, etc), or via some combination of both (or whether or not either would be prohibitively difficult). Or perhaps there are other RESTBase use-cases that are near enough to be removed that this won't be an issue?
{F41439373}
@Eevans i cant see the file you attached in your comment. If there is a concern for storage we can always switchover gradually by project.
That's strange, it's an image of storage utilization on the cluster. I can see it inline, and click to view it full page. I wonder what happens if I attach the same image again?
If there is a concern for storage we can always switchover gradually by project.
By project would work.
Change 985166 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):
[mediawiki/services/mobileapps@master] Introduce server side caching
Change 985166 merged by jenkins-bot:
[mediawiki/services/mobileapps@master] Introduce server side caching