| • GWicke | |
| Mar 19 2015, 10:49 PM |
| F177964: pasted_file | |
| Jun 11 2015, 4:29 PM |
We currently process cache updates for both RESTBase and Parsoid, but this is causing higher load on the Parsoid cluster than necessary. While VisualEditor is already using RESTBase for all wikipedias, there are still some users that rely on cached content from the Parsoid v1 API:
Kiwix is also still using the Parsoid API, but is not particularly latency-sensitive.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Remove Parsoid job runners | operations/puppet | production | +1 -15 | |
| Disable Parsoid cache updates | operations/mediawiki-config | master | +0 -24 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • GWicke | T93291 Disable Parsoid cache updates for wikis that are migrated to RESTBase | |||
| Resolved | cscott | T93292 Switch OCG to use https://rest.wikimedia.org/ | |||
| Resolved | • GWicke | T97840 rest.wikimedia.org needs to support commons | |||
| Resolved | • santhosh | T92359 Use Wikimedia REST API for accessing page data in Content Translation | |||
| Declined | None | T95199 Support body-only retrieval in normal HTML revision end points (restbase, body) | |||
| Resolved | • GWicke | T97393 Provide a way to know which revision was fetched while retrieving the latest html for a page title | |||
| Resolved | • GWicke | T93452 Enable group 1 wikis in RESTBase | |||
| Resolved | • GWicke | T94196 Thin out old revision renders |
I have still a few dumping processes running against the old API and in particular a WPEN dump I really don't want to have stopped in the middle. I don't know exactly what kind of impact this ticket might have for us, but if there is an important one, I would appreciate, if this can be postponed of around 10 days.
@Kelson, you might actually end up finishing your enwiki dump earlier if you switched to RB right now, even if you start that dump from scratch. Using a single instance of htmldumper with default settings (concurrency 50) the enwiki HTML should dump in well under a day, even with a remote connection.
In any case, the Parsoid V1 API will remain available for quite some time. We will however stop keeping its cache up to date soon, so dumps using it will become slower and will add more load to the Parsoid cluster as the percentage of cache misses will be higher.
I hear that mwoffliner is already converted to use RESTBase, so now it's only OCG and CX left to convert before we can stop updating the Parsoid caches.
Just a small reminder: rest.wikimedia.org is only available for wikipedia. We still rely pretty much on parsoid-lb.eqiad.wikimedia.org to create ZIM snapshots of all other WM projects.
Notice: I am aiming to switch off the old cache update jobs for wikipedias within the next two weeks.
Quick update: We are almost there. OCG is pending a deploy to the PHP extension, and cxserver might already be switched over.
cxserver migration is pending merging/deployment https://gerrit.wikimedia.org/r/#/c/207378/
But should happen with in next 2 weeks atmost.
@santhosh, thank you! That was merged now, so it seems that only OCG is left at this point.
@Kelson, the remaining public / non-special projects will be supported once https://gerrit.wikimedia.org/r/#/c/198433/ gets merged, which I hope will happen tomorrow.
All remaining public & active wikis are added in https://gerrit.wikimedia.org/r/#/c/217431/.
Change 217434 had a related patch set uploaded (by GWicke):
Disable Parsoid cache updates
https://gerrit.wikimedia.org/r/#/c/217431/ has been deployed, so https://gerrit.wikimedia.org/r/217434 can now be put up for SWAT.
Now live:
Next steps:
Change 217550 had a related patch set uploaded (by GWicke):
Remove Parsoid job runners
Scheduled the removal of Parsoid job runners ( https://gerrit.wikimedia.org/r/#/c/217550/ ) with @Joe for Monday, 2015/06/15