Page MenuHomePhabricator

Disable Parsoid cache updates for wikis that are migrated to RESTBase
Closed, ResolvedPublic


We currently process cache updates for both RESTBase and Parsoid, but this is causing higher load on the Parsoid cluster than necessary. While VisualEditor is already using RESTBase for all wikipedias, there are still some users that rely on cached content from the Parsoid v1 API:

  • OCG
  • Content Translation

Kiwix is also still using the Parsoid API, but is not particularly latency-sensitive.

Event Timeline

GWicke raised the priority of this task from to Medium.
GWicke updated the task description. (Show Details)
GWicke added projects: RESTBase, Parsoid.
GWicke added a subscriber: GWicke.

I have still a few dumping processes running against the old API and in particular a WPEN dump I really don't want to have stopped in the middle. I don't know exactly what kind of impact this ticket might have for us, but if there is an important one, I would appreciate, if this can be postponed of around 10 days.

@Kelson, you might actually end up finishing your enwiki dump earlier if you switched to RB right now, even if you start that dump from scratch. Using a single instance of htmldumper with default settings (concurrency 50) the enwiki HTML should dump in well under a day, even with a remote connection.

In any case, the Parsoid V1 API will remain available for quite some time. We will however stop keeping its cache up to date soon, so dumps using it will become slower and will add more load to the Parsoid cluster as the percentage of cache misses will be higher.

I hear that mwoffliner is already converted to use RESTBase, so now it's only OCG and CX left to convert before we can stop updating the Parsoid caches.

Just a small reminder: is only available for wikipedia. We still rely pretty much on to create ZIM snapshots of all other WM projects.

Notice: I am aiming to switch off the old cache update jobs for wikipedias within the next two weeks.

Quick update: We are almost there. OCG is pending a deploy to the PHP extension, and cxserver might already be switched over.

Any agenda to migrate the other projects to

cxserver migration is pending merging/deployment
But should happen with in next 2 weeks atmost.

@santhosh, thank you! That was merged now, so it seems that only OCG is left at this point.

@Kelson, the remaining public / non-special projects will be supported once gets merged, which I hope will happen tomorrow.

Change 217434 had a related patch set uploaded (by GWicke):
Disable Parsoid cache updates

Change 217434 merged by jenkins-bot:
Disable Parsoid cache updates

Now live:

pasted_file (328×577 px, 31 KB)

Next steps:

Change 217550 had a related patch set uploaded (by GWicke):
Remove Parsoid job runners

Scheduled the removal of Parsoid job runners ( ) with @Joe for Monday, 2015/06/15

Change 217550 merged by Ori.livneh:
Remove Parsoid job runners

GWicke claimed this task.