Page MenuHomePhabricator

Job for pre-emptive population of the ParserCache with Parsoid output
Closed, ResolvedPublic

Description

When we need to have parsoid output in the ParserCache for use by VE, but we don't want to use it for page views, then we should perform the pre-population on edit asynchronously.

Specifically, DerivedPageDataUpdater::doParsoidCacheUpdate should enqueue a job, rather than parsing directly.

Related Objects

StatusSubtypeAssignedTask
StalledNone
In ProgressNone
OpenNone
OpenNone
OpenNone
ResolvedJgiannelos
DeclinedNone
Resolvedhnowlan
In ProgressNone
Resolveddaniel
ResolvedNone
ResolvedNone
Resolvedmatmarex
ResolvedPRODUCTION ERRORdaniel
Resolveddaniel
DuplicatexSavitar
Resolveddaniel
Resolveddaniel
Resolveddaniel
ResolvedKCVelaga_WMF
Resolved ssastry
Resolveddaniel
OpenMSantos
OpenNone
OpenNone
ResolvedROdonnell-WMF
OpenBUG REPORTNone
OpenBUG REPORTNone
OpenBUG REPORTNone
OpenBUG REPORTNone
OpenNone
Resolveddaniel
Resolveddaniel
Resolveddaniel
Resolveddaniel
OpenNone
OpenNone
OpenNone
In ProgressNone
OpenNone
Resolveddaniel
Resolveddaniel
Resolveddaniel
Resolveddaniel
Resolveddaniel
ResolvedClement_Goubert
Resolveddaniel

Event Timeline

Please note that, both for this task and its duplicate, deployment in production should be coordinated with SRE as it will most likely need:

  1. a special configuration on kafka/changeprop, possibly
  2. moving of resources from the "parsoid" cluster (does it even make sense to have it separated anymore?) to the jobrunner cluster
  3. Raising memory limits for the jobrunner cluster

All of the above require time and coordination.

Change 860567 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@master] jobqueue: Warm parsoid parser cache using a job

https://gerrit.wikimedia.org/r/860567

On top of what Joe has said, Make sure it's disabled in wikidata and commons.

daniel added a subscriber: xSavitar.

Change 860567 merged by jenkins-bot:

[mediawiki/core@master] jobqueue: Warm parsoid parser cache using a job

https://gerrit.wikimedia.org/r/860567

On top of what Joe has said, Make sure it's disabled in wikidata and commons.

It is. The code is behind a switch (ParsoidCacheConfig->WarmParsoidParserCache). We just made it async when turned on.

InitializeSettings has:

17406-'wgParsoidCacheConfig' => [
17407-  'default' => [
17408-          'StashType' => null, // defaults to using MainStash
17409-          'StashDuration' => 24 * 60 * 60, // 24h in production, VE will fail to save after this time.
17410-          'CacheThresholdTime' => 0.0, // 0 means cache all
17411:          'WarmParsoidParserCache' => false, // whether parsoid output should be generated and cached after every edit
17412-  ],
17413-  'testwiki' => [
17414-          'StashType' => null, // defaults to using MainStash
17415-          'StashDuration' => 24 * 60 * 60, // 24h in production, VE will fail to save after this time.
17416-          'CacheThresholdTime' => 0.0, // 0 means cache all
17417:          'WarmParsoidParserCache' => true, // T320535: enable cache warming
17418-  ],
17419-  'mediawikiwiki' => [
17420-          'StashType' => null, // defaults to using MainStash
17421-          'StashDuration' => 24 * 60 * 60, // 24h in production, VE will fail to save after this time.
17422-          'CacheThresholdTime' => 0.0, // 0 means cache all
17423:          'WarmParsoidParserCache' => true, // T320534: enable cache warming
17424-  ],
17425-],
daniel claimed this task.