Change Details

Before we can switch VE to using Parsoid in MW (rather than going through RESTbase), we need to ensure that parsoid output for the page is present in the ParserCache. This may already be happening, because RESTbase is asking for rendered output afte revery edit, but we shold not rely on that. We should instead shedule jobs to render pages when their content changed. NOTE: To avoid overloading the Jobrunner cluster, the cache warming jobs should probably be routed to the Parsoid cluster. Steps (as per @Joe's suggestion) [] Increase the memory limit for the jobrunner cluster in mediawiki-config [] Move a few servers from the parsoid cluster to the jobrunner cluster (in puppet) [] Enable the jobs for wikis in batches, with SRE assistance. Possibly move more parsoid nodes to jobrunners if needed [] For each batch, configure restbase to disable the caching for that wiki - basically only send out the purges for its urls. [] rinse, repeat until nothing is left At this point, the scope of this specific task is complete. The remainder is here just to complete the list. Decide how we'll send the purges to the CDN for the restbase urls. My favourite option if we don't want to set up a complete system now would be to generate the purges when the warmup job is completed. [] Stop sending requests to restbase from change-propagation for parsoid. [] Move the requests for parsoid from all sources to bypass restbase (via the api gateway restbase compatibility layer) [] Kill the restbase storage for parsoid Alternative steps (elegant but riskier) Basically instead of point 1-4 of the previous procedure: [] Make changeprop handle these jobs separately. [] Either submit them to the parsoid cluster as jobs (this requires some SRE work) or (preferred) just calling the URL for the rendering of the page directly (this requires probably some work on changeprop) [] Once the parsercache is sufficiently warm, disable restbase caching for a batch of wikis, only keeping the purges emitted.

#### What? Before we can switch VE to using Parsoid in MW (rather than going through RESTbase), we need to ensure that parsoid output for the page is present in the ParserCache. This may already be happening, because RESTbase is asking for rendered output afte revery edit, but we shold not rely on that. We should instead shedule jobs to render pages when their content changed. NOTE: To avoid overloading the Jobrunner cluster, the cache warming jobs should probably be routed to the Parsoid cluster. #### How? Per @Joe's suggestion: [] Increase the memory limit for the jobrunner cluster in mediawiki-config [] Move a few servers from the parsoid cluster to the jobrunner cluster (in puppet) [] Enable the jobs for wikis in batches, with SRE assistance. Possibly move more parsoid nodes to jobrunners if needed [] For each batch, configure restbase to disable the caching for that wiki - basically only send out the purges for its urls. [] rinse, repeat until nothing is left At this point, the scope of this specific task is complete. The remainder is here just to complete the list. Decide how we'll send the purges to the CDN for the restbase urls. My favourite option if we don't want to set up a complete system now would be to generate the purges when the warmup job is completed. [] Stop sending requests to restbase from change-propagation for parsoid. [] Move the requests for parsoid from all sources to bypass restbase (via the api gateway restbase compatibility layer) [] Kill the restbase storage for parsoid Alternative steps (elegant but riskier) Basically instead of point 1-4 of the previous procedure: [] Make changeprop handle these jobs separately. [] Either submit them to the parsoid cluster as jobs (this requires some SRE work) or (preferred) just calling the URL for the rendering of the page directly (this requires probably some work on changeprop) [] Once the parsercache is sufficiently warm, disable restbase caching for a batch of wikis, only keeping the purges emitted.

#### What? Before we can switch VE to using Parsoid in MW (rather than going through RESTbase), we need to ensure that parsoid output for the page is present in the ParserCache. This may already be happening, because RESTbase is asking for rendered output afte revery edit, but we shold not rely on that. We should instead shedule jobs to render pages when their content changed. NOTE: To avoid overloading the Jobrunner cluster, the cache warming jobs should probably be routed to the Parsoid cluster. Steps (as p#### How? Per @Joe's suggestion): [] Increase the memory limit for the jobrunner cluster in mediawiki-config [] Move a few servers from the parsoid cluster to the jobrunner cluster (in puppet) [] Enable the jobs for wikis in batches, with SRE assistance. Possibly move more parsoid nodes to jobrunners if needed [] For each batch, configure restbase to disable the caching for that wiki - basically only send out the purges for its urls. [] rinse, repeat until nothing is left At this point, the scope of this specific task is complete. The remainder is here just to complete the list. Decide how we'll send the purges to the CDN for the restbase urls. My favourite option if we don't want to set up a complete system now would be to generate the purges when the warmup job is completed. [] Stop sending requests to restbase from change-propagation for parsoid. [] Move the requests for parsoid from all sources to bypass restbase (via the api gateway restbase compatibility layer) [] Kill the restbase storage for parsoid Alternative steps (elegant but riskier) Basically instead of point 1-4 of the previous procedure: [] Make changeprop handle these jobs separately. [] Either submit them to the parsoid cluster as jobs (this requires some SRE work) or (preferred) just calling the URL for the rendering of the page directly (this requires probably some work on changeprop) [] Once the parsercache is sufficiently warm, disable restbase caching for a batch of wikis, only keeping the purges emitted.