We should stop pregenerating /page/definitions responses to reduce the load on mobileapps. This should be safe to do given that the external request rate is very low and the current external vs. internal latencies are not too dissimilar.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Mholloway | T229286 Resolve service instability due to excessive event loop blockage since starting PCS response pregeneration | |||
| Resolved | • mobrovac | T231361 Stop pregenerating and storing /page/definition responses |
Change 534122 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/change-propagation/deploy@master] Do not pregenerate wiktionary definitions
Gerrit 534122 and PR 1181 address the CP and RB side, respectively. When deploying, pregeneration should be stopped first and quickly followed by a RB deploy so as to minimise the impact on both MCS and data consistency.
Mmm... The two patches completely neglect the long cache max-age we're setting - both patches remove all the purging. We need to first figure out what are we going to do with it? If we still wanna long cache with purges - we need to modify the CP patch. If we wanna shorten cache and have no purges - need to change RB patch. I personally thing we should try having long varnish caching with purging and no storage to see how this approach affects real life latencies.
Change 534122 merged by Mobrovac:
[mediawiki/services/change-propagation/deploy@master] Do not pregenerate wiktionary definitions, only purge it
Mentioned in SAL (#wikimedia-operations) [2019-09-11T08:19:28Z] <mobrovac@deploy1001> Started deploy [changeprop/deploy@56a8342]: Stop pregenerating enwiktionary page/definition - T231361
Mentioned in SAL (#wikimedia-operations) [2019-09-11T08:22:13Z] <mobrovac@deploy1001> Finished deploy [changeprop/deploy@56a8342]: Stop pregenerating enwiktionary page/definition - T231361 (duration: 02m 45s)
Change 535794 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/change-propagation/deploy@master] Config: Remove enwiktionary purging left-overs
Change 535794 merged by Mobrovac:
[mediawiki/services/change-propagation/deploy@master] Config: Remove enwiktionary purging left-overs
Mentioned in SAL (#wikimedia-operations) [2019-09-11T08:34:45Z] <mobrovac@deploy1001> Started deploy [changeprop/deploy@7a8ab89]: Stop pregenerating enwiktionary page/definition, take #2 - T231361
Mentioned in SAL (#wikimedia-operations) [2019-09-11T08:36:58Z] <mobrovac@deploy1001> Finished deploy [changeprop/deploy@7a8ab89]: Stop pregenerating enwiktionary page/definition, take #2 - T231361 (duration: 02m 13s)
Mentioned in SAL (#wikimedia-operations) [2019-09-11T09:08:32Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints - T231361 T232449
Mentioned in SAL (#wikimedia-operations) [2019-09-11T09:11:55Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints - T231361 T232449 (duration: 03m 24s)
Mentioned in SAL (#wikimedia-operations) [2019-09-11T10:57:24Z] <mobrovac> drop the wiktionary definition keyspace - T231361