Change 589674 merged by Krinkle:
[operations/mediawiki-config@master] Enable LCStoreStaticArray on depooled mw1407 for benchmarking

https://gerrit.wikimedia.org/r/589674

In T99740#6031681, @Joe wrote:

Specifically, I want to run tests on wiki pages and possibly shadowing real traffic.

I've depooled mw1407 and enabled LCStoreStaticArray in MediaWiki config there with host-specific condition (https://gerrit.wikimedia.org/r/589674). I don't want to disable puppet for several days there so whenever you're ready, apply https://gerrit.wikimedia.org/r/587299 via puppet or disable puppet and apply by hand in /etc/php/7.2/fpm/php.ini.

Confirmed via a local request and Logstash:

krinkle@mw1407:~$ curl 'http://mw1407.eqiad.wmnet/w/load.php' -H 'Host: aa.wikipedia.org' -H 'X-Wikimedia-Debug: 1; log'

type:mediawiki host:mw1407 message:"LocalisationCache"

* DEBUG | LocalisationCache using store LCStoreStaticArray

.. which for all other hosts logs LocalisationCache using store LCStoreCDB.

Mentioned in SAL (#wikimedia-operations) [2020-04-27T11:20:59Z] <_joe_> restarted php-fpm on mw1407 to pick up enlarged opcache values, T99740

Mentioned in SAL (#wikimedia-operations) [2020-04-27T13:30:11Z] <_joe_> depooled mw1409 as well as mw1407 for further benchmarking, T99740

Mentioned in SAL (#wikimedia-operations) [2020-04-27T13:38:14Z] <_joe_> repooling both mw1407 and mw1409 for tesing T99740

After running a few benchmarks on mw1407 (where LCStoreStaticArray is used) vs mw1409 (which uses cdb files), it seemed the change made little to no difference for the following urls:

• https://it.wikipedia.org/wiki/Nemico_pubblico_(film_1998)
• https://en.wikipedia.org/wiki/Main_Page
• https://en.wikipedia.org/wiki/Barack_Obama
• https://kk.wikipedia.org/w/load.php?debug=false&lang=kk&modules=ext.3d.styles%7Cext.cite.styles%7Cext.uls.interlanguage%7Cext.visualEditor.desktopArticleTarget.noscript%7Cext.wikimediaBadges%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.page.gallery.styles%7Cmediawiki.skinning.interface%7Cmediawiki.toc.styles%7Cskins.vector.styles%7Cwikibase.client.init&only=styles&skin=vector
• https://en.wikipedia.org/w/api.php?action=parse&text={{:Australia}}

And I didn't find any significant difference in performance, so I decided to take a look at the performance of the server inside the serving pool for ~ 15 minutes. In this case, we see an improvement in performance between 1 and 5%.

Personally, if there isn't an external factor (as: better maintenance, ability to run static analysis on the code, etc), I don't consider this small performance gain to justify the changes we'd need to perform.

Namely, we will need to ensure that every full sync of the code causes a full fleet restart of php-fpm, as the 3 gb of memory we reserve for php-fpm are not enough to contain the code twice over.

Intersingly, other effect noticed:

• CPU usage is slightly higher when using LCStoreArray
• overall Memory usage is higher with LCStoreArray, but not significantly enough to be a worry in our current setup. Every php worker uses ~ 1 GB of memory at startup vs ~ 500k in the normal setup.

I think 5% is huge! As part of T233886 and T189966, I took many work-days to achieve similar gains, and even that is becoming harder and harder without it turning into weeks of multi-person/cross-team dependencies. These kinds of gain will decide how much work it is going to take to achieve certain consistent latencies on the new REST api for example, and also make latencies generally more consistent.

The l10n store win isn't a one-time cost difference, but it scales with call frequency (T99740#5929577). This is among the reasons why on cold cache performance varies so wildely because we have many layers of caching on top of l10n store. LocalisationCache is global per language, then MessageCache is per-wiki per-language (based on hooks and on-wiki overrides), and then there is MessageBlobStore for ResourceLoader on a per-module basis. Until recently MessageBlobStore had a dedicated DB table and nightly clean up cron. This might have been needed as much if base l10n performed better. Yet even with MessageBlobStore, the cache-miss experience is still pretty bad. The first cache miss for any module, in any language, on any wiki (1001 * 419 * 944), can currently takes upwards of 5 seconds to compute on-demand. And that's on an unstressed server. For a single JS resource. Not to mention end-user latency, HTML/wikitext processing, CSS, .. page load performance is lost before it even begins.

Aside from performance, this change also has benefits for the deployment process:

• Generating the array files is about 4X faster than building the CDB binaries (4 min vs 1 min). – T218207#5138303
• The arrays are about 10% smaller than CDB in terms of uncompressed file size (e.g. 3.7M per file instead of of 4.2M) . – T99740#4609279
• When compressed as part of an image, they can presumably be much smaller still.

What we do today:

• Scap invokes MediaWiki to gather the hundreds of i18n JSON files from core and extensions, and builds a set of large CDB binaries, one per language.
• Scap converts CDB binaries to JSON files, with MD5 checksum files alongside them.
• Scap rscyncs the JSON files (diff-only, compressible).
• Scap instructs each app server to convert the JSON files back to CDB binaries, and verifies their MD5 checksum.

What we'll do instead:

• Scap invokes MediaWiki to build a set of PHP array files.
• Scap rsyncs the array files (diff-only, compressible).

I would expect deployments to be faster, with simpler tools. And for container images in the future to therefore be smaller and take less complexity/time to build.

In T99740#6085180, @Joe wrote:

• overall Memory usage is higher with LCStoreArray, but not significantly enough to be a worry in our current setup. Every php worker uses ~ 1 GB of memory at startup vs ~ 500k in the normal setup.

We might be able to bring this down a bit. The opcache config I staged was optimised for benchmarking latency, not memory. I rounded the number up significantly to make sure it would definitely use opcache and not fallback to re-parsing disk reads. But, I don't know if all of that allocated space is actually needed. See https://gerrit.wikimedia.org/r/587299 and T99740#5977799.

I concur with Timo that this change does seem worthwhile from our perspective.

In T99740#6085845, @Krinkle wrote:

I think 5% is huge! As part of T233886 and T189966, I took many work-days to achieve similar gains, and even that is becoming harder and harder without it turning into weeks of multi-person/cross-team dependencies. These kinds of gain will decide how much work it is going to take to achieve certain consistent latencies on the new REST api for example, and also make latencies generally more consistent.

Let me note that the difference is well below 2% in most cases, and that it's much smaller than the variations in backend response times we see daily due to a variety of other effects, which can be in the range 5-10% or higher.

Moreover: a simple badly optimized database query costs us a 20% of performance for hours, easily, in terms of backend response times. And it doesn't require us to radically change our production environment.

I will run more extensive tests so I have more precise results, but in terms of performance evaluation, this gain is barely noticeable over a full day, and well below what we would consider significant.
It also makes the memory usage of the whole process about 3x what it is in normal operating conditions, and raises the CPU usage.

From the point of view of overall backend performance (that is what I'm talking about) this is a third-order optimization, notwithstanding whatever your perception of it is. Also, those figures above are quite un-scientific - I thought the result was lackluster enough not to justify further analysis. I'll post more precise numbers, testing over a full day on two servers restarted in the same second.

But this is not even my main reason of worry ( coming below).

The l10n store win isn't a one-time cost difference, but it scales with call frequency (T99740#5929577). This is among the reasons why on cold cache performance varies so wildely because we have many layers of caching on top of l10n store. LocalisationCache is global per language, then MessageCache is per-wiki per-language (based on hooks and on-wiki overrides), and then there is MessageBlobStore for ResourceLoader on a per-module basis. Until recently MessageBlobStore had a dedicated DB table and nightly clean up cron. This might have been needed as much if base l10n performed better. Yet even with MessageBlobStore, the cache-miss experience is still pretty bad. The first cache miss for any module, in any language, on any wiki (1001 * 419 * 944), can currently takes upwards of 5 seconds to compute on-demand. And that's on an unstressed server. For a single JS resource. Not to mention end-user latency, HTML/wikitext processing, CSS, .. page load performance is lost before it even begins.

We can't completely revisit how we deploy software for such a tail gain. I find it hard to believe that those 5 seconds are completely due to CDB files. Is that the case?
If so, I'm sure there are ways to keep that time down without polluting a single php-fpm cache with 2.5 GB of additional php data.

Aside from performance, this change also has benefits for the deployment process:

[CUT]

I would expect deployments to be faster, with simpler tools. And for container images in the future to therefore be smaller and take less complexity/time to build.

I don't think the two statements above are correct. And these are my main worries.

For scap deploys, we'll need to perform a full rolling restart of all appservers for every non sync-file change, as a single train deploy can fill the opcache up easily on a server using 3 GB of opcache.

When I proposed to do a full rolling restart at every release, it was deemed unpractical, dangerous and basically refused by the release engineering and performance teams. Let me note that this might allow us to go back to not validating opcache, which would make deployments much more atomic :)

A single deploy (for the train, but probably for most SWATs too) will require a restart, making it significantly slower than it is now. A full safe rolling restart of our application servers might take up to 5 minutes or more.

As for container images:

• Why should they be smaller using php arrays isntead of cdb files? I would expect the opposite to be true (we're talking about compressed file sizes)
• Having a 3 GB overhead of RAM usage would kill our ability to run much smaller installations of php-fpm in parallel, and force us to run "fat pods", which is decidedly suboptimal - kubernetes doesn't like to have to allocate very large chunks of one server's memory. Also, we'd get less available workers per server, because of the 2.5 GB memory overhead.

Basically for every pod we'd need a 3 GB opcache space + 3 GB apcu space *even before* we try to allocate workers. It's a 50% increase (from 4 GB to 6 GB) of the baseline occupied memory.

More in general, php-fpm performs better (much, much more than 1%) when you can keep its concurrency low - so much that we've discussed running multiple php-fpms with smaller footprint on a physical appserver even before we have moved to kubernetes. So increasing the memory footprint of the single daemon seems dangerous.

In T99740#6085180, @Joe wrote:

• overall Memory usage is higher with LCStoreArray, but not significantly enough to be a worry in our current setup. Every php worker uses ~ 1 GB of memory at startup vs ~ 500k in the normal setup.

We might be able to bring this down a bit. The opcache config I staged was optimised for benchmarking latency, not memory. I rounded the number up significantly to make sure it would definitely use opcache and not fallback to re-parsing disk reads. But, I don't know if all of that allocated space is actually needed. See https://gerrit.wikimedia.org/r/587299 and T99740#5977799.

Also don't forget the 3 GB of opcache memory usage. I'll post more precise numbers in a followup.

If our problem is having a local, highly available cache of these data, we can explore other avenues, like storing those data into a local memcached on all servers, as we're thinking of installing it for other reasons. On one hand, that would make the cache be slower (possibly), but it will also allow the cache to be shared between php-fpm instances.

Anyways, given the gain seems significant, I'll run more precise tests today.

Mentioned in SAL (#wikimedia-operations) [2020-04-28T07:52:38Z] <_joe_> running benchmarks on mw1407 (LCStoreStaticArray) and mw1409 (LCStoreCDB) for T99740: restart php-fpm, pool for 5 minutes to warmup caches, then depool both servers.

Assuming we'll be ok with restarting php-fpm at every release, I reduced the amount of strings memory and opcache allocated on mw1407 from the values in the puppet patch. I am now using 300 MB of interned strings cache and 3.3 GB of opcache space. These figures can be reduced further probably.

I am now running the following tests:

1. restart the appserver, run traffic through it for 30 minutes, evaluate if there is any significant performance gain over the whole period.
2. run each test benchmark I listed before, in parallel on both servers (so that we can hope no external factors affect our results), with growing concurrency
3. Repeat the above tests when reducing further the opcache usage on mw1407 *and* turning off opcache revalidation completely

First, the results of the real traffic test. These are averages over 10 minutes, starting after 20 minutes of having both servers pooled. This is an attempt at smoothing out the effects of very slow queries at higher percentiles, that can be traffic dependent.

* The memory used by php-fpm is measured by running:

$ ps -eo rss,command | awk 'BEGIN {mem=0} {if (/php-fpm/) { mem+=$1 }} END {print mem}'

** The shared memory is calculated by running:

master_pid=$(ps -eo pid,ppid,command | awk '{if (/php-fpm\:/ && $2 == 1) {print $1}}'); sudo pmap $master_pid | awk '{if ($4 == "zero") {c+=$2}} END {print c * 1024}'

My conclusion is that, while clearly slightly faster than LCStoreCDB, LCStoreStaticArray requires way more resources in terms of RAM usage, in particular for the shared memory that reaches almost 10 GB per php-fpm pool.

If we can tailor those numbers down a bit (for instance, reducing the size of the APC pool, and by programmatically restarting php-fpm at each release, thus removing the need for revalidating the opcache) I think the difference could be reduced to a smaller, more manageable number. I'm still unconvinced the perf gain would make it worthwile.

Some more data:

Rendering the enwiki Barack Obama page, with concurrency of 25, gives this response time distribution (over 10k requests - so the p99 can be found at 9900 requests):

In this case, we don't see significant differences.

Same thing, for a lighter page (https://it.wikipedia.org/wiki/Nemico_pubblico_(film_1998))

In this case, we have more differences, still well below 1% in p99

Finally, what happens when we try to load a resource via load.php, with concurrency of 40:

as you can see, differences are negligible here too.

In T99740#6089198, @Joe wrote:

Some more data:

Rendering the enwiki Barack Obama page, with concurrency of 25, gives this response time distribution (over 10k requests - so the p99 can be found at 9900 requests):

In this case, we don't see significant differences.

Same thing, for a lighter page (https://it.wikipedia.org/wiki/Nemico_pubblico_(film_1998))

In this case, we have more differences, still well below 1% in p99

Finally, what happens when we try to load a resource via load.php, with concurrency of 40:

as you can see, differences are negligible here too.

One rather random note and feel free to correct me if I'm wrong. Articles themselves are not heavy in using l10n (unless in multilingual projects like commons/wikidata). Checking Special pages or action history might yield a different result.

In T99740#6089626, @Ladsgroup wrote:

In T99740#6089198, @Joe wrote:

Some more data:

Rendering the enwiki Barack Obama page, with concurrency of 25, gives this response time distribution (over 10k requests - so the p99 can be found at 9900 requests):

In this case, we don't see significant differences.

Same thing, for a lighter page (https://it.wikipedia.org/wiki/Nemico_pubblico_(film_1998))

In this case, we have more differences, still well below 1% in p99

Finally, what happens when we try to load a resource via load.php, with concurrency of 40:

as you can see, differences are negligible here too.

One rather random note and feel free to correct me if I'm wrong. Articles themselves are not heavy in using l10n (unless in multilingual projects like commons/wikidata). Checking Special pages or action history might yield a different result.

This is a valid point! I'll repeat the test on a special page. I focused on things that get requested the most (articles and load.php), but it makes sense we also try with a special page.

@Joe, I appreciate the effort you put in to evaluating this change! If you have the patience to put up with some more annoying kibitzing from me, I have a few questions :)

Hammering one or two pages may not be representative. The performance test should force MediaWiki to look up entries in the l10n cache at the same rate as production, and that may not happen if every request is a parser cache hit. (Timo, please correct me if I'm wrong.) It's the traffic test we ought to pay attention to.

In T99740#6088175, @Joe wrote:

metric	LCStoreStaticArray	LCStoreCDB	diff
RSS (MB)*	60.7	99	+39%

Shouldn't this be -39%?

My conclusion is that, while clearly slightly faster than LCStoreCDB, LCStoreStaticArray requires way more resources in terms of RAM usage, in particular for the shared memory that reaches almost 10 GB per php-fpm pool.

A 5% improvement in p50 page load time looks pretty significant to me.

If we can tailor those numbers down a bit (for instance, reducing the size of the APC pool, and by programmatically restarting php-fpm at each release, thus removing the need for revalidating the opcache) I think the difference could be reduced to a smaller, more manageable number. I'm still unconvinced the perf gain would make it worthwile.

How do you weigh RAM vs. performance? How much RAM headroom do app servers have currently?

In T99740#6089095, @Joe wrote:

Just to be clearer: we achieved a much larger improvement in the average latency of requests by switching to persistent connections to our session storage:

I can't tell the effect size from this graph, and I'm not sure what point you're making. It's generally the case with performance tuning that over time you pay more for diminishing improvements, no? The opportunity cost for LCStoreStaticArray should be evaluated relative to other unrealized opportunities the Foundation could be pursuing with the same resources. If there are lower-hanging fruit, what are they?

This is a valid point! I'll repeat the test on a special page. I focused on things that get requested the most (articles and load.php), but it makes sense we also try with a special page.

Special pages with many messages, apart from Special:AllMessages of course, might be Special:Tags, Special:Gadgets and Special:Version. Pages with few messages repeated many times could be Special:SiteMatrix or any query page with high limit.

On Special:Log and the like, there can be higher variance due to things like querying for user preferences (grammatical gender especially), if I remember well from some work Tim did on it years ago.

In T99740#6089198, @Joe wrote:

• Rendering the enwiki Barack Obama page, with concurrency of 25
• load a resource via load.php, with concurrency of 40

I haven't confirmed it, but I suspect most interface messages used during page views are behind other layers of caching. The skin sidebar and wikitext parser both involve a good numebr of messages, but both have dedicated caches. The higher percentiles don't resemble the cache-miss scenarios per se either as these may've been warmed up in Memcached prior to the benchmark, but even if not, there is enough variance in these URLs from other factors that the outliers are likely extremes from other code paths, not LCStore.

ResourceLoader is possibly the largest consumer of interface messages (in terms of how many it fetches per http request, given it has to bundle them upfront). It has a dedicated cache and uses it for all interface messages fetched from its code paths (MessageBlobStore).

An isolated bench on the LCStore calls was already done at T99740#5929577 (see the far end of that comment), and in other comments and on other tasks. But if we want to do this with higher concurrency and in production, I'd recommend patching MW locally to disable part of Memcached.

For example, the most frequently used URL for ResourceLoader is https://en.wikipedia.org/w/load.php?modules=startup&lang=en&skin=vector&only=scripts. This is requested from every pageview in production, and requires thousands of messages (every message of every module, to determine each module URLs's version key). Below is from mwdebug1001 with an opcache patch applied, for the load.php?startup url, with 1 warmup, and 3 runs:

This 20%-reduced traffic is representative of the traffic we get after a new branch or other major deployment where we get these requests for each wiki/language/skin/platform combination to backfill caches (mobile/desktop * wikis * languages * skins = 2*940*310*5 = 2.9M) .

@ php-1.35.0-wmf.28/includes/resourceloader/MessageBlobStore.php
- $result = $cache->getMulti( array_values( $cacheKeys ), $curTTLs, $checkKeys );
+ $result = []; # $cache->getMulti( array_values( $cacheKeys ), $curTTLs, $checkKeys );

@ wmf-config/CommonSettings.php
- $wgLocalisationCacheConf['storeClass'] = LCStoreCDB::class;
+ $wgLocalisationCacheConf['storeClass'] = LCStoreStaticArray::class;

@Joe I admit I don't have a good understanding of the RAM cost and overhead we have. It would help to better quantify the budget/buffer/cost here.

At the risk of bringing more bad news.. The memory limit for MW web requests is 600M currently. This is set to 1.4G on Parsoid servers, and I believe the current expectation is that we'd need to apply this to other app servers before Parsoid can be used within MW. Would that be similarly concerning? Or less concerning (given per-request)?

Mentioned in SAL (#wikimedia-operations) [2020-05-06T08:02:46Z] <_joe_> restarted php-fpm with tweaked parameters on mw1407, now briefly pooling for traffic (T99740)

In T99740#6100595, @ori wrote:

@Joe, I appreciate the effort you put in to evaluating this change! If you have the patience to put up with some more annoying kibitzing from me, I have a few questions :)

Hammering one or two pages may not be representative. The performance test should force MediaWiki to look up entries in the l10n cache at the same rate as production, and that may not happen if every request is a parser cache hit. (Timo, please correct me if I'm wrong.) It's the traffic test we ought to pay attention to.

That's why I also ran a test of re-parsing where I got no big difference either, but I do agree - that's why I also ran some tests with actual production traffic.

In T99740#6088175, @Joe wrote:

metric	LCStoreStaticArray	LCStoreCDB	diff
RSS (MB)*	60.7	99	+39%

Shouldn't this be -39%?

No, I twisted my fingers, it's using more memory, but it's not really relevant govien the actual numbers.

My conclusion is that, while clearly slightly faster than LCStoreCDB, LCStoreStaticArray requires way more resources in terms of RAM usage, in particular for the shared memory that reaches almost 10 GB per php-fpm pool.

A 5% improvement in p50 page load time looks pretty significant to me.

I don't think it is, given we have quite a few ongoing issues that cause the latency to spike up by more than 20% - for example, still unexplained surges in memcache request rate - for which we have no instrumentation nor - clearly - time for investigation.

See this for instance on mw1407 during a period where we had to re-pool it during a network outage that forced us to depool a whole rack of appservers:

The mean latency had serveral spikes of 10-25%, and a sustained plateau, in correspondence of spikes in memcached request rates:

So: I think there are other areas we should focus on first. I would agree that in a void where we've repaired all of our larger culprits this would be a good perf gain, even if I would still be doubtful about its costs.

If we can tailor those numbers down a bit (for instance, reducing the size of the APC pool, and by programmatically restarting php-fpm at each release, thus removing the need for revalidating the opcache) I think the difference could be reduced to a smaller, more manageable number. I'm still unconvinced the perf gain would make it worthwile.

How do you weigh RAM vs. performance? How much RAM headroom do app servers have currently?

One thing all of my current and previous benchmarking has shown consistently is that php-fpm's performance degrades under higher concurrency, no matter the amount of cpu / ram /disk we throw at the problem. There are fundamental bottlenecks in php-fpm that are less severe when it's serving 50 req/s instead of 150. Being able to run 3 separated php-fpm instances on a single physical server would improve both our scalability and our latencies.

So, it's not just about where they are currently, it's where we want to get. Having such a huge memory requirement (basically, 8-10 GB for apc/lcstaticarray, plus the N GB to serve requests) would practically kill our ability to run mediawiki within kubernetes, or even to just run multiple instances per machine if we choose not to get there for some reason.

I have some ideas on how to overcome this - basically using a shared cache instead of a per-instance one, see also T244340 and T248005. but that would beg the question - is that faster than using CDB when accessing the data itself?

Anyways, this can be worked on further. Sadly, I have other priorities at the moment - but I'm happy to come back to the discussion once I have time for it again.

Basically, my precondition for seeing this in production right now would be:

• Stop revalidating opcache (which seeems a good idea given the occasional corruptions we see anyways)
• Rolling restart php-fpm with every scap run (this is currently supported in scap, but needs to be tested)
• Set opcache to be what can contain one train release, not multiple ones like we do today
• Make checks *pre-deploy* to ensure we don't get over said limit.

For instance, I was able to pool mw1407 with the following configuration:

opcache.validate_timestamps = 0
opcache.interned_strings_buffer = 300
opcache.max_accelerated_files = 15000
opcache.max_wasted_percentage = 10
opcache.memory_consumption = 2000

This would mean a 1GB increase in our current opcache memory usage, and would not impair any of our plans for containerization either, as we can probably compensate that by reducing the size of APCu anyways.

Last time I proposed to not revalidate cache and just roll-restart php-fpm everywhere there was some resistance from Release Engineering folks - but I see that as the best way forward if you want to see this to production.

Change 592867 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Revert "Enable LCStoreStaticArray on depooled mw1407 for benchmarking"

https://gerrit.wikimedia.org/r/592867

Change 592867 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Enable LCStoreStaticArray on depooled mw1407 for benchmarking"

https://gerrit.wikimedia.org/r/592867

Mentioned in SAL (#wikimedia-operations) [2020-05-06T08:43:54Z] <oblivian@deploy1001> Synchronized wmf-config/CommonSettings.php: Reverting change on mw1407 T99740 (duration: 01m 16s)

Change 630592 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[mediawiki/tools/scap@master] Feature flag PHP L10n generation

https://gerrit.wikimedia.org/r/630592

Change 630592 merged by jenkins-bot:
[mediawiki/tools/scap@master] Feature flag PHP L10n generation

https://gerrit.wikimedia.org/r/630592

Change 651228 had a related patch set uploaded (by Ahmon Dancy; owner: Ahmon Dancy):
[operations/mediawiki-config@master] Disable PHP L10n in beta cluster

https://gerrit.wikimedia.org/r/651228

Change 651228 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable PHP L10n in beta cluster

https://gerrit.wikimedia.org/r/651228

Change 883707 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/mediawiki-config@master] Revert "Disable PHP L10n in beta cluster"

https://gerrit.wikimedia.org/r/883707

This is somewhat important for mw-on-k8s so even if the perf gains are small, it's still a good cause to see done.

In T99740#8559738, @Ladsgroup wrote:

This is somewhat important for mw-on-k8s so even if the perf gains are small, it's still a good cause to see done.

Can you expand on what makes this important for mw-on-k8s?

I can't find it but I had it somewhere that serviceops mentioned this, in one of their slides as part of mw-on-k8s challenges because cdb files are too big for images and harder to maintain and build. Can't find it though :/

From the point of view of image size and memory usage, it might be better to merge all the languages into a single file, and combine identical values by having a numerically indexed global value array. Maybe also abbreviate the message keys by mapping them to integers. We can tolerate a little bit of indirection at runtime.

In T99740#6111784, @Joe wrote:

Basically, my precondition for seeing this in production right now would be:

5 years later a number of these are now taken care of as side effect of other improvements in the hosting stack.

• Stop revalidating opcache (which seeems a good idea given the occasional corruptions we see anyways)
• Rolling restart php-fpm with every scap run (this is currently supported in scap, but needs to be tested)

MediaWiki and php-fpm now live inside an OCI container as part of a Kubernetes Pod. Code only changes via container replacement which means the opcache doesn't need to check for on-disk updates and we do get a fresh php-fpm runtime environment for each deployment.

• Set opcache to be what can contain one train release, not multiple ones like we do today

The Pretrain project (T369112) builds single MediaWiki version PHP containers. That might make it a useful test platform for future experiments in replacing CDB with something else. We do not yet have a functional deployment of Pretrain, so it is not an immediately available operational platform for new experiments, but it is likely to become available in the next 4-6 months. I have a hope that one of the projects to follow the initial Pretrain deployment will be single MediaWiki version containers for the other wikis as well.

• Make checks *pre-deploy* to ensure we don't get over said limit.

A validation of opcache utilization sounds like something we could work into the deployment process. I'm not exactly sure what we would need to do if the limit validation failed, but I'm sure that could be worked out.

In T99740#11352349, @tstarling wrote:

From the point of view of image size and memory usage, it might be better to merge all the languages into a single file, and combine identical values by having a numerically indexed global value array. Maybe also abbreviate the message keys by mapping them to integers. We can tolerate a little bit of indirection at runtime.

I did a brief experiment in this vein; a quick indirection level still using CDBs reduced my l10n cache dir from 850M to 475M, and compressing serialized message payloads with deflate got it down to 300M. Using a lower-level index over the id numbers vs a hashmap for the id->id and id->string lookups should save further space.

[Update] Yeah, using binary packed arrays instead of CDB for the id->id and id->string mappings gets it down to like 140M with the smaller indexes and deflate compression for the payloads. This is a 5x reduction in file size that has to be shuffled around for _relatively_ little added complexity, which feels like a good start.

Container deploys might also benefit from breaking up the common strings table into multiple files, where we have an "old" common file that is retained as-is (and doesn't have to be re-sent each deploy) and a "new" file that collects any new strings not in the old common file.

https://phabricator.wikimedia.org/T410694#11398762

Very happy with the results of my holiday hacking: T410694#11418011

• dropped the exotic binary file experiments, the opcode cache just performs too much better with pre-instantiated strings and arrays
• implemented read-time fallback processing and message key string indirection to massively reduce on-disk file size and opcache size of cached arrays
• removed the duplicate list of message keys since it can be read from a common list

I think this is quite promising, and with some cleanup would be worth considering for some testing in a more realistic environment.

Change #1212717 had a related patch set uploaded (by Krinkle; author: Bvibber):

[mediawiki/core@master] Language: shrink LCStoreStaticArray by doing fallback at read time

https://gerrit.wikimedia.org/r/1212717

Change #1212717 merged by jenkins-bot:

[mediawiki/core@master] Language: shrink LCStoreStaticArray by doing fallback at read time

https://gerrit.wikimedia.org/r/1212717

Change #1219190 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/puppet@production] scap: Add php_l10n build in Beta Cluster

https://gerrit.wikimedia.org/r/1219190

Change #1219190 merged by Ori:

[operations/puppet@production] scap: Add php_l10n build in Beta Cluster

https://gerrit.wikimedia.org/r/1219190

Change #1219155 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] Language: minor tweak and simplify in LCStoreStaticArray

https://gerrit.wikimedia.org/r/1219155