It would be useful to experiment with igbinary serialization (via apc.serializer INI setting) for apcu to see if it improves memory efficiency and CPU usage possibly as well. Any real deployment should wait on the php7 migration though.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T432486 FY2026-2027 ST6.4 Logged-in user performance (tracking) | |||
| Open | Krinkle | T432488 [Hypothesis] ST6.4.2 Increase MediaWiki backend capacity for logged-in pageviews (tracking) FY2026-2027 | |||
| Open | None | T302623 2022-2026: Improve Backend Pageview Timing | |||
| Open | MGoncalves-WMF | T225074 Evaluate using igbinary for MW php-apcu at WMF |
Event Timeline
Comment Actions
Looking around on "the Internet", sources say igbinary is by default slower than php by default, but when setting igbinary.compact_strings=Off it should be faster.
In production, php-igbinary is already provisioned on app servers and indeed with igbinary.compact_strings=Off set. But it looks to be unused. Neither apcu nor memc uses it.
I was unable to trace since when or for what it was first installed. Perhaps someone considered this in the past and then forgot or intentionally turned stopped using it? I honestly thought we were using it already, but in tracing the Puppet changes I could not find any past version where apc.serializer was set to igbinary.
Cross-linking, there is a newer idea that is likely even better: T288705: Experiment with 'default' apc.serializer string reuse
Comment Actions
APCu serializer evaluation: local results
We evaluated PHP's native APCu serializer against igbinary in a local MediaWiki Docker environment. The workload used Main_Page with different extensions enabled to exercise MediaWiki's page-rendering and cache paths, including AbuseFilter, Gadgets and VisualEditor, Scribunto and SyntaxHighlight. Python/Pygments was installed in the running container to support the external Pygmentize configuration.
For the controlled runs, we restarted the MediaWiki PHP-FPM service before each serializer run and verified that the cold APCu snapshot was empty. We then sent 20 warm-up requests and discarded their timing results, saved a warm APCu snapshot, and measured 100 requests with ApacheBench.
We completed three runs per serializer using the same workload, alternating the serializer order.
Results
Across all runs the arithmetic average of the per-run mean latency was PHP: 240.612 ms versus igbinary: 239.465 ms.The average throughput was PHP: 4.160 requests/second versus igbinary: 4.178 requests/second. The median APCu payload memory was approximately 29.8% less using igbinary, with PHP 546,952 bytes and igbinary 383,752 bytes. In order to see CPU regression, CPU usage was captured for two runs per serializer using the PHP container's cgroup CPU counter during the measured 100-request batch, a small difference is shown, for PHP 23.278 CPU-seconds versus igbinary 23.190 CPU-seconds.
| Metric (average across 3 runs) | PHP | igbinary | Difference |
| Requests per second | 4.103 | 4.127 | igbinary 0.6% higher |
| Mean latency | 243.750 ms | 242.387 ms | igbinary 1.363 ms lower |
| p50 latency | 241.667 ms | 241.333 ms | igbinary 0.334 ms lower |
| p90 latency | 252.333 ms | 249.667 ms | igbinary 2.666 ms lower |
| p95 latency | 256.667 ms | 255.000 ms | igbinary 1.667 ms lower |
| p99 latency | 268.333 ms | 267.000 ms | igbinary 1.333 ms lower |
| APCu payload memory after warm-up | 546,947 bytes | 383,752 bytes | igbinary 29.8% lower |
The full report files are available in APCU Evaluation
Next step
The local results show APCu payload memory reduction of ~29.8% with igbinary for the three runs. Page latency and throughput were ~2ms between serializers.The results are enough to justify a controlled Beta trial. For next steps we can enable igbinary for Beta test and monitor in logstash.
Comment Actions
@MGoncalves-WMF Two questions:
Is this with PHP 8.5, php-apcu 5.1.28, and php-igbinary 3.2.16 on Debian Bookworm? (i.e. latest MediaWiki-Docker dev/bookworm-php85-fpm:1.0.0)
How does the memory reduction come out with igbinary.compact_strings=Off?
The earliest trace I could find of php-igbinary in production was 2012 (https://gerrit.wikimedia.org/r/c/operations/puppet/+/29471) as part of installing native php-memcached (instead of the pure php MemcachedClient.php implementation). Shortly afterward, igbinary.compact_strings=Off was disabled (https://gerrit.wikimedia.org/r/c/operations/puppet/+/29892). Despite having little to no use of igbinary in prod, we've kept the package and this configuration all these years for performance reasons. Upstream documentation indeed says that while reads (igbinary unserialization) are faster than php serialization, writes (igbinary serialization) is slower unless you disable igbinary.compact_strings.
MediaWiki may have an unusually high rate of apcu writes, so we may want to keep it this way. It would make for an incremental way to get benefits with lower risk. On the other hand, the cost of experimentation, monitoring, provisioning, rollout etc may be more worthwhile if we try a more optimistic setup rather than a smaller change. We'll discuss this with SRE to decide on the experiment. Knowing the difference igbinary.compact_strings makes today with MediaWiki-Docker workloads would help inform this decision.
A lot has changed since 2012, so it's not clear whether this performance trade-off still holds merit today. Upstream issues like https://github.com/igbinary/igbinary/issues/10 suggest that by 2016, enough may have changed to no longer be worth turning off. MediaWiki itself has also reduced apcu writes (T315472), motivated by lock contention which has improved in PHP 8.5 (T435072).
Comment Actions
A few APCU read/write rates from https://grafana.wikimedia.org/d/U7JT--knk/mediawiki-on-k8s (APCU inserts, and APCU hits):
| mw-web (high) | 6:1 | 3.6M read/s | 0.6M write/s |  |
| mw-web (normal) | 7:1 | 1.7M read/s | 0.2M write/s |  |
| mw-api-ext | 2:1 | 180K read/s | 91K write/s |  |
| mw-api-int | 2:1 | 130K read/s | 58K write/s |  |
| mw-jobrunner | 3:1 | 31K read/s | 12K write/s |  |
I was hoping for at least 10:1, but we alas we still have a relatively high rate of write/serialize calls.
Comment Actions
@Krinkle
Yes, the comparison was run with the latest MediaWiki-Docker PHP-FPM image, for previous and actual runs:
- Debian GNU/Linux 12 (Bookworm)
- PHP 8.5.9
- APCu 5.1.28
- igbinary 3.2.16
Running new batches for igbinary with igbinary.compact_strings set to Off and On for each resulted:
- Off: 23.444 s, 24.344 s, 23.397 s
- On: 23.530 s, 23.811 s, 23.867 s
The medians are 23.444 s for Off and 23.811 s for On, around 1.6% difference. Although the previous 29.8% reduction compared 2 different serializers, we could see a difference for APCu payload memory between On and Off settings too. Off was 414,120 bytes and On 366,136 bytes, showing 11.6% reduction for igbinary with compact_strings=On against igbinary with compact_strings=Off.
Based on these, testing more optimistic approach in beta seems worthwhile if we could monitor the CPU/ latency /APCu memory for each (mw-web, mw-api..) before deciding.
Comment Actions
Mentioned in SAL (#wikimedia-releng) [2026-10-07T17:27:12Z] <obvious-dust> Cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/1352263 ref T225074
Comment Actions
Mentioned in SAL (#wikimedia-releng) [2026-10-07T17:34:20Z] <obvious-dust> Info: Applying configuration for deployment-mediawiki16.deployment-prep.eqiad1.wikimedia.cloud , Cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/1352263 ref T225074