Production Excellence #15: September 2019
Monthly update on our strive for operational excellence.

How’d we do in our strive for operational excellence last month? Read on to find out!

📊 Month in numbers
  • 5 documented incidents. [1]
  • 22 new errors reported. [2]
  • 31 error reports closed. [3]
  • 213 currently open Wikimedia-prod-error reports in total. [4]

There were five recorded incidents last month, equal to the median for this and last year. – Explore this data.

To read more about these incidents, their investigations, and pending actionables; check Incident documentation § 2019.

*️⃣ A Tale of Three Great Upgrades

This month saw three major upgrades across the MediaWiki stack.

Migrate from HHVM to PHP 7.2

The client-side switch to toggle between HHVM and PHP 7.2 saw its final push — from the 50% it was at previously, to 100% of page view sessions on 17 September. The switch further solidified on 24 September when static MediaWiki traffic followed suit (e.g. API and ResourceLoader). Thanks @jijiki and @Joe for the final push. – More details at T219150 and T176370.

Drop support for IE6 and IE7

The RFC to discontinue basic compatibility for the IE6 and IE7 browsers entered Last Call on 18 September. It was approved on 2 Oct (T232563). Thanks to @Volker_E for leading the sprint to optimise our CSS payloads by removing now-redundant style rules for IE6-7 compat. – More at T234582.

Transition from PHPUnit 4/6 to PHPUnit 8

With HHVM behind us, our Composer configuration no longer needs to be compatible with a “PHP 5.6 like” run-time. Support for the real PHP 5.6 was dropped over 2 years ago, and the HHVM engine supports PHP 7 features. But, the HHVM engine identifies as “PHP 5.6.999-hhvm”. As such, Composer refused to install PHPUnit 6 (which requires PHP 7.0+). Instead, Composer could only install PHPUnit 4 under HHVM (as for PHP 5.6). Our unit tests have had to remain compatible with both PHPUnit 4 and PHPUnit 6 simultaneously.

Now that we’re fully on PHP 7.2+, our Composer configuration effectively drops PHP 5.6, 7.0 and 7.1 all at once. This means that we no longer run PHPUnit tests on multiple PHPUnit versions (PHPUnit 6 only). The upgrade to PHPUnit 8 (PHP 7.2+) is also unlocked! Thanks @MaxSem, @Jdforrester-WMF and @Daimona for leading this transition. – T192167

📉 Outstanding reports

Take a look at the workboard and look for tasks that might need your help. The workboard lists error reports, grouped by the month in which they were first observed.


Or help someone that’s already started with their patch:
Open prod-error tasks with a Patch-For-Review

Breakdown of recent months (past two weeks not included):

  • February: 1 report was closed. (1 / 5 reports left).
  • March: 4 / 10 reports left (unchanged).
  • April: 8 / 14 reports left (unchanged). ⚠️
  • May: The last 4 reports were resolved. Done! ❇️
  • June: 9 of 11 reports left (unchanged). ⚠️
  • July: 4 reports were fixed! (13 / 18 reports left).
  • August: 6 reports were fixed! (8 / 14 reports left).
  • September: 12 new reports survived the month of September.

🎉 Thanks!

Thank you, to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

📖“I'm not crazy about reality, but it's still the only place to get a decent meal.


[1] Incidents. –

[2] Tasks created. –

[3] Tasks closed. –

[4] Open tasks. –

Written by Krinkle on Oct 24 2019, 11:25 PM.
Principal Engineer (Wikimedia Performance)
"Love" token, awarded by kostajh."Like" token, awarded by abi_.

Event Timeline