Production Excellence #40: January 2022
Monthly update on our strive for operational excellence.

How’d we do in our strive for operational excellence last month? Read on to find out!


There were no incidents this January. Pfew! Remember to review and schedule Incident Follow-up work in Phabricator. These are preventive measures and tech debt mitigations written down after an incident is concluded. Read about past incidents at Incident status on Wikitech.

proderr-incidents 2022-01.png (800×1 px, 166 KB)


During 2021, I compared us to the median of 4 incidents per month, as measured over the two years prior (2019-2020).

I'm glad to announce our median has lowered to 3 per month over the past two years (2020-2021). For more plots and numbers about our incident documentation, refer to Incident stats.

Since the previous edition, we resolved 17 tasks from previous months. In January, there were 45 new error reports of which 28 have been resolved within the same month, the remaining 17 have carried over to February.

With precisely 17 tasks both closed and added, the workboard remains at the exact total of 298 open tasks, for the third month in a row. That's quite the coincidence.

Take a look at the workboard and look for tasks that could use your help.

View Workboard

Figure 1: Unresolved error reports by month.

For the month-over-month numbers, refer to the spreadsheet data.


Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

It could mean that that point in time contains some cosmic significance.., as if it were the temporal junction point of the entire space-time continuum… Or it could just be an amazing coincidence.

Written by Krinkle on Feb 4 2022, 4:32 AM.
Principal Engineer (Wikimedia Performance)

Event Timeline