Production Excellence #35: August 2021
Monthly update on our strive for operational excellence.

How’d we do in our strive for operational excellence last month? Read on to find out!


Zero documented incidents last month. Isn't that something!

Learn about past incidents at Incident status on Wikitech. Remember to review and schedule Incident Follow-up in Phabricator, which are preventive measures and other action items to learn from.

proderr-incidents 2021-08.png (834×2 px, 288 KB)

Image from Incident graphs.


In August we resolved 18 of the 156 reports that carried over from previous months, and reported 46 new failures in production. Of the new ones, 17 remain unresolved as of writing and will carry over to next month.

The number of new errors reports in August was fairly high at 46, compared to 31 reports in July, and 26 reports in June.

Unresolved error reports, stacked by month

The backlog of "Old" issues saw no progress this past month and remained constant at 146 open error reports.

Total open production error tasks, by month.

Unified graph:

proderr-unified 2021-08.png (1×1 px, 78 KB)

💡 Did you know:

You can zoom in to your team's error reports by using the appropriate "Filter" link in the sidebar of our shared workboard.

Take a look at the workboard and look for tasks that could use your help.

View Workboard


Last few months in review:

Jan 2021 (50 issues)3 left.
Feb 2021 (20 issues)6 > 5 left.
Mar 2021 (48 issues)13 > 10 left.
Apr 2021 (42 issues)18 > 17 left.
May 2021 (54 issues)22 > 20 left.
Jun 2021 (26 issues)11 > 10 left.
Jul 2021 (31 issues)16 > 12 left.
Aug 2021 (46 issues)+ 17 new unresolved issues.


156issues open, as of Excellence #34 (July 2021).
-18issues closed, of the previously open issues.
+17new issues that survived August 2021.
155issues open, as of today (3 Sep 2021).

For more month-over-month numbers refer to the spreadsheet.


Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

Written by Krinkle on Sep 8 2021, 3:53 AM.
Principal Engineer (Wikimedia Performance)
"Like" token, awarded by kostajh."Yellow Medal" token, awarded by thcipriani."Love" token, awarded by Quiddity.

Event Timeline