Production Excellence #35: August 2021
Monthly update on our strive for operational excellence.

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

Zero documented incidents last month. Isn't that something!

Learn about past incidents at Incident status on Wikitech. Remember to review and schedule Incident Follow-up in Phabricator, which are preventive measures and other action items to learn from.

proderr-incidents 2021-08.png (834×2 px, 288 KB)

Image from Incident graphs.


Trends

In August we resolved 18 of the 156 reports that carried over from previous months, and reported 46 new failures in production. Of the new ones, 17 remain unresolved as of writing and will carry over to next month.

The number of new errors reports in August was fairly high at 46, compared to 31 reports in July, and 26 reports in June.

Unresolved error reports, stacked by month

The backlog of "Old" issues saw no progress this past month and remained constant at 146 open error reports.

Total open production error tasks, by month.

💡 Did you know:

You can zoom in to your team's error reports by using the appropriate "Filter" link in the sidebar of our shared workboard.

Take a look at the workboard and look for tasks that could use your help.

View Workboard


Progress

Last few months in review:

Jan 2021 (50 issues)3 left.
Feb 2021 (20 issues)6 > 5 left.
Mar 2021 (48 issues)13 > 10 left.
Apr 2021 (42 issues)18 > 17 left.
May 2021 (54 issues)22 > 20 left.
Jun 2021 (26 issues)11 > 10 left.
Jul 2021 (31 issues)16 > 12 left.
Aug 2021 (46 issues)+ 17 new unresolved issues.

Tally:

156issues open, as of Excellence #34 (July 2021).
-18issues closed, of the previously open issues.
+17new issues that survived August 2021.
155issues open, as of today (3 Sep 2021).

For more month-over-month numbers refer to the spreadsheet.


Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

Written by Krinkle on Wed, Sep 8, 3:53 AM.
Principal Engineer (Performance)
Projects
None
Subscribers
W333Com
Tokens
"Like" token, awarded by kostajh."Yellow Medal" token, awarded by thcipriani."Love" token, awarded by Quiddity.

Event Timeline