Production Excellence #27: December 2020
Monthly update on our strive for operational excellence.

How’d we do in our strive for operational excellence last month? Read on to find out!

📈 Incidents

1 documented incident in December. [1] In previous years, December typically had 4 or fewer documented incidents. [3]

Learn about recent incidents at Incident documentation on Wikitech, or Preventive measures in Phabricator.


📊 Trends

Unresolved error reports stacked by recent month
Total open production error tasks, by month

Month-over-month plots based on spreadsheet data. [4] [2]


📖 Outstanding errors

Take a look at the workboard and look for tasks that could use your help.
https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months:

  • ⚠️ July 2019 (2 of 18 issues left): no change.
  • ⚠️ August 2019 (1 of 14 issues): no change.
  • ⚠️ September 2019 (2 of 12 issues): One task resolved (-1).
  • ⚠️ October 2019 (5 of 12 issues): no change.
  • ⚠️ November 2019 (1 of 5 issues): no change.
  • ⚠️ December 2019 (4 of 9 issues), no change.
  • ⚠️ January 2020 (2 of 7 issues), no change.
  • February 2020 (2 of 7 issues left), no change.
  • March 2020 (2 of 2 issues left), no change.
  • April 2020 (9 of 14 issues left): no change.
  • May 2020 (7 of 14 issues left): no change.
  • June 2020 (7 of 14 issues left): no change.
  • July 2020 (9 of 24 new issues): no change.
  • August 2020 (23 of 53 new issues): no change.
  • September 2020 (13 of 33 new issues): One task resolved (-1).
  • October 2020 (35 of 69 new issues): Four issues fixed (-4).
  • November 2020 (14 of 38 new issues): Five issues fixed (-5).
  • December 2020: 22 of 33 new issues survived the month and remained unresolved (+33; -22)
Recent tally
149as of Excellence #26 (15 Dec 2020).
-11closed of the 149 recent issues.
+22new issues survived December 2020.
160 as of 27 Jan 2020.

🎉 Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof


Footnotes:

[1] Incident documentation 2020, Wikitech.
[2] Open tasks, Wikimedia-prod-error, Phabricator.
[3] Wikimedia incident stats by Krinkle, CodePen.
[4] Month-over-month, Production Excellence spreadsheet.

Written by Krinkle on Feb 4 2021, 5:46 AM.
Principal Engineer (Wikimedia Performance)
Projects
None
Subscribers
None

Event Timeline