Production Excellence #26: November 2020
Monthly update on our strive for operational excellence.

Howโ€™d we do in our strive for operational excellence last month? Read on to find out!

๐Ÿ“ˆ Incidents

Zero documented incidents in November. [1] That's the only month this year without any (publicly documented) incidents. In 2019, November was also the only such month. [3]

Learn about recent incidents at Incident documentation on Wikitech, or Preventive measures in Phabricator.


๐Ÿ“Š Trends

Unresolved error reports stacked by recent month
Total open production error tasks, by month

The overall increase in errors was relatively low this past month, similar to the November-December period last year.

What's new is that we can start to see a positive trend emerging in the backlogs where we've shrunk issue count three months in a row, from the 233 high in October, down to the 181 we have in the ol' backlog today.

Month-over-month plots based on spreadsheet data. [5]


๐Ÿ“– Outstanding errors

Take a look at the workboard and look for tasks that could use your help.
โ†’ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months:

  • โš ๏ธ July 2019 (2 of 18 tasks): One task closed (-1).
  • โš ๏ธ August 2019 (1 of 14 tasks): no change.
  • โš ๏ธ September 2019 (3 of 12 tasks): no change.
  • โš ๏ธ October 2019 (5 of 12 tasks): no change.
  • โš ๏ธ November 2019 (1 of 5 tasks): no change.
  • โš ๏ธ December 2019 (3 of 9 tasks left), no change.
  • January 2020 (3 of 7 tasks left), One task closed (-1).
  • February (2 of 7 tasks left), no change.
  • March (2 of 2 tasks left), no change.
  • April (9 of 14 tasks left): no change.
  • May (7 of 14 tasks left): no change.
  • June (7 of 14 tasks left): no change.
  • July 2020 (9 of 24 new tasks): no change.
  • August 2020 (23 of 53 new tasks): Three tasks closed (-3).
  • September 2020 (14 of 33 new tasks): One task closed (-1).
  • October 2020 (39 of 69 new tasks): Six tasks closed (-6).
  • November 2020: 19 of 38 new tasks survived the month and remain open today (+38; -19)
Recent tally
142as of Excellence #25 (23 Oct 2020).
-12closed of the 142 recent tasks.
+19survived November 2020.
149 as of today, 15 Dec 2020.

The on-going month of December, has 19 unresolved tasks so far.


๐ŸŽ‰ Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

โ€“ Timo Tijhof


โ ย  The plot "thickens" as they say. Why, by the way? Isย it a soup metaphor? โž

Footnotes:

[1] Incident documentation 2020, Wikitech.
[2] Open tasks, Wikimedia-prod-error, Phabricator.
[3] Wikimedia incident stats, Krinkle, CodePen.
[4] Month-over-month, Production Excellence (spreadsheet).

Written by Krinkle on Dec 15 2020, 9:49 PM.
Principal Engineer (Performance)
Projects
None
Subscribers
None
Tokens
"Like" token, awarded by abi_.

Event Timeline