HomePhabricator
Production Excellence #24: September 2020
Monthly update on our strive for operational excellence.

How’d we do in our strive for operational excellence last month? Read on to find out!

📈 Incidents

5 documented incidents. [1] Historically, that's right on average for the time of year. [3]

For more about recent incidents see Incident documentation on Wikitech, or Preventive measures in Phabricator.


📊 Trends

month-stack-2020-10-23.png (1×1 px, 113 KB)
total-stack-2020-10-23.png (941×1 px, 97 KB)

Month-over-month plots based on spreadsheet data. [5]


📖 Outstanding errors

Take a look at the workboard and look for tasks that could use your help.
https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months:

  • ⚠️ July 2019 (4 of 18 tasks left): no change.
  • ⚠️ August 2019 (1 of 14 tasks left): no change.
  • ⚠️ September 2019 (3 of 12 tasks left): no change.
  • ⚠️ October 2019 (6 of 12 tasks left), no change.
  • November (3 of 5 tasks left): no change.
  • December (3 of 9 tasks left), no change.
  • January 2020 (4 of 7 tasks left), One task closed.
  • February (2 of 7 tasks left), no change.
  • March (2 of 2 tasks left), no change.
  • April (10 of 14 tasks left): no change.
  • May (7 of 14 tasks left): no change.
  • June (7 of 14 tasks left): Three tasks closed.
  • July 2020 (10 of 24 new tasks): Three tasks closed.
  • August 2020 (31 of 53 new tasks): Six tasks closed.
  • September 2020: 17 of 33 new tasks survived the month of September and remain open today.
Recent tally
106as of Excellence #23 (Sep 23rd).
-13closed of the 106 recent tasks.
+17survived September 2020.
110as of today, Oct 23rd.

Previously, we had 106 unresolved production errors from the recent months up to August. Since then, 13 of those were closed. But, the 18 errors surviving September raise our recent tally to 110.

The workboard overall (including errors from 2019 and earlier) holds 343 open tasks in total, an increase of +47 compared to the 296 total on Sept 23rd.


🎉 Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof


🕵️‍♀️ Holmes: “So, she pulled five bullets out of you?”

     Shinwell: ”That's right.”
     Holmes: “I too have been shot five times. But, uh..., separate occasions.”
     Shinwell: “That's... great.”

Footnotes:

[1] Incidents. – wikitech.wikimedia.org/wiki/Incident_documentation
[2] Open tasks. – phabricator.wikimedia.org/maniphest/query…
[3] Wikimedia incident stats. – codepen.io/Krinkle/full/wbYMZK
[4] Month-over-month plots. – docs.google.com/spreadsheets/d/1tRC…

Written by Krinkle on Oct 23 2020, 11:51 PM.
Principal Engineer (Wikimedia Performance)
Projects
None
Subscribers
None

Event Timeline