Page MenuHomePhabricator

Write scripts to track cycle time of tasked tickets and velocity [8 pts]
Closed, ResolvedPublic

Description

Write script to calculate time from when a ticket enters Tasked column in the Analytics-Backlog board to when it enters the Done column in Analytics-Kanban board

Discussed in Aug 6, 2015 Retrospective

also calculate biweekly velocity based on points of tasks that are in done and resolved.

Event Timeline

ggellerman raised the priority of this task from to Needs Triage.
ggellerman updated the task description. (Show Details)
ggellerman added a project: Analytics-Backlog.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 6 2015, 5:47 PM

Alternatively, the end of the cycle could be when the task is marked as Resolved from Done column in the Analytics-Kanban board

Thanks to @ksmith, here is info about @RobLa's script:

code:
https://github.com/robla/phab-wbstatus

Running instance, pointed at some arbitrary week that contains data:
https://mw-core-wbstatus.wmflabs.org/?r=2015-03-16_to_2015-03-20

Per Kevin S, note the information like "Still working on it (since Wed, Mar 11)"

kevinator renamed this task from Write script to track cycle time of tasked tickets to Write scripts to track cycle time of tasked tickets and velocity .Aug 7 2015, 4:24 PM
kevinator triaged this task as High priority.
kevinator updated the task description. (Show Details)
kevinator set Security to None.
kevinator moved this task from Incoming to Prioritized on the Analytics-Backlog board.
kevinator renamed this task from Write scripts to track cycle time of tasked tickets and velocity to Write scripts to track cycle time of tasked tickets and velocity [8 pts] .Aug 17 2015, 4:36 PM
kevinator moved this task from Prioritized to Tasked on the Analytics-Backlog board.
mforns claimed this task.Aug 21 2015, 6:02 PM
mforns edited projects, added Analytics-Kanban; removed Analytics-Backlog.
mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.

IIRC we discussed about using something similar to this to measure productivity:

"lead time per point"

(lead-time1 + lead-time2 + ... + lead-timeN) / (points1 + points2 + ... + pointsN)

Just for the tasks that got resolved (or moved to done) in a given time range.

I like this metric. Now, I can also see 2 potential problems:

  • It can happen that a certain week only long-lead-time tasks get resolved, giving the impression that we worked slowly, where it could be that various other tasks were put into ready to delpoy and thus were not counted in the average. The contrary can happen as well.
  • Also, the quantity of points moved to the right is "not counted" in this metric. We could have a good productivity measure having moved very little points to the right (if moved very quickly), and vice-versa.

I thought about another metric that wouldn't be affected by the task granularity so much, I'll try to describe it:

  • In a given time range:
  • Every time a task moves one column to the right, you add 1/4 of that task's points.
  • Every time a task moves one column to the left, you subtract 1/4 of that task's points.
  • Moving from "Next up" to "Paused" and vice-versa doesn't add nor subtract (As if they were the same column).
  • The remaining sum gives you a point throughput, points moved to the right.

Why 1/4? Because there are 4 steps to get from "Next up"/"Paused" to "Done". So when a task is resolved you'll get the entirety of its points. This model would attribute the following weights to the kanban pipeline:

  • 50% Doing the stuff (25% from "Next up" to "In progress" and 25% from "In Progress" to "In CR")
  • 25% Code review
  • 25% Deployment

I know that this is very simplistic and that many tasks do not fit in this model, but it may be useful anyway (I think it's a good approximation)! Note that if you have a task that doesn't have CR and/or deployment, when you move it directly to done, it will count all steps. We can discuss this model and change it if you are interested :]


So, with that, even if we do not resolve many tasks before our retrospective, we can see the value of our week! Or we can identify a weak week (o.O) even if we resolved many tasks.

To implement this, we can use Robla's code, which assumes there's a folder with a copy of the workboard html for each day: Parse the html for the tasks and compare their positions start-date vs. end-date. We can wget with query=all, which will return the resolved tasks visible in the "Done" (right-most) column.

mforns moved this task from In Progress to Paused on the Analytics-Kanban board.Aug 24 2015, 4:58 PM

Another idea is to apply the same 4-step logic to lead time:

  • Given a start date and an end date,
  • And using the 4-step model described above:
  • Predict the total lead time of a partially-finished task based on the time it has taken to get to the current position,
  • And average the measuring of all tasks (per task points).

Note that, the longer the time range, the closer this metric would be with the first proposal for lead time.
As well, the previous "points-moving-to-the-right" metric, the longer the time range, the closer it would be to a straightforward closed points count.

I really like the thinking here. I'd like to propose tracking three numbers on a weekly basis, which I think eliminate the need for lead time. An example is the easiest way to explain, let's say we have a task that goes through our kanban board like this:

Next Up -> In Progress -> Code Review -> Paused -> In Progress -> Ready To Deploy -> Done -> Resolved

Then I'd like to propose:

  • Paused and In Progress are the same column essentially, so moving between them in either direction has 0 change.
  • One way to sum up the tasks then, is to have a Net sum as Marcel Proposes, +1/4 * points for moves to the right, -1/4 * points for moves to the left, in the example above, where P is the number of points for the task:
Next Up -> In Progress -> Code Review -> Paused -> In Progress -> Ready To Deploy -> Done -> Resolved
         + P/4          + P/4          + -P/4    + 0            + 2P/4             + P/4   = P
  • And the other way, to show the "thrashing" that happens to a task, is just to apply an absolute value to any change, so:
Next Up -> In Progress -> Code Review -> Paused -> In Progress -> Ready To Deploy -> Done -> Resolved
         + P/4          + P/4          + |-P/4|  + 0            + 2P/4             + P/4   = 6P/4

This way, we know that this task took our team more effort to context switch away from / and back to, or unblock, or review several times, or whatever the reason for the movements to the left.

We could also used the transition to "Resolved" to count the points into a third metric, one that only measures what's been approved as officially finished.

As an outside observer, I'll express minor discomfort with the concept of getting "partial credit" just for advancing a task to the next column. My standard model is that a task is done, or it's not done. Statuses like 25% done or 75% done are often misleading, and I would hesitate to put anything in place that encourages that kind of thinking.

Obviously this tool is intended to solve your concerns, not mine, so feel free to disregard this comment if it isn't meaningful for your situation.

@ksmith

Thanks for your input!

+1 to the idea that a task is done or it's not done

@mforns @Milimetric
We limit the number of tasks that we allow to be In Progress rather than the number of points. So I would also like to keep the notion of velocity (how many points are finished in a given iteration length) separate from the cycle time of an individual task to travel from the Tasked column on the backlog board to the Done column on the analytics-kanban board.

I like the idea of measuring context switching (clever!), but I am wondering if that could be handled as its own metric?

@ksmith @ggellerman
I understand your concerns. You're right, the "it's X% complete", most of the time, turns out to be wrong :]

However, considering we're using a kanban flow as opposed to a sprint flow, we're not planning to finish a set of tasks before a deadline. So, many times, when we get to the retrospective meeting, lots of tasks are actually on-going, either because they weren't finished since the last retro, or because they were started in the middle of the week.

I think the straightforward closed-points-count metric is really good for sprint evaluation, or even long-term (monthly/quarterly) kanban evaluation; but not for weekly kanban evaluation. I think this metric wouldn't faithfully reflect our results in this particular case.

@mforms: Actually I would argue that # tasks completed is exactly appropriate in Kanban. Showing fewer tasks completed in a week could help point to problems with too much WIP, or batch sizes being larger than ideal. Visualizing the percentage could help obscure both of those dysfunctions.

Showing tasks (or points) completed will be "lumpier" than showing a progress percentage, but to me that's a feature, not a bug.

@mforns

Thanks for feedback! Let's talk about this more in tomorrow's retro

@ksmith @ggellerman

Actually I would argue that # tasks completed is exactly appropriate in Kanban. Showing fewer tasks completed in a week could help point to problems with too much WIP, or batch sizes being larger than ideal. Visualizing the percentage could help obscure both of those dysfunctions.

I agree. You changed my mind :]

I still see the partial-credit metric as valuable, though. Maybe we should have both, as @Milimetric suggested?

@ksmith

Well done!

@mforns
There is no partial credit- tasks are done or not done, but I do see value in measuring context switches that occur during the cycle time of a ticket entering the Tasked column on the backlog to the Done column of the kanban board.

Speaking of which, I suspect that we might be introducing scope creep into an 8 pt task ;)

Discussed more with @mforns and @Milimetric and @JAllemandou at standup today:

  1. Reviewed Kanban emphasis on finishing tasks, keeping them small (when possible) and limiting work in progress.
  2. @Milimetric convinced me that the metric about the number of times and directions a task moves will also convey the task's cycle time

2.a. I expressed deep concerns about calling it, "partial credit" (see 1)

  1. We agreed to use a 2 week window rather than 1 week

Thanks, Analytics Eng team!

mforns moved this task from Paused to In Progress on the Analytics-Kanban board.Aug 31 2015, 3:12 PM

Change 235470 had a related patch set uploaded (by Mforns):
Add script to calculate kanban productivity metrics

https://gerrit.wikimedia.org/r/235470

Change 235470 merged by Milimetric:
Add script to calculate kanban productivity metrics

https://gerrit.wikimedia.org/r/235470

mforns moved this task from In Code Review to Done on the Analytics-Kanban board.Sep 2 2015, 9:35 PM
kevinator closed this task as Resolved.Sep 9 2015, 5:56 PM

This could be really useful to other teams, so please share the results publicly when you are ready.

mforns added a comment.Sep 9 2015, 8:53 PM

@ksmith
Sure, maybe I can give a lightning talk about this.
Thanks for the heads up!

@ksmith:
I've also let @Awjrichards know about the script as well