Page MenuHomePhabricator

How to address the long tail of low priority tasks in active projects
Closed, ResolvedPublic

Description

But we also need better tools/strategies to better plan low/lowest priority bugs - these are the bugs which take 'forever' to be fixed, and cause disenfranchisement when the various 'stakeholders'' have different expectations about the priority that should be given to the problem. I don't see how Priority Low/Lowest/Needs volunteer/None/whatever is part of the solution to the 'long tail' problem.

Very true. While we continue developing tools and processes helping the mainly professional developers deal better with the higher priority tasks, the long tail of lower priority tasks that are not even considered in anybody's backlogs keeps growing.

We need to be better at

  • providing mechanisms for external contributors to identify low priority tasks that should be promoted in project backlogs
  • promoting easy tasks suitable for new / casual / junior contributors
  • connecting popular priorities with maintainers' plans
  • reviewing contributions fast (see T78768)
  • closing marginal tasks that clearly have little to no chance to be addressed ever

What else? And which specific actions can we take here and now to start solving the problem?


See Also (though not "low priority" per se):
T113706: Define Cut-off date for abandoning old changesets in our code review tool (abandoning a rotting changeset = declining a rotting task?)
T105942: Come up with potential criteria for scoring project/task ideas
T107833: Build an initial work backlog for Community Tech based on one or more existing community wishlists

Related Objects

StatusSubtypeAssignedTask
ResolvedQgil
ResolvedDicortazar
ResolvedQgil
ResolvedQgil
ResolvedQgil
InvalidNone
InvalidNone
ResolvedNone
ResolvedAklapper
ResolvedQgil
ResolvedAklapper
OpenNone
ResolvedQgil
ResolvedAklapper
ResolvedAklapper
ResolvedAklapper
ResolvedDicortazar
ResolvedDicortazar
ResolvedAcs
ResolvedDicortazar
ResolvedDicortazar
ResolvedDicortazar
ResolvedDicortazar
ResolvedDicortazar
ResolvedDicortazar
InvalidDicortazar
ResolvedDicortazar
ResolvedDicortazar
ResolvedAklapper
ResolvedDicortazar
ResolvedDicortazar
DuplicateNone
ResolvedDicortazar
ResolvedBawolff
Resolved mmodell
ResolvedNone
Resolved mmodell
ResolvedLegoktm
Resolvedtstarling
Resolvedgreg
ResolvedAklapper
ResolvedAklapper
ResolvedAklapper
ResolvedAklapper
ResolvedNone
DeclinedNone
DeclinedAklapper

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

promoting easy tasks

That's related to review speed (T78768), and to awareness of established developers to judge the complexity of a task and to actually set the easy tag so tasks will end up on Annoying little bugs too. How can we make developers set the easy tag more often?
(related but off-topic now: When do developers add TODO comments in the code instead of filing a task, see Storey, Margaret-Anne and Ryall, Jody and Bull, R. Ian and Myers, Del and Singer, Janice: TODO or to bug: exploring how task annotations play a role in the work practices of software developers. ICSE 2008, Leipzig, Germany, pages 251-260)

connecting popular priorities with maintainers' plans

cf. T78482 which is now T802. Popularity is only one potential input for decisions, and I'd be extremely careful with it because it's so hard to measure properly.
“If I had asked people what they wanted, they would have said faster horses” (though there is no evidence that Henry Ford ever said that).

closing marginal tasks that clearly have little to no chance to be addressed ever

That's only done when some triager is experienced and in constant contact with a dev team and can judge whether a request is rather 'esoterical' (who am I to judge as a triager?). Again, it requires willingness to spend a lot of time explaining to ticket reporters why certain requests will not be fulfilled. You will not have that patience when you have enough stuff on your plate.

It may be useful to create a stale-patch project/tag,

I don't want to investigate that as long as we use Gerrit instead of Diffusion but integrated metrics after the migration (e.g. stressing in the UI that a patch is X months old and/or has not received reviews yet, queries based on age) would be lovely.
This must be automated in the very same system, otherwise we end up with stale/outdated "stale-patch" tags. You always need someone to care and update such stuff and there have been enough fixed-in-Gerrit-but-never-closed-in-Bugzilla" tickets with Patch_To_Review status in Bugzilla, I can tell you.

e.g. create a column for patch not in gerrit,

Rather uncommon - BZ had approx. 112 tickets with rather ancient patches attached that won't apply cleanly, and there's not many cases of inline pasting of Git-formatted patches either (could not find a good regex to query for simple "diff -pu" ones).

@Aklapper, @Qgil are there any specific asks of the Team Practices Group here, or did you just want to keep this on our radar?

@Awjrichards, at least the #Engineering-Community team thinks that "How to address the long tail of low priority tasks" should be a topic discussed among developer teams, to see which actions can help progress in that direction. Does the Team Practices team agree with this goal? What is your opinion about how to pursue a solution? If ECT and TP are aligned, then we will have a slightly better chance of succeeding.

IMHO the main solution here is reducing (feature) development at WMF, so that there is less cleanup for the community to do.

Data collected on June 23rd, 2014. I have a very vague assumption there might be a lower growth rate when the project is dominated by corporate developers.

Numbers below 1 mean that more reports are closed than filed. Unsurprising, because KDE and RedHat bugzillas host some projects which periodically close all bugs against old releases (usually to compensate the mass of bugs created by automatical reporting tools bundled in OS).

Community backlogs are becoming more relevant. The WMDE-TechWish with its German-Community-Wishlist and the upcoming Community-Tech team with the expected evolution of the Project Ideas in Meta, should help in the following months introducing some organized prioritization in this long queue. Commons, All-and-every-Wiktionary, and other content projects are also involved in the selection of tasks that matter.

So community backlogs are part of the solution, and it should be possible to define i.e. a quarterly goal around them. Any takers?

@Aklapper is defining two individual goals for the upcoming quarter: T102611: Goal: Drive more complex long-term tasks closer to resolution and T102611: Goal: Drive more complex long-term tasks closer to resolution. Hopefully they will contribute some progress here as well.

What other specific goals could we set to progress with this epic task?

If as a result of T102920: What to do with old open patches for unmaintained/inactive repositories when not even the uploader responds we can define some Phabricator projects INACTIVE, then we could agree on a way to prioritize or tag their related tasks. Removing tasks from inactive projects from the long tail could help bringing a bit more focus to the rest of tasks related with active projects.

If as a result of T102920: What to do with old open patches for unmaintained/inactive repositories when not even the uploader responds we can define some Phabricator projects INACTIVE, then we could agree on a way to prioritize or tag their related tasks.

(General comment apart from what I wrote already: "Priority" means different things to different stakeholders. We have not defined that yet, cf. T103556#1458377 )

If a project is just inactive (but its code could still be useful if picked up by anybody), its tasks' priorities should reflect the reality. If noone works on the project's codebase, every task is de facto "Lowest priority".
Different case: A project has been deprecated for years / obsoleted already && never had a large deployment base && the codebase "really should not be worked on anymore", not even for smaller maintenance tasks. In that case its tasks should be closed as declined.

Regarding inactive projects (and their priority in Phabricator), I've moved that to T102920#1571882 instead.
Also I'm changing the task summary from "How to address the long tail of low priority tasks" into "How to address the long tail of low priority tasks in active projects".

Aklapper renamed this task from How to address the long tail of low priority tasks to How to address the long tail of low priority tasks in active projects.Aug 25 2015, 4:47 PM

I see a bunch of aspects mixed here:

  • Open tasks (both bugs and enhancement/feature requests) which can be either
    • not addressed due to limited manpower / interest / architecture complexity / technical debt, etc
    • not updated (does the described problem still exists; TestMe tag) due to limited manpower / interest
  • Open tasks in projects which are maintained but have teams / maintainers which do not really use the Priority field, either because they prefer Sprint boards or because the number of open tasks is small and "overviewable". Plus priority is per task and not per each project associated with a task, and there are different interpretations what it should mean and who "drives" a task, see T87411
  • Not covered here, see T101686 and its followup T78768: Open tasks with patches in different states (unreviewed / reviewed with +1 and unmerged / reviewed with -1).
  • Not covered here, see T102920: Open tasks in inactive / unmaintained projects
In T78639#850019, @Qgil wrote:
  • providing mechanisms for external contributors to identify low priority tasks that should be promoted in project backlogs

IMO that is now https://meta.wikimedia.org/wiki/2015_Community_Wishlist_Survey (but as Gryllida already wrote: Regardless of task priority, also applies to high priority tasks.)

  • promoting easy tasks suitable for new / casual / junior contributors

IMO that is good first task and https://www.mediawiki.org/wiki/Annoying_little_bugs which is hard to keep up-to-date (T122683).
Also there are very different types of newcomers which we do not really differentiate: People who started programming vs. people who know programming concepts but do not know well the specific programming language, people who know programming and the language but are used to corporate and do not know FOSS culture and workflows yet. Mozilla has two separate programs for that ("Good First Bug" vs. "Mentored Bugs"). Our onboarding docs could always be better but I'll tackle that in T129068: Improve code contribution guidelines for patch authors.

  • connecting popular priorities with maintainers' plans

IMO that is now https://meta.wikimedia.org/wiki/2015_Community_Wishlist_Survey in combination with https://www.mediawiki.org/wiki/WMF_product_development_process (which is a draft and maybe not too clear who's supposed to drive it)

  • reviewing contributions fast

CR process dealt with in T101686 and its followup T78768 but not here.

In T78639#850019, @Qgil wrote:
  • closing marginal tasks that clearly have little to no chance to be addressed ever
In T78639#852912, @Qgil wrote:

A marginal task might make some sense when it is created in 2007, but not so much in 2015. I believe we have many cases like this, especially in the former Bugzilla "Lowest" range. Even if it feels like it is nicer not to close them, at some point they contribute involuntarily to the difficulty to find the old tasks that are still relevant.

This is the discussion of setting "RESOLVED WONTFIX/declined" (not in scope of the project, bad idea) vs. setting "lowest priority" (maintainers accept a contributed patch). Maintainers can use both by gardening the backlog but they often don't have the time for that or interest to go through lots of ancient tasks.
There are some issue tracker instances which automatically close older tasks (e.g. RedHat Bugzilla, Ubuntu in Launchpad, to some extent KDE projects) as "someone will report it again if it still happens" to keep lists cleaner (if people don't manage to restrict their search results by date of last update which is possible). And we are in an open system with only-theoretical unlimited contributor manpower.
As I wrote here already before: "it requires willingness to spend a lot of time explaining to ticket reporters why certain requests will not be fulfilled. You will not have that patience when you have enough stuff on your plate."

Summary

Apart from

  1. code review problems already moved to their own tasks,
  2. the fact that the "Priority" field is not consistently applied/used,
  3. the fact that the "Priority" field is not self-explanatory which is T87411,
  4. the fact that we don't have many people retesting older tickets, and maintainers/teams not consistently triaging older tasks by both tagging as TestMe when appropriate or closing as declined when they are against the project direction,
  5. potential future improvements to our general development onboarding process (not specifically related to our long tail of low prio tasks only, but definitely a potential influencing factor) to increase our contributor base,

the only potential actionable items I see in this task are

  • improving general expectation communication: "If you don't work on it, don't expect anybody else to work on it." Regardless of any task priority. Communication as part of the #WMF-Product-Development-Process is related.
  • discussing whether the Wikimedia tech community wants to automatically tag or close older tasks which would both require convincing reasons (problem definition; see "marginal tasks" above) and good criteria (only feature requests? also bug reports? after which time of which action not applied? opt-out for certain projects/maintainers?) to identify such tasks.
  • foster a routine / team practice in (WMF) teams to regularly garden their Phabricator task backlogs by correctly prioritizing and communicating expectations (see above). This might be Team-Practices territory.

I'd love to see good first task split into #one-liner / good first task vs #Mentored-Bugs . IMO 'easy' can give the wrong impression, and sometimes a bug which looks easy turns out to be more complicated when analysed, and even very complicated if good design necessitates a rewrite with backwards compatibility required.

Have we looked into adding a 'Mentor' field in Phabricator, like exists in Mozilla's but tracker? It would also be useful for Possible-Tech-Projects .

+1 for mentored bugs. Google Code-In is very successful, and the chances of actually winning it are tiny - I don't think much of the motivation would be lost if we took away the competition part (or maybe not even the competition, just the prize).

Make some kind of portal where it is easy to browse small bugs by technology and difficulty level, and mentors are available to ask for help; it's a much smoother experience than pointing beginners at the Easy tag and watching them sink or swim.

I'd love to see good first task split into #one-liner / good first task vs #Mentored-Bugs .
Have we looked into adding a 'Mentor' field in Phabricator, like exists in Mozilla's bug tracker?

"Do Onboarding Programs Work?" by Adriaan Labuschagne and Reid Holmes (2015) looked at Mozilla's 'Good First Bugs' (GFB) and 'Mentored Bugs' (MeB; something that Wikimedia does not have) programs, taking quantitative data from Bugzilla and a qualitative survey with 11 developers with at least 10 contributions who started in one of these programs. Any quotes in this comment are taken directly from that paper:

  • "The likelihood that a developer's first contribution is successful" is 86% for Meb&&GFB, 80% for MeB, 67% for GFB, 73% for not in any program, hence "The dropout rate for program participants is much higher than for developers that did not start in a program" and "the data suggests that while developers in onboarding programs are more likely to succeed with their first attempts they are generally less likely to become long-term contributors". Possible reasons: "attract individuals who would not otherwise have attempted to make a contribution at all" who are "less equipped to transition into long-term contributors". "downplaying the difficulty of tackling a GFB may in fact increase the chances of failure since the expectation may be that these bugs should be 'easy'".
  • Interviewees appreciated mentorship, listed a variety of starting points (Bugsahoy, Bugzilla search, IRC, Codefirefox.com), and underestimated the required effort to contribute compared to size and impact of the contribution.
  • Note that "it is unclear whether these successful contributors would have started contributing without these programs, this paper provides quantitative evidence that these programs alone do not automatically improve the odds of a new developer becoming a long-term contributor" but newcomers might potentially "continue to be actively involved in the project in other ways, such as reporting bugs" (which was not measured in their paper).

So I agree that we might have a gap between "trivial easy tasks" and "advanced complex task" in onboarding, but due to the first bullet point, I am generally not convinced if efforts are worth the costs. (Please convince me that I'm wrong.) We might want to advertise Possible-Tech-Projects (and/or https://www.whatcanidoforwikimedia.org/ ) more instead.
(And as long as Vagrant isn't trivial to set up for Windows users I'm not happy with calling them good first task either, to be honest.)

While the goal of converting to long term devs is useful, it isnt the goal of this task. Lots of one time devs is useful to address the long tail, and #one-liners are useful as beginner GCI tasks, where the objective is to let a newbie figure out the tools necessary to get a change through the system.

Also mentors of mentored tasks may not be interested in conversion. The mentor may only be interested in helping one bug be fixed, and they are may not be a good coder but instead have good domain knowledge and able to do QA (e.g. a wiki user who knows the business side wanting to help a coder without any wiki experience who may not even be able to understand the bug description without assistance.)

@Aklapper and everyone interested in this: my apologies for dropping the ball on notification on E148. I thought some auto-notifications would happen as a result of my mentioning it in E148, but that doesn't appear to be the case from this task's history.

Here are the more relevant bits of the E148 minutes:

3:00 PM <wm-labs-meetbot`> Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-03-16-21.00.html
3:00 PM <wm-labs-meetbot`> Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-03-16-21.00.txt
3:00 PM <wm-labs-meetbot`> Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-03-16-21.00.wiki
3:00 PM <wm-labs-meetbot`> Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-03-16-21.00.log.html

Meeting ended at 22:00:02 UTC.

It may be useful to create a stale-patch project/tag

This happened now via Patch-Needs-Improvement (see T129221).

I've summarized the three potential actionable items that I see in comment T78639#1910648.
If there is interest, anyone could create separate dedicated tasks for each item.

I do not see anything left for me in this task to work on (and personally I'd close this task), hence I remove myself as assignee.

(and personally I'd close this task)

Yeah, maybe this is the right thing to do.

Qgil assigned this task to Aklapper.

Alright, then. Thank you for the discussion and lessons learned in this task, which leads us to the other tasks identified.