Page MenuHomePhabricator

Goal: Define potential actions to reduce code review queues and waiting times
Closed, ResolvedPublic

Description

https://office.wikimedia.org/wiki/Health_check_survey_results/FY2014-15_Q3 indicates a growing concern about code review. Our code review queues keep growing, I'd say still faster than our concerns, and this trend can only lead to some variation of *collapse*.

http://korma.wmflabs.org/browser/gerrit_review_queue.html shows these monthly snapshots of open chagesets waiting for review (-1 and work in progress are not counted):

  • May 2013: 250
  • May 2014: 1033
  • May 2015: 1549

In order to stop this trend, revert it, and clean our queues we need a radical change.

Any actions must be coordinated with the migration from Gerrit to Differential in T114311.

Problems

The current situation is the result of a combination of problems. Let's identify them, creating an own task when appropriate:

  • Prioritization / weak open source culture: more pressure to write new code than to review patches contributed.
  • Unmaintained repositories: lack of owners is a problem, lack of disclaimers for potential contributors to these repos too. (T102920)
  • Volunteers having a hard time getting their contributions reviewed, also in active repos (T78768)
  • Lack of sync between developer teams: team A stuck because team B doesn't review their patches.
  • Our code review practices are not common across reviewers and don't focus on efficiency (T101686#1355049)
  • Our Continuous Integration tooling could be better at catching trivial errors or code style issues (T101686#1355049)
  • MediaWiki core, a case on its own: biggest project, unclear ownership, not all maintainers can review all areas.
  • Noise in metrics: dozens of basically unmaintained repositories containing 1-2 very old patches make difficult to dig into the really problematic repositories. See also T102112; T102920
  • Poor quality of patches: require a lot of effort to push forward, making reviewers to pass instead of review with -1.
  • ... ?

Metrics

Potential approaches and solutions

See preliminary results at https://www.mediawiki.org/wiki/User:AKlapper_%28WMF%29/Code_Review

Related Objects

StatusAssignedTask
Duplicate Qgil
Resolved Qgil
Resolved Qgil
Resolved Qgil
ResolvedDicortazar
Resolvedgreg
InvalidNone
InvalidNone
ResolvedAklapper
OpenNone
DeclinedNone
OpenNone
Resolved Qgil
ResolvedAklapper
ResolvedAklapper
ResolvedAklapper
ResolvedDicortazar
ResolvedDicortazar
ResolvedAcs
ResolvedDicortazar
ResolvedDicortazar
ResolvedDicortazar
ResolvedDicortazar
ResolvedDicortazar
ResolvedDicortazar
InvalidDicortazar
ResolvedDicortazar
ResolvedDicortazar
ResolvedAklapper
ResolvedDicortazar
ResolvedDicortazar
DuplicateNone
ResolvedDicortazar
ResolvedBawolff
Resolvedmmodell
ResolvedNone
Resolvedmmodell
ResolvedLegoktm
Resolvedtstarling
Resolvedgreg
ResolvedAklapper
ResolvedAklapper
ResolvedAklapper
ResolvedAklapper
ResolvedNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Aklapper claimed this task.Sep 1 2015, 2:08 PM
Aklapper updated the task description. (Show Details)
Aklapper added a project: DevRel-October-2015.
Jay8g added a subscriber: Jay8g.Sep 24 2015, 3:34 AM
Qgil added a comment.EditedSep 25 2015, 7:08 AM

An experiment: T113695: Clean the code review queue of analytics/wikistats. Developer-Advocacy could be in charge of filing a task like this one for the repository leading the classification of repositories with oldest open changesets.

EDIT: on a second thought, maybe not just the pole position, but the 10 slowest repositories. Different repos frequently affect different developers, and there is no reason not to ping #2 and #3 while #1 is being addressed.

Tgr added a comment.Oct 13 2015, 10:58 PM

There is a proposed WikiDev16 session about improving code review, especially for volunteers: T114419: Event on "Make code review not suck". You are welcome to comment there if interested.

Aklapper renamed this task from Goal: Reduce code review queues and waiting times to Goal: Define potential actions to reduce code review queues and waiting times.Dec 3 2015, 4:24 PM
Aklapper raised the priority of this task from Normal to High.Dec 3 2015, 4:45 PM
Aklapper moved this task from Backlog to Doing on the DevRel-December-2015 board.Dec 10 2015, 1:44 PM

It seems as though looking at queue size focuses our attention on "time to merge", whereas a more important metric for a lot of new contributors would be "time to first response". While both are obviously important, it seems that the most occurrences for long-term success of a contributor are:

  • Getting prompt, friendly feedback
  • Constructive coaching through the process of first merge

Any metrics we define in this area should measure artifacts of that healthy behavior. Obsessing about the size of the review queue encourages reviewers to dispose of bad code quickly. "Dispose" could mean "reject", or it could be "blindly merge"; either outcome is probably awful for new contributors and/or our codebase.

@Qgil and @Aklapper, do you all have metrics that emphasize the healthy behavior described?

Tgr added a comment.Dec 15 2015, 8:40 AM

Do reviewers have a way to know who is a new contributor? Finding a way to alerting people about that and/or to list recent patches by new contributors would be a good start.

Wikipedia invested a lot of effort into engaging and retaining newcomers; we could look at what work there and consider what would be the equivalent for new developers. E.g. how would mentors or a Teahouse look for MediaWiki?

(That said, while thinking about new developers is important, I don't think it is sufficient to fix our code review problem. Experienced developers are both hit worse (because they submit more complex patches which less people are able to review) and the productivity loss by failing to review their code is larger.)

Tgr added a comment.Dec 15 2015, 9:26 AM

Quoting myself from the WikiDev task:

It would be nice to run a survey asking volunteers (or everyone) about their experience as reviewers and reviewees, so that we have something more than opinions by the summit.

Do reviewers have a way to know who is a new contributor?

I use https://www.mediawiki.org/wiki/Gerrit/Reports/Open_changesets_by_newbie_owner

Tgr added a comment.Dec 15 2015, 9:33 PM

I wasn't aware of that, thanks!
Although if we focus on initial response times, some sort of "recent changesets by newbie owner" would be more helpful.

I wasn't aware of that, thanks!
Although if we focus on initial response times, some sort of "recent changesets by newbie owner" would be more helpful.

Interesting! Do we have a mechanism to assign "newbie owners" for newbies? "Newbie owner" is funny wording, since I'm going to guess that many people might take just as much/more offense to implying that they are "owned" as they would being called a "newbie". In the name of bikeshedding the naming: "new Wikimedia developer" and "mentor"?

Nemo_bis added a comment.EditedDec 16 2015, 9:03 AM

Although if we focus on initial response times, some sort of "recent changesets by newbie owner" would be more helpful.

I use the history/watchlist for that.

Robla, I don't understand what you are asking. "Owner" is simply the name gerrit uses (as opposed to author and committer). If you're asking about the procedure to act on the list, it's very simple: watchlist the page, check new stuff and review or add reviewers as needed. I've been doing this regularly since the report existed.

Interesting! Do we have a mechanism to assign "newbie owners" for newbies? "Newbie owner" is funny wording, since I'm going to guess that many people might take just as much/more offense to implying that they are "owned" as they would being called a "newbie". In the name of bikeshedding the naming: "new Wikimedia developer" and "mentor"?

I think "newbie owner" means that newbies 'own' changesets, not that newbies are 'owned' by someone else. But a mentoring program would be useful indeed :)

Are we actively looking for and training potential future reviewers?

Are we actively looking for and training potential future reviewers?

AFAIK training happens in gerrit. As for actively looking for potential +2'ers, I do, using http://koti.kapsi.fi/~federico/crstats/

Aklapper added a comment.EditedJan 2 2016, 1:01 AM

Preliminary results at https://www.mediawiki.org/wiki/User:AKlapper_%28WMF%29/Code_Review , to be fine-tuned in the next days but this task is pretty much done.

Closing after T114419 has happened.

Aklapper updated the task description. (Show Details)Jan 2 2016, 1:02 AM

The success criteria of this quarterly goal is:

Identify potential actions for WMF Engineering to address the code review queue of changesets submitted by volunteers (to propose and discuss with Eng teams, Team-Practices and volunteer maintainers in Q1/2016)

We need the status report for our imminent quarterly review.

I still have to go through the outcome of T114419 to finish this task (and same as for T114419: Define action items as subtasks if needed - likely to be the same ones).
Hence adding DevRel-January-2016 here.

Izno added a subscriber: Izno.Feb 29 2016, 2:15 PM
Aklapper moved this task from Backlog to Doing on the DevRel-March-2016 board.