This is a followup to T88531: Goal: Organize a Gerrit Cleanup Day on September 23, 2015 and is to be incorporated in T115844: Have a second Gerrit (Code Review backlog) cleanup day.
Regarding the Cleanup Day itself
Communication before the Day
- ☑ Weekday: Chose Wednesday to not end up merging changes before branching / deployment train
- Timing: End of quarter isn't great as teams are busy anyway. Also note that some teams do have high-priority time-sensitive tasks.
- Agreeing on a Cleanup date and getting commitment seems to be easier in a WMF Engineering management meeting instead of sending out emails
- TODO: Add a banner on Gerrit as [[ https://gerrit-review.googlesource.com/Documentation/config-themes.html#_html_header_footer | a theme with GerritSiteHeader.html ]]
- ☑ Edited Phabricator default dashboard to display announcement on homepage
- TODO: Request a banner on mediawiki.org
- TODO: Add an event to Phabricator calendar
- ☑ Sent early email to wikitech-l, pywikibot, ops, wikidata mailing lists
- ☑ Sent email to internal WMF Engineering list
- ☑ Added entry to internal WMF Engineering calendar
- ☑ A week before, contacted teams (mailing lists and team leaders if existing) whether they are ready or not help (e.g. Pywikibot, Wikidata)
- TODO: Some partially rather unused lists (e.g. performance@) were contacted as no better/clearer ones were found or existed
- TODO: Must send earlier than one week: Ended up in some moderation queues without getting timely clearance; plus hard to find out names of some mailing list (e.g. fr-tech)
- ☑ Two days before, sent reminder to wikitech-l
- TODO: More widely announce, e.g. tech-ambassadors@? Where is our extended target audience?
Social aspects
- Different levels of team preparation and enthusiasm. Examples: Discovery team planning page; Yuvi's internal ops@ email listing potential reviewers for patches; RelEng preparations.
- Different understandings of expectation of joining among teams or individuals ("is it really mandatory to join?")
- "I did some reviews, but basically our team considers itself too busy to participate. More accurately: When it was discussed, it was left up to each developer to decide themselves whether they want to participate. I wish we would have done more."
- Basically no cross-team interaction seen in #wikimedia-dev (or it happened in other places?), everybody sticking to their areas.
- TODO: List of main contacts (Name, IRC nick, IRC channel) to also include timezone info?
- TODO: No formal buddy system was created - wikipage with section "I'm looking for a review-buddy"? Europeans (early timezone) with only few changesets: How can they help others and "find a buddy" in their timezone? How to improve involving volunteers, also those who realize on IRC that this is going on right now?
- TODO: Praise: Stats to highlight the most active/effective developers / repositories / teams during the Day. More efficient communication structure welcome. No clear numbers on participation of WMF teams and who exactly was (not) involved. Asking "How is the Cleanup Day going? How can we help?" in many IRC channels is not the most efficient way, checking with each main contact (timezones!) is neither; following the Gerrit notifications is cumbersome.
General Code Review aspects / issues
- Created initial Gerrit queries per team/area in Phabricator task for the Cleanup Day; requested help for improvement from each team. Mixed results: Some teams came up with better queries or fixed their list of projects, no feedback from other teams. Improve queries (and responsibilities). Related in WMF context: T115854: Document code area responsibilities (Phab projects, Git repos) of each WMF engineering/dev team
- Some teams mentioned that initial queries included deprecated projects
- Allow marking inactive repositories as such: T102920
- Potential "not our responsibility / NIH" attitude of individuals though repository is a related area
- unclear responsibility for cross-areas patches, e.g. touching i18n parts (L10N eng?) of extensions (extension maintainer?) like 229078?
- T115852: Fix unclear maintenance responsibilities for some parts of MediaWiki core repository: Unclear responsibility for parts of MediaWiki core so people might avoid looking at its patches and rather stick to their "easier to find" areas?
- "Is there a list of deployed but not really owned repos that people could prefer looking at patches in?". Regarding MW Core: "I was hoping to find something less intimidating to point some newer teammates at". Related in WMF context: T115854: Document code area responsibilities (Phab projects, Git repos) of each WMF engineering/dev team
- Better stats welcome to highlight the most active/effective developers / repositories / teams in korma.wmflabs.org.
- Have a policy to review contributions within x days?: T113707: Decision on Wikimedia Foundation service-level agreement on code review times
- T114419: Event on "Make code review not suck" (WikiDev16 talk)
- Technical restrictions in Gerrit:
- [[ https://phabricator.wikimedia.org/T63463 | file: queries in gerrit not enabled ]], hence not possible to search specific paths, e.g. of the MediaWiki core codebase
- T115851: Reviewing a patch in Gerrit and removing yourself from reviewers list deletes the review flag: When someone reviews a patch in Gerrit (e.g. CR-2) and then removes themself from the list of reviewers, that CR-2 is removed too. Numerous examples exist: 190717, 120811, 175394, 175192, 171562, etc.
- T115850: Decide/document whether "controversial" changes should be (temporarily) marked as CR-2 until wider agreement reached: Document to also mark "controversial" changes as CR-2 (example)?
- Document counterproductive actions? Mass comments like "-1 needs manual rebase" are not helpful ("The -1 discourages other reviewers, without actually checking the substance of the patch. And Gerrit's 'Can't merge' indicator is often broken")
- Define a cut-off date for old tasks to abandon them (can be restored)?: T113706: Define Cut-off date for abandoning old changesets in our code review tool
- As @Reedy wrote, "half tempted to abandon stuff pre 2014 that's untouched, that's sitting CR-1/V-1. 29 patches in mw core alone." - have some discussion/policy for that? See past discussions: http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/60150/ , http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/65931/ , http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/76459/ , http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/76640/
- Have review comment template(s) for Gerrit? Like "Hi $Username, sorry that your proposed code changes have not received a timely review. Insert reason for CR-1, e.g. manual rebase. Would you fancy updating your patch? See https://www.mediawiki.org/wiki/Gerrit/Tutorial#Amending_a_change_.28your_own_or_someone_else.27s.29 for more information."
- Technical threats to validity of numbers:
- Stats on Korma need improvement on user affiliation data: unreliable "Independent/Unknown" data after fixing T100189 (more people without affiliation than actually the case due to non-official email addresses used) but we did not go through the list of accounts afterwards. T112527: Correct affiliation for code review contributors of the past 30 days
- T112661: "Oldest open Gerrit changesets without code review" panel should filter WIP etc
- T108507: Patches with Verified -1 should not be counted as open in our code review metrics
Evaluation
- ☑ We gathered Korma data to compare one day before and after Cleanup Day (though at 00:00 UTC / 17:00 PST hence might be cut-off at the end):
- Total number of changesets waiting for a reviewer action and Unknown affiliation: 628 (was 751 ~ -17%)
- Total number of changesets waiting for a reviewer action and Independent affiliation: 124 (was 159 ~ -23,1%)
- Total number of changesets waiting for a reviewer action, Unknown affiliation and open in the last three months: 261 (was 335 ~ -22%)
- Total number of changesets waiting for a reviewer action, Independent affiliation and open in the last three monthts: 53 (was 71 ~ -25,4%
Additional statistics show a 10 % decrease in open patches:
- ~200 changesets closed, half of which by abandoning. https://www.mediawiki.org/w/index.php?title=Gerrit/Reports/Changesets_by_status&diff=prev&oldid=1897960
- ~100 changesets closed the following day, also half of which by abandoning (in good part thanks to Hashar&Siebrand cleaning up core and i18n respectively). https://www.mediawiki.org/w/index.php?title=Gerrit/Reports/Changesets_by_status&diff=next&oldid=1897960
Regarding the measurements of success, we have failed:
- 👎 All WMF developer teams join the Day.
- 👎 All patches contributed by volunteers and all patches from the past 3 months have at least one review.