Presented the results yesterday at Release Engineering's lunch and learn:

• Slides: https://docs.google.com/presentation/d/12K7k0LkgxK8ovPM0zLaK7Q9az7jZca0tAV91uwSIib8/edit#slide=id.p
• Filenames that cause blockers spreadsheet: https://docs.google.com/spreadsheets/d/15xO-y0SopWvpjmztwvrvZf9z2V8NJ3gtufkCzjDApjE/edit#gid=1646264321
• Notebook with more insights: https://github.com/mirrys/release-engineering-data/blob/main/metrics_reng_data.ipynb

Todos after the meeting:

• Rethink the final metric including all signals that cause overhead work for the team: bugs, rollbacks, blockers, with special focus on the ones happening on the third day of deployment.
• Analyze more in-depth the filenames and their relation with train delays, to potentially come up with a list of "problematic filenames" which can suggest potentially problematic patches.

How are filenames associated with bugs? For example mediawiki/extensions/GrowthExperiments/modules/homepage/addlink/RecommendedLinkToolbarDialog.js is listed as being associated with 9 bugs. I see on https://data.releng.team/train/bug_file?_sort=id&filename__exact=modules%2Fhomepage%2Faddlink%2FRecommendedLinkToolbarDialog.js that there are 5 entries. Four of those entries are a patch to master + a backport to a wmf branch (e.g. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/695039) ; should those be counted as distinct bugs in the tally? Or should a fix to master plus a backport count as a single bug?

In T289567#7714296, @kostajh wrote:

How are filenames associated with bugs? For example mediawiki/extensions/GrowthExperiments/modules/homepage/addlink/RecommendedLinkToolbarDialog.js is listed as being associated with 9 bugs. I see on https://data.releng.team/train/bug_file?_sort=id&filename__exact=modules%2Fhomepage%2Faddlink%2FRecommendedLinkToolbarDialog.js that there are 5 entries. Four of those entries are a patch to master + a backport to a wmf branch (e.g. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/695039) ; should those be counted as distinct bugs in the tally? Or should a fix to master plus a backport count as a single bug?

I'm not sure about the discrepancy between the 9 bugs and 5 bugs.

I can answer what a bug is for the purpose of train-stats.

The way I found "bugs" is for each train version:

1. Find all backports to that version
2. Check if there is a Bug trailer in the backport
3. If the phabricator task is a subtype of "bug" or "error" then it's a bug

This algorithm has two problems:

1. It double-counts backports to multiple branches of the same patch
2. It under-counts real bugs because: (a) some bugs don't have an associated task (b) some backports fail to mention the task (c) sometimes a bug or error report is not a subtype of "bug" or "error report"

The primary advantage is that it is certain that every backport it counts is a real bug and not a feature or feature flag or some other innocuous deployment. I'd be interested in thoughts about how to programatically fix problem 1