I fixed the bug in the data gathering. revision_tags in mediawiki_history is NULL if an edit has no tags. To exclude edits with a specific tag (or set of tags) we therefore either need to use coalesce() to make it an empty array if it's NULL, or have a clause that tests and allows NULL-values before checking for the presence of a given tag.

After regathering the data and rerunning the analysis, I found that the conclusions were the same, users in the Homepage group make fewer edits than the treatment group when only counting non-tagged edits. I have not bothered to calculate specific estimates on average number of edits. That's partly because the key question here is how we explain an 85% increase in productivity given the existing analysis, and partly because we can only do this non-tagged analysis using data from when the variant test was running, and all our signals suggests the variant test had a negative impact on productivity. To get better estimates, we'll therefore instead want to rerun this analysis with a different cohort.

Coming back to the key question, we've made a new estimate using predictions from the productivity model for edits in Article & Article talk across both the activation and retention period. Looking squarely across all wikis and platforms (desktop/mobile), we grabbed posterior predictions from the model. Using that approach, Control group users are estimated to make 1.40 edits in that time period, and Homepage group users are estimated to make 1.70 edits, for a gain of 21.7%. Note that this estimate treats all wikis as equal, rather than weigh them by number of registrations. I'll be looking into making a weighted estimate instead, but don't see that as blocking the completion of this task. Also, an average of 1.4/1.7 edits might seem high, but that's because we're using a zero-inflated model. This means that a lot of users who register but don't make an edit are characterized as "never going to edit anyway" and discarded from the prediction. The result of that is that the average is higher than if they were included, of course.

This 1.4/1.7 edit estimate is now what we use in the analysis report.