In T257577: Investigate TwoColConflict opt-out metrics, explain whether there's a trend, we found 600,000 users opted-out of the TwoColConflict beta feature. These must be phantom opt-outs. Investigate:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | awight | T260353 Repair damage caused by misbehaving settings migration | |||
| Resolved | Lena_WMDE | T260250 Investigate TwoColConflict opt-out bug |
The relevant piece of code is, I believe, this one here where we migrate the user's Beta setting to the real one: https://phabricator.wikimedia.org/diffusion/ESCC/browse/master/includes/Hooks/TwoColConflictHooks.php$340. The assumption here is that the $options array might still contain a value for the Beta preference. We migrate this value only when it is the number 0 (but not when it's the string "0"), and then delete it.
There are 2 sources where this value can come from:
I started manually testing different scenarios, cleaning the database first:
In other words: We have no idea if a 0 in the database means that the user actively disabled the Beta feature, or just never touched it. The "migration" we do right now means we disable the feature for all users that made some unrelated changes to their preferences since TwoColConflict is a Beta feature. That's certainly not all of them, but probably a high number. Probably exactly the number we see. :-(
Thank you for the initial investigation into opt-out scenarios!
Could we also investigate whether there's some way to distinguish between the intentional and unintential opt-outs within the database? In other words, can we determined a list of users who made the active choice to opt-out? This would apply both for past data and future data. If it's not possible for past data/current data being collected, then is there a way for us to start collecting this data from now going forward?
I believe there is no way. See, when you visit your preferences, and there is a new Beta feature (which is off by default, as it should be), how should anyone know what it means when you don't touch that checkbox? Was this intentional?
Even if we somehow track users that enable a Beta feature and disable it later, do we really know what this means? Maybe it was a mistake and the user never ever experienced the new TwoColConflict interface? Maybe it was a very early stage and the user disabled it because of a bug that is long fixed? Especially the later is – in my experience – extremely common. When users say they don't like a new thing it's typically because of one or a few very minor issues. Sometimes as minor as a bad localization. I believe the majority of the decisions we see are something like this, and we should not keep these forever, as if the feature we actively develop never changed since these few users furiously disabled a – by definition – incomplete Beta feature.
I'm optimistic about this. The preference form is currently not designed to preserve this information, but there's a very good argument for why this is important, and I can imagine trivial implementations such as making the "tri-state" visible, so explicit vs implicit opt-out is detectable. Event handlers in the form and comparing form entry and exit states should make it possible to see a real opt-out. Will require coordination with the BetaFeatures maintainers.