For the record, I believe I already know how to pull data to check the first and second questions, but further insight would be appreciated.

@Jdlrobson if you have any links or documentation on this that would be super helpful :)

You can access this information in https://hue.wikimedia.org/
Example query: select * from event.mobilewebuiactionstracking where month = 4 and day = 13 and hour = 1 and event.action = 'click

We log "click" event on any mobile element which has data-event-name attribute
We log "init" events to allow you to determine sessions without any clicks.

Running Array.from($('[data-event-name]')).map((a) => a.getAttribute('data-event-name')) on any page in developer console will show you all the available name values.

In T305928#7851695, @Jdlrobson wrote:

You can access this information in https://hue.wikimedia.org/
Example query: select * from event.mobilewebuiactionstracking where month = 4 and day = 13 and hour = 1 and event.action = 'click

We log "click" event on any mobile element which has data-event-name attribute
We log "init" events to allow you to determine sessions without any clicks.

Running Array.from($('[data-event-name]')).map((a) => a.getAttribute('data-event-name')) on any page in developer console will show you all the available name values.

This is great, I'll take a look at this data soon and let you know if I have any questions, thanks!

Do you have any information about how you've done A/B testing in the past that might be helpful to guide us? I could imagine doing an A/B test with this overflow menu, for example, to explore the impact on new users. I'm not sure if standard practices are documented anywhere.

Do you have any information about how you've done A/B testing in the past that might be helpful to guide us?

Unfortunately, there is no general way to do A/B testing and in the past every A/B test has been bespoke. We've begun generalizing an approach in Vector with code inside Vector and WikimediaEvents, but that currently doesn't work on the Minerva (mobile skin).

The way this works - is we log an event to mediawiki.web_AB_test_enrollment with the name of the experiment, the web_session_id and the bucket group. We can then join events in any of the other tables on web_session_id to see what group the event occurred in.

It would be interesting to partner on a more generalized approach that also applies to Minerva. @phuedx has also been separately pushing us in that direction (See T298750)

Relevant code:

• https://github.com/wikimedia/Vector/blob/master/resources/skins.vector.es6/AB.js
• https://github.com/wikimedia/mediawiki-extensions-WikimediaEvents/blob/master/modules/ext.wikimediaEvents/webABTestEnrollment.js

In T305928#7852316, @Samwalton9 wrote:

I could imagine doing an A/B test with this overflow menu, for example, to explore the impact on new users. I'm not sure if standard practices are documented anywhere.

We already log the bucketised user edit count. If you're only looking to see if interaction with the AMC overflow menu varies with experience (assuming edit count is a proxy for experience), then we already have that data (and no one has to write any more code!).

As @Jdlrobson says, there's no standard library for running A/B test. I hope to start developing a prototype next quarter (Q1, FY22-23). That said, the instrument can be resurrected fairly easily and, since your cohort is logged-in users, you can easily vary the treatment by user ID using the modulo operator, i.e.

const TREATMENTS = [ 'control', 'a', 'b', 'c' ];

const treatment = TREATMENTS[ mw.user.getId() % TREATMENTS.length ];

Can we answer the above questions with the data currently available to us?

• Do administrators use the mobile web interface to delete and protect pages? Yes, it looks like both deletions and protections have the mobile web edit (and Advanced mobile edit) tags, so we should be able to query this.
• Do editors use the mobile web interface to move pages? As above, moves also have mobile tags.
• Do editors click the other buttons in the overflow menu? Which links are used most? Yes, from what I can tell the relevant name values for the overflow menu (as mentioned above) are ui.overflowmenu for opening the overflow, and then:

There is also a Download PDF button for articles, but it seems not to be logged.

• How many edits are being made from the mobile web interface by users who currently have AMC turned off? Kind of. We can easily query the number of edits being made with or without Advanced mobile edit, and look at user data for users who have edits with and without that tag. More directly, user preferences seem to be available via the MariaDB servers but the vast majority of user preferences are redacted in other data sources.

What metrics were removed as part of this patch? Do we need them again?

It looks like the new (current) schema just superseded the one removed here.

What other metrics are available to evaluate this feature, if any?

• MobileWebUIActionsTracking also contains data about interactions with other elements of the page, so we can get some contextual data for how overflow menu interactions compare with other elements of the page.

Worth noting that the sampling rate for this schema appears to be 10% for all projects except en.wiki, where it is 1%

I'm considering this done for now since the questions are answered; I'll follow up with the actual analysis, which I've started looking at.

Important caveat to Do editors click the other buttons in the overflow menu? Which links are used most? - all users see the overflow menu on user pages, so the links which are present on user pages feature most prominently in the data, simply because more users can see them. As far as I can tell the event schema doesn't record which page the user was on, so it's not currently possible to disentangle this data by, for example, only considering non-user pages.