The schema looks good to me.

I see a number of things in the task description that are out of scope for now (like help panel search and varying homepage layout) or measured manually. Maybe it would add clarity to identify them as such.

Are newcomers more likely to ask questions to mentors who have a high edit count and short time since last edit compared to mentors with lower edit counts and/or longer time since last edit?

This can be done by adding the mentor's edit count and last active date to the mentor module's impression event and then find that event at analysis time or propagate those details and log them in action_data for all other events from the mentorship module.

Change 499873 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[mediawiki/extensions/GrowthExperiments@master] [poc] Track hover events on modules

https://gerrit.wikimedia.org/r/499873

@MMiller_WMF @nettrom_WMF I've updated the schema definition, please let me know if everything there looks OK.

@kostajh : This looks good to me, and I've gone through the measurement specification and updated it to reflect the current state of what we're capturing.

Change 499873 abandoned by Sbisson:
[poc] Track hover events on modules

https://gerrit.wikimedia.org/r/499873

@SBisson -- I struck through the parts of the task description that are not relevant to the schema being built in this task.

@kostajh @nettrom_WMF -- here are my questions and comments:

• Similarly to my comment on T219432, doesn't the email module have three states? You can need to add your email, need to confirm your email, or be complete.
• Similarly to my comment on T219432, the HomepageVisits schema also records the states of modules, but does it differently with binaries. Is it bad to record the same sort of information in two different ways in two different schemas?
• For the impact module, not only should we not record the number of the link in the list, we should also not record the actual link that was clicked. I wanted to make sure of that, since it wasn't mentioned in the "N.B." in the schema.
• I see that "start" is not enumerated as a module in the list. Are we treating the overall "start" module as its own thing, or only the four sub-modules inside? I see that in Beta Wiki right now, the "start" module is its own thing.
• Do you think we will run into any volume or throughput concerns with the "hover-in" and "hover-out" events? Now that I'm playing with this on Beta Wiki, I can easily trigger 10 events in a second. Will this cause any issues in Kafka or anything? Potentially a question for Analytics Engineering.

Similarly to my comment on T219432, doesn't the email module have three states? You can need to add your email, need to confirm your email, or be complete.

Commented in T219432, but since we treat the UX of email module as "complete" or "incomplete" that's what the representation in the schema shows. We can definitely change this, just let me know what you prefer to have in the data.

For the impact module, not only should we not record the number of the link in the list, we should also not record the actual link that was clicked. I wanted to make sure of that, since it wasn't mentioned in the "N.B." in the schema.

Right, this is not done. You'll either see linkId=impact-article-title;activated=true or linkId=impact-pageviews;activated=true in action_data

I see that "start" is not enumerated as a module in the list. Are we treating the overall "start" module as its own thing, or only the four sub-modules inside? I see that in Beta Wiki right now, the "start" module is its own thing.

We could have the start module parent container as a module, but what would be the goal? We would be recording a lot of duplicated hover-in/hover-out events.

Do you think we will run into any volume or throughput concerns with the "hover-in" and "hover-out" events? Now that I'm playing with this on Beta Wiki, I can easily trigger 10 events in a second. Will this cause any issues in Kafka or anything? Potentially a question for Analytics Engineering.

Yes that's a question for analytics engineering. My sense is that since we are showing this to newly created users on a specific page requiring login, we're unlikely to create a flood of events that would overwhelm the infrastructure but definitely a good idea to check before it goes live :)

In T219435#5088554, @kostajh wrote:

Similarly to my comment on T219432, doesn't the email module have three states? You can need to add your email, need to confirm your email, or be complete.

Commented in T219432, but since we treat the UX of email module as "complete" or "incomplete" that's what the representation in the schema shows. We can definitely change this, just let me know what you prefer to have in the data.

Do we want to add a field to this schema with the state of the module, either an enum like we did for HomepageVisit, or a string (for more flexibility)? That would turn the module name into a field that only records the module's name and not its state (as happens with the impact module), and there's no split between module and action_data when it comes to where state information is kept. After thinking about this a bit more, having these in separate and consistent fields could make things easier to work with, particularly as the number of modules and/or states increases.

Thanks, @kostajh and @nettrom_WMF.

Like I said on T219432, I do think we should record all three states of the email module (however you want to do it is fine with me).

I agree that the start module does not need to recorded as its own module. But I do want to point out that in Beta Wiki right now, "hover-in" and "hover-out" events are being recorded for the "start" module separately from the four module inside it.

I'm putting this back In Progress to make the two changes above.

@Milimetric @Ottomata -- pinging you here because of our potential throughput concerns. In this EventLogging schema, we're recording "hover-in" and "hover-out" events for 11 different rectangles on Special:Homepage. If a user moves their mouse around quickly, they could easily fire like 10 events in a second. We'll be giving the feature (and accompanying instrumentation) to some thousands of users over the next several months. Do you think this could cause issues?

Assuming worst case 'thousands' of users == 10000 and they all do 10 events per second, that would be a problem. The largest volume schema right now is CitationUsagePageLoad at around 800 events per second total. I think you should aim to keep the volume at much lower than that if you can. 300ish events / second would be good. We can probably handle more, but that also means that we'll just be using up our capacity faster and eventually not be able to accommodate new schemas.

EventLogging events are processed by a single server and all schemas share it. Modern Event Platform work will help us scale this, but we won't have that up for external clients until probably Q2ish next FY.

BTW, I had a thought about the schema design while skimming this ticket. In the meeting we had a month (or two?) ago, @netrom_WMF asked about the pros and cons using the same schema for different event types. I good rule of thumb that I think I didn't mention is this: If it doesn't make sense to aggregate over the different event types, then they should probably be different events. IIUC, here it seems ok, because you could e.g. average the total events per second as something like 'interactions per second'?

@nettrom_WMF @MMiller_WMF I updated the schema https://meta.wikimedia.org/w/index.php?title=Schema:HomepageModule&oldid=18998619, could you please take a look and let me know if that is OK before I update the code?

Updated schema per discussion with @nettrom_WMF https://meta.wikimedia.org/w/index.php?title=Schema%3AHomepageModule&type=revision&diff=18999329&oldid=18998619

Assuming worst case 'thousands' of users == 10000 and they all do 10 events per second, that would be a problem. The largest volume schema right now is CitationUsagePageLoad at around 800 events per second total. I think you should aim to keep the volume at much lower than that if you can. 300ish events / second would be good. We can probably handle more, but that also means that we'll just be using up our capacity faster and eventually not be able to accommodate new schemas.

@Ottomata, @nettrom_WMF and I discussed this for a bit today. We propose to sample the the over-in/hover-out interactions, such that 10% of all visits to Special:Homepage have hover-in/hover-out logging enabled. Does that seem like it would be OK? That way, we will have sessions with a "full" set of data for us to analyze interaction time, but the majority of sessions would have far fewer events logged.

Hm, my math was for worst case based on your estimate of 'thousands of users'. 10% of 10000 is 1000, and if they are each generating 10 events per second, that's still 10000 events per second. I know 10K users each doing 10 per second is a huge overestimate, but still it might be good to aim smaller at first. 0.3% sampling would get you down to 300 / second there (10000*1000*0.003 == 300) . If you think 'thousands of users' is more like 3K,  then 1% would be about 300 events per second.

Maybe it'd be good to try and get a better estimate of volume? Page interactions will be hard to guess at I know, but perhaps you could better estimate number of active users?

Change 502545 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Homepage logging: Track state and action data separately

https://gerrit.wikimedia.org/r/502545

Change 502545 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Homepage logging: Track state and action data separately

https://gerrit.wikimedia.org/r/502545

Both @MMiller_WMF and I did some back-of-the-envelope estimation of active users today, and reach roughly the same conclusions, that we'll be well below thousands of active users at a given time. I based mine on the number of users who register on these wikis and used a generous estimate, and find that we might be looking at 10 visits per minute.

That being said, I still think we can start by sampling, and then later adjust our sample based on the level of activity we see.

Here's how I got my estimate:
Across the three wikis, we're looking at roughly 2,000 (Czech), 2,000 (Korean), and 6,500 (Vietnamese) registrations per month (March 2019 registrations from WikiStats 2.0), for a total of 10,500. Half of these won't have the homepage enabled since we're running an A/B test, so we're down to 5,250.

Out of these, we know that 15–30% of them visit their user page during the first 24 hours (T217624). I want to overestimate, so I'll simplify and say that 1/3 of them do, giving us 1,750 visits to the homepage on their first day.

We'll assume half of them come back and visit the homepage the next day, meaning we add another 875 visits. The amount of users who come back after that depends on the long tail of visits. Because the Homepage is largely geared towards users who edit, I'll estimate it based on the shape of the retention curve that I've recently calculated for Czech and Korean (in a different project). For the first week, the retention is 10% (which I'll ignore since we already said 50% visit on the second day), 5%, 2.5%, 2%, 2%, 3%. The second week is largely 1.5%. That results in another 25%, but I'll be very generous and up it to 50% and add another 875 visits.

That brings us to 3,500 visits per day. If we assume users are inactive for about 8 hours, that gives us 16 active hours and 200 visits per hour. If we also assume that this average isn't a flat line, I'll triple the estimate for the more active hours and land at 600 visits. That's about 10 per minute.

Thanks! So ~10 visits / minute, with a very high guess of 10 interactions per user per second is 1.67 events per second? Sounds easy peasy to me then :)

Agreed, it looks like we'll stay well below the limit. The team discussed this today, and I've added to our list of leading indicators to also measure the volume of hover events, to make sure that we stay below the limits. That will then also provide us with an indication of how this will scale when adding additional wikis.

Nothing that I updated P8309 with the current state of the schema, so that's also reflected here.

The current limitations and open questions for hover-in.hover-out events are summarized in Homepage hovers issues and ideas.

One of the scenarios illustrating the issue "Some kinds of link and button clicks cause a hover-out after the click and some don’t" described in the document above will be the following:

Moving to Needs PM review column.