Page MenuHomePhabricator

Homepage: HomepageModule schema
Closed, ResolvedPublic

Description

Requirements from draft measurement specifications document (requirements that are pasted in, but that are not related to the schema on this task, either because the feature described isn't built or because they will be measured some other way, are struck through):

  1. Modules
    1. General
      1. How often is each module clicked on? What is the proportion of users who click on each one?
      2. How many modules do users tend to interact with (where “interact” means any click)?
      3. Which modules tend to be interacted with repeatedly?
      4. Which modules are interacted with most by those users who are activated and retained (acknowledging that the causality may be going in either direction)?
      5. How long do desktop users hover over each module? (analogous measurement for mobile depends on pending mobile design)
      6. Does the placement of the module on the page appear to influence interactions?
      7. What is the impact on retention from doing the call-to-action on each module?
        1. For example: what percentage of users who contact their mentor are retained?
    2. Specific modules
      1. Help module
        1. What percent of users click one or more of the help links?
          1. If so, which link(s) do they click?
        2. What percent of users use the search functionality?
          1. 1: Focus on the search box
          2. 2: Submit a search, number of characters in search
          3. 3: Click a result, number of results, ranking of clicked result
        3. How far do users go in the workflow of asking a question on the Help Desk?
          1. 1: Click to ask a question
          2. 2: Type a question
          3. 3: Submit
          4. 4: Click a link in the confirmation dialog
        4. What is the average length of questions asked?
        5. What percent of users who post a question return to view the answer?
        6. What percent of newcomers that had no email address add one when asking a question?
        7. What percent of newcomers confirm their email address (within 48 hours) after asking a question?
        8. What percent of newcomers ask a question without an email address?
      2. Mentorship module
        1. What percent click on a link to learn more about the mentor?
        2. How far do users go in the workflow of asking their mentor a question?
          1. 1: Click to ask a question
          2. 2: Type a question
          3. 3: Submit
          4. 4: Click a link in the confirmation dialog
        3. What is the average length of questions asked?
        4. (Measured by hand) What percent of users who post a question receive an answer from their mentor?
          1. Do they get an answer from someone that is not their mentor?
        5. What percent of users who post a question appear to return to view the answer, by doing any of the following?
          1. Clicking the Echo notification saying they have been pinged in the response.
          2. Clicking on the link in the “Your recent questions” list.
          3. Responding to the answer on the talk page.
        6. (Measured by hand) What percent of newcomers who ask a question post a second time on their mentor’s talk page?
          1. How often does the first question become a conversation?
          2. How often is a second question asked?
          3. How often do conversations move beyond transactional wiki talk?
        7. What percent click on one of the links to view their question after they’ve asked it?
        8. What percent of newcomers that had no email address add one when asking a question?
        9. What percent of newcomers confirm their email address (within 48 hours) after asking a question?
        10. What percent of newcomers ask a question without an email address?
        11. Are newcomers more likely to ask questions to mentors who have a high edit count and short time since last edit compared to mentors with lower edit counts and/or longer time since last edit?
      3. Impact module
        1. What percent click on a link when the module is in its “unactivated state” (when the user has no edits to articles)?
        2. What percent click to view an article?
        3. What percent click to view the pageviews analysis tool?
        4. What percent click to view all their contributions?
        5. How often do users return to open the pageviews analysis tool multiple times?
      4. Start module
        1. What percent of users that had no email address add an email address through this module?
        2. What percent of users confirm their email address through this module?
        3. What percent of users click the button for the tutorial?
        4. What percent of users click the button to start their user page, and what percent of them actually save a user page?
        5. What percent of users click the link to learn more about creating a user page?

Proposed schema

1{
2 "description": "Logs interactions with the homepage modules on the Special:Homepage provided by the GrowthExperiments extension",
3 "properties": {
4 "action": {
5 "type": "string",
6 "required": true,
7 "enum": [
8 "impression",
9 "hover-in",
10 "hover-out",
11 "link-click"
12 ],
13 "description": "Action the user took:\n\n- impression: Module is shown to user\n\n- hover-in: User mouse pointer enters the element.\n\n- hover-out: User mouse pointer leaves the element\n\n- link-click: User clicks a link in the module"
14 },
15 "action_data": {
16 "type": "string",
17 "description": "Additional data for certain actions:\n\nFor action=link-click: symbolic name of the link being clicked. n.b. for impact module links, we will not append a number for each link in the list.\n\nFor mentorship module, we will add the mentor edit count and the unformatted timestamp that the editor was last active."
18 },
19 "user_id": {
20 "type": "integer",
21 "required": true,
22 "description": "The user id."
23 },
24 "user_editcount": {
25 "type": "integer",
26 "required": true,
27 "description": "The total number of edits of the user."
28 },
29 "module": {
30 "type": "string",
31 "enum": [
32 "start-account",
33 "start-tutorial",
34 "start-userpage",
35 "start-email",
36 "impact",
37 "help",
38 "mentorship"
39 ],
40 "required": true,
41 "description": "The module the user is interacting with"
42 },
43 "state": {
44 "type": "string",
45 "enum": [
46 "complete",
47 "incomplete",
48 "activated",
49 "unactivated",
50 "noemail",
51 "unconfirmed",
52 "confirmed"
53 ],
54 "required": false,
55 "description": "The state of the module the user is interacting with."
56 },
57 "is_mobile": {
58 "type": "boolean",
59 "required": true,
60 "description": "Whether the mobile site is being used."
61 },
62 "homepage_pageview_token": {
63 "type": "string",
64 "required": true,
65 "description": "One-time token per page load. This is a random user session ID that will be exported to the client-side when HomepageVisit events are recorded."
66 }
67 }
68}

Event Timeline

kostajh created this task.Mar 27 2019, 6:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 27 2019, 6:28 PM
kostajh renamed this task from Homepage: HomepageModule schema to Homepage: [DRAFT] HomepageModule schema.Mar 27 2019, 6:57 PM
kostajh updated the task description. (Show Details)
kostajh updated the task description. (Show Details)Mar 27 2019, 7:15 PM
Restricted Application added a subscriber: MusikAnimal. · View Herald TranscriptMar 27 2019, 7:15 PM
kostajh updated the task description. (Show Details)Mar 27 2019, 7:19 PM
kostajh updated the task description. (Show Details)
kostajh claimed this task.Mar 27 2019, 8:02 PM
kostajh edited subscribers, added: SBisson, Catrope, nettrom_WMF; removed: MusikAnimal.
kostajh updated the task description. (Show Details)Mar 28 2019, 1:36 PM

The schema looks good to me.

I see a number of things in the task description that are out of scope for now (like help panel search and varying homepage layout) or measured manually. Maybe it would add clarity to identify them as such.

Are newcomers more likely to ask questions to mentors who have a high edit count and short time since last edit compared to mentors with lower edit counts and/or longer time since last edit?

This can be done by adding the mentor's edit count and last active date to the mentor module's impression event and then find that event at analysis time or propagate those details and log them in action_data for all other events from the mentorship module.

kostajh renamed this task from Homepage: [DRAFT] HomepageModule schema to Homepage: [REVIEW] HomepageModule schema.Mar 28 2019, 3:46 PM
kostajh moved this task from Incoming to In Progress on the Growth-Team (Current Sprint) board.

Change 499873 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[mediawiki/extensions/GrowthExperiments@master] [poc] Track hover events on modules

https://gerrit.wikimedia.org/r/499873

kostajh renamed this task from Homepage: [REVIEW] HomepageModule schema to Homepage: HomepageModule schema.Mar 29 2019, 2:15 AM
kostajh updated the task description. (Show Details)
kostajh moved this task from In Progress to Needs PM Review on the Growth-Team (Current Sprint) board.
kostajh added a subscriber: MMiller_WMF.

@MMiller_WMF @nettrom_WMF I've updated the schema definition, please let me know if everything there looks OK.

@kostajh : This looks good to me, and I've gone through the measurement specification and updated it to reflect the current state of what we're capturing.

Change 499873 abandoned by Sbisson:
[poc] Track hover events on modules

https://gerrit.wikimedia.org/r/499873

MMiller_WMF updated the task description. (Show Details)Apr 4 2019, 10:51 PM

@SBisson -- I struck through the parts of the task description that are not relevant to the schema being built in this task.

@kostajh @nettrom_WMF -- here are my questions and comments:

  • Similarly to my comment on T219432, doesn't the email module have three states? You can need to add your email, need to confirm your email, or be complete.
  • Similarly to my comment on T219432, the HomepageVisits schema also records the states of modules, but does it differently with binaries. Is it bad to record the same sort of information in two different ways in two different schemas?
  • For the impact module, not only should we not record the number of the link in the list, we should also not record the actual link that was clicked. I wanted to make sure of that, since it wasn't mentioned in the "N.B." in the schema.
  • I see that "start" is not enumerated as a module in the list. Are we treating the overall "start" module as its own thing, or only the four sub-modules inside? I see that in Beta Wiki right now, the "start" module is its own thing.
  • Do you think we will run into any volume or throughput concerns with the "hover-in" and "hover-out" events? Now that I'm playing with this on Beta Wiki, I can easily trigger 10 events in a second. Will this cause any issues in Kafka or anything? Potentially a question for Analytics Engineering.

Similarly to my comment on T219432, doesn't the email module have three states? You can need to add your email, need to confirm your email, or be complete.

Commented in T219432, but since we treat the UX of email module as "complete" or "incomplete" that's what the representation in the schema shows. We can definitely change this, just let me know what you prefer to have in the data.

For the impact module, not only should we not record the number of the link in the list, we should also not record the actual link that was clicked. I wanted to make sure of that, since it wasn't mentioned in the "N.B." in the schema.

Right, this is not done. You'll either see linkId=impact-article-title;activated=true or linkId=impact-pageviews;activated=true in action_data

I see that "start" is not enumerated as a module in the list. Are we treating the overall "start" module as its own thing, or only the four sub-modules inside? I see that in Beta Wiki right now, the "start" module is its own thing.

We could have the start module parent container as a module, but what would be the goal? We would be recording a lot of duplicated hover-in/hover-out events.

Do you think we will run into any volume or throughput concerns with the "hover-in" and "hover-out" events? Now that I'm playing with this on Beta Wiki, I can easily trigger 10 events in a second. Will this cause any issues in Kafka or anything? Potentially a question for Analytics Engineering.

Yes that's a question for analytics engineering. My sense is that since we are showing this to newly created users on a specific page requiring login, we're unlikely to create a flood of events that would overwhelm the infrastructure but definitely a good idea to check before it goes live :)

Similarly to my comment on T219432, doesn't the email module have three states? You can need to add your email, need to confirm your email, or be complete.

Commented in T219432, but since we treat the UX of email module as "complete" or "incomplete" that's what the representation in the schema shows. We can definitely change this, just let me know what you prefer to have in the data.

Do we want to add a field to this schema with the state of the module, either an enum like we did for HomepageVisit, or a string (for more flexibility)? That would turn the module name into a field that only records the module's name and not its state (as happens with the impact module), and there's no split between module and action_data when it comes to where state information is kept. After thinking about this a bit more, having these in separate and consistent fields could make things easier to work with, particularly as the number of modules and/or states increases.

MMiller_WMF added subscribers: Ottomata, Milimetric.

Thanks, @kostajh and @nettrom_WMF.

Like I said on T219432, I do think we should record all three states of the email module (however you want to do it is fine with me).

I agree that the start module does not need to recorded as its own module. But I do want to point out that in Beta Wiki right now, "hover-in" and "hover-out" events are being recorded for the "start" module separately from the four module inside it.

I'm putting this back In Progress to make the two changes above.

@Milimetric @Ottomata -- pinging you here because of our potential throughput concerns. In this EventLogging schema, we're recording "hover-in" and "hover-out" events for 11 different rectangles on Special:Homepage. If a user moves their mouse around quickly, they could easily fire like 10 events in a second. We'll be giving the feature (and accompanying instrumentation) to some thousands of users over the next several months. Do you think this could cause issues?

Assuming worst case 'thousands' of users == 10000 and they all do 10 events per second, that would be a problem. The largest volume schema right now is CitationUsagePageLoad at around 800 events per second total. I think you should aim to keep the volume at much lower than that if you can. 300ish events / second would be good. We can probably handle more, but that also means that we'll just be using up our capacity faster and eventually not be able to accommodate new schemas.

EventLogging events are processed by a single server and all schemas share it. Modern Event Platform work will help us scale this, but we won't have that up for external clients until probably Q2ish next FY.

BTW, I had a thought about the schema design while skimming this ticket. In the meeting we had a month (or two?) ago, @netrom_WMF asked about the pros and cons using the same schema for different event types. I good rule of thumb that I think I didn't mention is this: If it doesn't make sense to aggregate over the different event types, then they should probably be different events. IIUC, here it seems ok, because you could e.g. average the total events per second as something like 'interactions per second'?

@nettrom_WMF @MMiller_WMF I updated the schema https://meta.wikimedia.org/w/index.php?title=Schema:HomepageModule&oldid=18998619, could you please take a look and let me know if that is OK before I update the code?

Updated schema per discussion with @nettrom_WMF https://meta.wikimedia.org/w/index.php?title=Schema%3AHomepageModule&type=revision&diff=18999329&oldid=18998619

Assuming worst case 'thousands' of users == 10000 and they all do 10 events per second, that would be a problem. The largest volume schema right now is CitationUsagePageLoad at around 800 events per second total. I think you should aim to keep the volume at much lower than that if you can. 300ish events / second would be good. We can probably handle more, but that also means that we'll just be using up our capacity faster and eventually not be able to accommodate new schemas.

@Ottomata, @nettrom_WMF and I discussed this for a bit today. We propose to sample the the over-in/hover-out interactions, such that 10% of all visits to Special:Homepage have hover-in/hover-out logging enabled. Does that seem like it would be OK? That way, we will have sessions with a "full" set of data for us to analyze interaction time, but the majority of sessions would have far fewer events logged.

Hm, my math was for worst case based on your estimate of 'thousands of users'. 10% of 10000 is 1000, and if they are each generating 10 events per second, that's still 10000 events per second. I know 10K users each doing 10 per second is a huge overestimate, but still it might be good to aim smaller at first. 0.3% sampling would get you down to 300 / second there (10000*1000*0.003 == 300) . If you think 'thousands of users' is more like 3K,  then 1% would be about 300 events per second.

Maybe it'd be good to try and get a better estimate of volume? Page interactions will be hard to guess at I know, but perhaps you could better estimate number of active users?

Change 502545 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Homepage logging: Track state and action data separately

https://gerrit.wikimedia.org/r/502545

Change 502545 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Homepage logging: Track state and action data separately

https://gerrit.wikimedia.org/r/502545

Both @MMiller_WMF and I did some back-of-the-envelope estimation of active users today, and reach roughly the same conclusions, that we'll be well below thousands of active users at a given time. I based mine on the number of users who register on these wikis and used a generous estimate, and find that we might be looking at 10 visits per minute.

That being said, I still think we can start by sampling, and then later adjust our sample based on the level of activity we see.

Here's how I got my estimate:
Across the three wikis, we're looking at roughly 2,000 (Czech), 2,000 (Korean), and 6,500 (Vietnamese) registrations per month (March 2019 registrations from WikiStats 2.0), for a total of 10,500. Half of these won't have the homepage enabled since we're running an A/B test, so we're down to 5,250.

Out of these, we know that 15–30% of them visit their user page during the first 24 hours (T217624). I want to overestimate, so I'll simplify and say that 1/3 of them do, giving us 1,750 visits to the homepage on their first day.

We'll assume half of them come back and visit the homepage the next day, meaning we add another 875 visits. The amount of users who come back after that depends on the long tail of visits. Because the Homepage is largely geared towards users who edit, I'll estimate it based on the shape of the retention curve that I've recently calculated for Czech and Korean (in a different project). For the first week, the retention is 10% (which I'll ignore since we already said 50% visit on the second day), 5%, 2.5%, 2%, 2%, 3%. The second week is largely 1.5%. That results in another 25%, but I'll be very generous and up it to 50% and add another 875 visits.

That brings us to 3,500 visits per day. If we assume users are inactive for about 8 hours, that gives us 16 active hours and 200 visits per hour. If we also assume that this average isn't a flat line, I'll triple the estimate for the more active hours and land at 600 visits. That's about 10 per minute.

Ottomata added a comment.EditedApr 10 2019, 1:23 PM

Thanks! So ~10 visits / minute, with a very high guess of 10 interactions per user per second is 1.67 events per second? Sounds easy peasy to me then :)

Agreed, it looks like we'll stay well below the limit. The team discussed this today, and I've added to our list of leading indicators to also measure the volume of hover events, to make sure that we stay below the limits. That will then also provide us with an indication of how this will scale when adding additional wikis.

Nothing that I updated P8309 with the current state of the schema, so that's also reflected here.

The current limitations and open questions for hover-in.hover-out events are summarized in Homepage hovers issues and ideas.

One of the scenarios illustrating the issue "Some kinds of link and button clicks cause a hover-out after the click and some don’t" described in the document above will be the following:

actionlogged
(1)On Homepage move your mouse in Help moduleevent.HomepageModule action: "hover-in" module: "help"
(2)Click on "Ask the help desk"event.HelpPanel action: "review" editor_interface: "homepage_help"
(3)no user actionsevent.HomepageModule action: "hover-out" module: "help"
(4)Start typing in the question fieldevent.HelpPanel action: "enter-question-text" editor_interface: "homepage_help"
(5)Click to submit the questionevent.HelpPanel action: "submit-attempt" module: "help"
(6)event.HelpPanel action: "submit-success" module: "help"
(7)Close the Help dialogevent.HelpPanel action: "close" editor_interface: "homepage_help"
(8)no user actionsevent.HomepageModule action: "hover-in" module: "start-email"

Moving to Needs PM review column.

MMiller_WMF closed this task as Resolved.Jul 3 2019, 11:51 PM

Data is getting stored correctly and being used for reporting.