Discussed the Growth team metrics with @MMiller_WMF and updated the table with links to Growth's reports.

In T226208#5272031, @kostajh wrote:

The homepage_pageview_token is different, and you should not see any other events for the second visit token since the user is now on the tutorial title.

Thought I'd respond to this and make it actionable in case this moves forward:

We've calculated this for our target wikis individually, and now also for analyzing effects of interventions across multiple wikis with multilevel models (using simulations to estimate the statistical power). For the latter, we also simulated scenarios with additional wikis to understand whether those would allow us to detect smaller effect sizes in a reasonable amount of time.

Upon reviewing this patch, @mforns raised concerns about the information we store about the mentors in this schema. It likely makes it possible to connect mentor and mentee, so the suggestion is to bucket edit counts. Thinking about this, we might want to bucket "time since last activity" as well to make both pieces of information less identifying.

@MMiller_WMF : Sure! I've added counts of the number of users who had events on desktop and mobile in EditorJourney, to account for any users who might've switched between sites. Then the table and the percentages work out like this:

I restricted the query to visits to their contributions specifically. This uses the 90 days worth of data from EditorJourney, and is limited to non-autocreated accounts that were not registered through the API, and that were not known test accounts by Growth Team members. A user can be counted twice by visiting the page on both desktop and mobile, but the number of users who do so might be rather small.

Verified that this is now deployed and data is retained/purged as expected.

In T222836#5241803, @SBisson wrote:

A lot depend on the module names in the code so I would prefer to not suffix them.

In T222836#5239817, @SBisson wrote:

I think we can add information to the "impression" action of HomepageModule to specify whether the impression is

• "desktop" (like it is now)
• "mobile summary" (the shorter preview shown on the mobile homepage)
• "mobile details" (the full mobile version, shown in a subpage currently but will be in a mobile overlay when T222833 is done)

In T216220#5238107, @SBisson wrote:

@nettrom_WMF, @MMiller_WMF any thoughts about where you want to see the event logging data for opening and closing that tooltip? I was thinking we could stuff it as new actions in HomepageModule. Something like open-nopageviews-tooltip and close-nopageviews-tooltip (or anything else you would prefer). The mobile flag would tell us if the open/close are caused by taps (mobile) or hover (desktop).

@MMiller_WMF : I've drafted some suggested addition to the measurement specification, could you take a look at those and let me know if further changes are needed? Also, I updated this task's description so it mentions that we'd also like to capture clicks on the "back" buttons in the modules on the mobile site, because that kind of interaction isn't available on the desktop site.

In T222836#5229868, @Cntlsn wrote:

All sounds good to me. Apart from hovers, I think the biggest difference for the mobile version of the homepage would be to measure the interactions with the modules preview, so thanks for adding that!

In T222836#5227806, @MMiller_WMF wrote:

@nettrom_WMF -- the reason I included the instrumentation for the tabs (even though we're not doing it on desktop) is because we're making this special effort to implement the tabs on mobile, and because Readers Web might be interested in it. But now that I'm thinking about it, we better skip it. We're not looking at it on desktop, and it will be confusing to have it for one platform and not the other. I'll change it in the description.

@MMiller_WMF : The list in the description seems fine to me. I'm wondering about the need for capturing the user browsing behavior to their user and/or user talk page, though. We don't do that for desktop, so why capture that specifically for mobile? Is there a particular question we're interested in answering here? (In the measurement plan, we're mainly interested in how users get to the homepage, not how they leave it)

The patch for this has been reviewed and +2'ed. Could it be merged in time for next week's deployment? Let me know if something on my end is needed to make that happen.

Documenting that we've published the results in mediawiki.org, see this analytics update. As mentioned there, we're working on further analysis grouping data from all wikis together.

Adding the other analysts as this might affect schemas they work with.

This is unlikely to be an issue for the schemas that the Growth Team works with, as those only store short-term tokens.

As far as I'm concerned, we don't need to store the specific rule since I can look that up in the log in the MW database. It's perhaps more important to understand why the post failed (e.g. because the user was blocked, or AbuseFilter triggered), rather than specifically what caused it? So for me, knowing the error message works, but that's largely because I can look the specifics up when needed.

We moved the state of the modules in the HomepageModule schema into the state field, which for the email module is either "noemail", "unconfirmed", or "confirmed".

I've used the data for all the leading indicator measurements and done a bit of digging into the data around that. Found no apparent issues with the instrumentation. Closing this as resolved, and if we find specific issues during further analysis we'll open specific tasks for those.

@JAllemandou and @Ottomata : you're both listed as reviewers on https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/508626/ A chance one of you could get that reviewed soon?

The patch was submitted for review yesterday morning (Pacific time). I'll update this task and move it to "In progress" on the current sprint board.

@JAllemandou : Thanks for your patience while this has been stuck in my backlog! I think this looks great as a first step and will enable me to answer the remaining questions around Partial Blocks that the Anti-Harassment Tools Team is interested in.

We've so far not found any significant differences and at the moment other things take priority. Moving this to Q1 so that we can pick it up again then.

I've again updated P8308 with the current state of the schema, to reflect changes to referer_route, referer_namespace, and impact_module_state.

A draft has been written and sent to @MMiller_WMF for review, so this ends up in his column for now.

In T222239#5149298, @kostajh wrote:

So, "review" in this context should be treated as "open". The reason it's called "review" is because it is basically identical to to the "review" part of the help panel (it's using the same code). If it's important to change, we could look into it.

Based on the measurement specification, if I open one of the Help or Mentor modules from the homepage, enter some text and submit it, then close the module, I'd expect the following events to be recorded in the HelpPanel schema:

I have verified that HelpPanel data is now being sanitized and stored correctly, so it's time to close this task.

The previous results of analyzing the abandonment rate between the survey and control groups can be found in T206380#5045440. That analysis includes a split between desktop and mobile registrations, and we found that on the mobile side there was no significant difference in abandonment, while on the desktop side the difference was statistically significant. Because of this difference in effect I've also chosen to split further analysis in the same way.

We've decided on a data retention strategy and added the schema to the whitelist in T220033. I don't see a need to keep this task open.

@JAllemandou : Thanks for clarifying that, very much appreciated!

From reading this, it sounds to me like it'll be possible to identify these events for pages because of the difference between the creation and first edit timestamps. What I'm wondering is if this means that we'll have two fields in mediawiki_page_history? In other words, that we'll have page_creation_timestamp, which reflects the creation timestamp per the logging table, and page_first_edit_timestamp, which has the timestamp of the earliest edit for that page?

Nothing that I updated P8309 with the current state of the schema, so that's also reflected here.

Noting that I've updated P8308 with the current state of the schema, so that's reflected here as well.

Agreed, it looks like we'll stay well below the limit. The team discussed this today, and I've added to our list of leading indicators to also measure the volume of hover events, to make sure that we stay below the limits. That will then also provide us with an indication of how this will scale when adding additional wikis.

Both @MMiller_WMF and I did some back-of-the-envelope estimation of active users today, and reach roughly the same conclusions, that we'll be well below thousands of active users at a given time. I based mine on the number of users who register on these wikis and used a generous estimate, and find that we might be looking at 10 visits per minute.

In T219432#5095574, @kostajh wrote:

@nettrom_WMF please confirm that https://meta.wikimedia.org/w/index.php?title=Schema%3AHomepageVisit&type=revision&diff=18999420&oldid=18984461 is OK with you. Thanks!

In T219435#5088554, @kostajh wrote:

Similarly to my comment on T219432, doesn't the email module have three states? You can need to add your email, need to confirm your email, or be complete.

Commented in T219432, but since we treat the UX of email module as "complete" or "incomplete" that's what the representation in the schema shows. We can definitely change this, just let me know what you prefer to have in the data.

In T219432#5086949, @MMiller_WMF wrote:

I just glanced over this, and I think it looks good. The only questions I have are:

• @nettrom_WMF -- I think we probably discussed this and I forgot exactly -- we are recording the states of the modules in the schema so that it is easy to answer the question of "what state was the homepage in when the user visited?" Is that right? To which of our research questions it that related?

@Niharika : I'm not sure who on the Analytics Engineering team is responsible, and I noticed that neither of the tasks are assigned to anyone. My current understanding is that these changes are likely to arrive with the next snapshot, which should be available in a few days, or the one after that (in early May).

In T219432#5070818, @kostajh wrote:

More questions:

1. Do you just care about reading vs editing, or do you want to know if the user was looking at e.g. page info (action=info), or page history (action=history), etc.
2. referer_namespace for now this is set based on the assumption that the user has clicked on 1) homepage tab from one of user page or user talk page or 2) personal tools link from any page. There is another scenario, where someone has [[Special:Homepage]] on some arbitary page; do we want to try to attempt to get the namespace from visits when a user clicks on a link on some random page? I assume we don't need that right now, but if we do please let me know.

In T219432#5069653, @kostajh wrote:

In other words, would a key/value approach work better than separate fields?

Do you mean doing something like `"module_states": "impact=unactivated;start_tutorial=complete;start_userpage=incomplete"? That could work althoughiIt seems like that would be cumbersome to parse, while adding a new property each time a new module is added doesn't seem like too much work. I'll defer to you on this one.

@Niharika : I picked this up again last week. At this point, I'd like to wait until partial block data is in the Data Lake to continue the work, because then I'll get block duration and edit revert detection for free rather than handle those myself. It would also be great to have IP blocks in the Data Lake, because so far a lot of the partial blocks are of IPs.

The measurement specification already listed the HelpPanel schema as an associated schema. I've now also updated the spec to mention how the helppanel_session_id and homepage_pageview_token fields will match for Homepage-related events.

@kostajh : This looks good to me, and I've gone through the measurement specification and updated it to reflect the current state of what we're capturing.

In T219432#5068374, @kostajh wrote:

@MMiller_WMF @nettrom_WMF I updated the schema per our meeting earlier today. I'm not sure how you'd like to have the start module and impact module states logged @nettrom_WMF, so please have a look at those fields in particular and if you'd prefer a different format let me know.

In T219432#5067041, @kostajh wrote:

I'm a bit confused at this point about what we're proposing to do here, so here's a question to try to clear that up: is the suggestion here to use EditorJourney for capturing this, rather than have an event in the proposed HomePage schema with action=impression?

Yes. The proposal is to log to EditorJourney schema rather than create a new one.

I'm a bit confused at this point about what we're proposing to do here, so here's a question to try to clear that up: is the suggestion here to use EditorJourney for capturing this, rather than have an event in the proposed HomePage schema with action=impression?

Just noting the discussion we had in T219387, as mentioned there this change is OK with me! I've also updated the measurement specification so it mentions the HelpPanel schema and how we'll be using it.

In T219387#5066123, @kostajh wrote:

In theory I would favor splitting out the question asking part, as that's how the code has evolved over time – there's now a generic question posting API and a generic question poster JS widget that's then used by three different implementations. That said, it will be more work to revise this (both creating a new schema, and updating the code) so unless it's a big help to your analysis then I'd say we leave it as is.

@MMiller_WMF : this looks good to go to me!

Should we consider renaming the schema since it's becoming a more generic schema for posting questions to pages? Or split out the question-asking part from the search & links part (in other words: de-normalize the schema)?

We've added table with the findings as a new section in the Help Panel experiment plan. There's also a description of the affected indicators and which ones we've acted upon.

I went back and dug into the data around the spike in Echo blacklist usage in October 2017 a bit more. From what I can tell, there doesn't seem to be a reason to suspect the data is invalid. Since the spike does make interpretation of the remainder of the data very difficult, and we're in this case mainly interested in development over time with a higher weight to more recent developments, I changed the start date of the Echo blacklist usage graphs to 2018-01-01 as that removes the spike.