With T226222 being closed, I see no reason to keep this open any longer. If a specific mobile instrumentation issue comes up, there'll be a separate task for it.

+1 to @leila's rough draft. Could add something to the last sentence to emphasize that these datasets removes the need for additional processing, to make the point about adding context stronger? E.g:

In T224958#5394397, @dmaza wrote:

@Niharika I think we might need approval from someone on the analytics team. Do you know who can we ping here?

I think this can be closed. Ran the query below on Hive, and it didn't find any mobile events for the Help Panel with event.editor_interface set to other. IIRC this problem was fixed in an earlier patch, but I don't know when exactly.

The Help Panel part of this (T226729) was deemed complete by @MMiller_WMF shortly after deployment, and now by me after running the Help Panel leading indicator notebook against the data. At this point data flowing in appear to be correct. Handing this off to @MMiller_WMF for final review.

BTW, I've updated and run the Help Panel leading indicators notebook on Arabic (with the exception of the last question about viewing the answer because that requires more manual labor), and there doesn't appear to be any issues with the data there either.

The Homepage part of this (T228120) is now done.

I've run data from deployment up until now through the Leading Indicator analysis, and do not find anything there that is cause for concern. The additional test of verifying that all Homepage-related events in the HelpPanel data have a matching visit in HomepageVisit, and similarly for HomepageModule, also passed. As far as I can tell, this task is now completed.

Export is done, handing it over to @MMiller_WMF for review.

The Welcome Survey part of this (T226221) is now done.

I've looked at the data and find no reason to be concerned. Treatment/control group assignment is where it should be (on or about 50/50), response rate after a week is good (71.5%), users are primarily answering all questions (87.4% do), abandonment rate in the survey group is low, and there are no free text responses in the data.

As far as I know, the tool has been developed and deployed, so this task can be closed.

I updated my leading indicators notebook since the Homepage's been deployed on viwiki for a little while, and the statistics for desktop users there are similar to those in Czech and Korean, meaning there's no indication of data issues there. I've also verified that all events in HomepageModule have matching events in HomepageVisit, and that all Homepage-related events in the HelpPanel data have matching events in HomepageVisit. Lastly, I went through a sample of mobile events and verified that the events make sense and appear to match user behavior. Since I find no indication that we're having data issues on viwki, I'm closing this task.

I've completed gathering data for this analysis using the 2019-06 MediaWiki history snapshot. Moving it to "In Progress" as I'll start doing the actual analysis.

For users who come to their homepage from the survey confirmation screen button, we will want to record that they came from that button.

OK. I'll probably add referer_route: welcomesurvey to the HomepageVisit schema.

I've added dates to the report, updated the link to it from the analytics updates sub-page, and blanked the report page I created and replaced it with a link to the new place so there's only a single source for this.

@JTannerWMF : thanks for the ping! I think was an interesting experiment with regards to documenting issues, and while our data gathering and analysis continues, I don't see a strong reason to keep this open.

I added Vietnamese WP to my Leading Indicator notebook and all measurements there appear sane at this point. Will follow up again next week and inspect a couple of user sessions to verify that things are in working order.

I've examined the data, and we do not have any events stored beyond 24 hours after user registration, and obfuscation appears to be working correctly. Data also appears to be flowing in, and for example distribution of user actions is similar to other wikis. Don't see a reason to not close this.

Duplicate of T227239.

Based on this conversation it looks like I should upgrade the priority of T207526 and add that we're particularly interested in learning whether users interact with a welcome-style message. I can schedule that for next week, unless @MMiller_WMF suggests otherwise.

@Niharika : Does AHT have any EventLogging schemas that are whitelisted?

There doesn't appear to be much demand for this and I have other projects that require my attention, both in this capacity as well as my volunteer capacity. Declining this.

Growth has three EventLogging schemas in use where data is retained:

With the deployment of this patch and the closing of T225471, the purging / retention strategy for the Homepage project that was proposed as part of the measurement plan is in effect. Handing this off to @MMiller_WMF for final review.

The leading indicator report is on-wiki. Moving to PM for review.

Have confirmed that the data is now flowing into the Data Lake and appears to be correct. Closing this as resolved.

@Cntlsn : We have data on interactions with Echo notifications, so I could run some analysis on that. Are we trying to determine what proportion of users get a notification, and of those, how many click on them? In other words, that we're trying to learn if the notification is a useful way of directing the user's attention to the Homepage?

@Isaac : The footnote on the Norwegian translation of the notice about the survey is in German. Let me know if you need a Norwegian translation of the English text for that as well.

@Niharika : I'm not sure why this task exists and Product Analytics is tagged in it. Since it so closely resembles its parent task (which I've been using to keep track of the work I've been doing) I'm closing this one. Feel free to reopen it if there's a compelling reason to.

Discussed the Growth team metrics with @MMiller_WMF and updated the table with links to Growth's reports.

In T226208#5272031, @kostajh wrote:

The homepage_pageview_token is different, and you should not see any other events for the second visit token since the user is now on the tutorial title.

Thought I'd respond to this and make it actionable in case this moves forward:

We've calculated this for our target wikis individually, and now also for analyzing effects of interventions across multiple wikis with multilevel models (using simulations to estimate the statistical power). For the latter, we also simulated scenarios with additional wikis to understand whether those would allow us to detect smaller effect sizes in a reasonable amount of time.

Upon reviewing this patch, @mforns raised concerns about the information we store about the mentors in this schema. It likely makes it possible to connect mentor and mentee, so the suggestion is to bucket edit counts. Thinking about this, we might want to bucket "time since last activity" as well to make both pieces of information less identifying.

@MMiller_WMF : Sure! I've added counts of the number of users who had events on desktop and mobile in EditorJourney, to account for any users who might've switched between sites. Then the table and the percentages work out like this: