Page MenuHomePhabricator

nettrom_WMF (Morten Warncke-Wang)
Data Analyst, Product Analytics

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Aug 21 2018, 8:23 PM (44 w, 45 s)
Availability
Available
IRC Nick
Nettrom
LDAP User
Unknown
MediaWiki User
MWang (WMF) [ Global Accounts ]

Recent Activity

Today

nettrom_WMF moved T226222: Homepage: mobile instrumentation post-deployment QA from Triage to Next Up on the Product-Analytics board.
Tue, Jun 25, 4:56 PM · Product-Analytics, GrowthExperiments-Homepage, GrowthExperiments, Mobile, Growth-Team (Current Sprint)
nettrom_WMF added a project to T226222: Homepage: mobile instrumentation post-deployment QA: Product-Analytics.
Tue, Jun 25, 4:56 PM · Product-Analytics, GrowthExperiments-Homepage, GrowthExperiments, Mobile, Growth-Team (Current Sprint)

Fri, Jun 21

nettrom_WMF added a comment to T226128: Add relevant Growth and AHT annual plan reports to MediaWiki report page.

Discussed the Growth team metrics with @MMiller_WMF and updated the table with links to Growth's reports.

Fri, Jun 21, 6:05 PM · Better Use Of Data, Product-Analytics

Thu, Jun 20

nettrom_WMF added a comment to T226208: Homepage: do not log HomepageVisit when handling tutorial visit POST requests.

The homepage_pageview_token is different, and you should not see any other events for the second visit token since the user is now on the tutorial title.

Thu, Jun 20, 8:32 PM · MW-1.34-notes (1.34.0-wmf.11; 2019-06-26), Growth-Team (Current Sprint), GrowthExperiments-Homepage, GrowthExperiments

Wed, Jun 19

nettrom_WMF moved T217803: Homepage: experiment from Triage to Doing on the Product-Analytics board.
Wed, Jun 19, 9:33 PM · Product-Analytics, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF added a project to T217803: Homepage: experiment: Product-Analytics.
Wed, Jun 19, 9:33 PM · Product-Analytics, GrowthExperiments-Homepage, Growth-Team (Current Sprint)

Fri, Jun 14

nettrom_WMF added a comment to T216220: Homepage: design for indicating that the user should check back for pageviews.

Thought I'd respond to this and make it actionable in case this moves forward:

Fri, Jun 14, 4:24 PM · MW-1.34-notes (1.34.0-wmf.10; 2019-06-18), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)

Wed, Jun 12

nettrom_WMF closed T201658: Growth: power calculations for experiments as Resolved.

We've calculated this for our target wikis individually, and now also for analyzing effects of interventions across multiple wikis with multilevel models (using simulations to estimate the statistical power). For the latter, we also simulated scenarios with additional wikis to understand whether those would allow us to detect smaller effect sizes in a reasonable amount of time.

Wed, Jun 12, 7:22 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF added a comment to T225471: Homepage: add schemas to EventLogging whitelist.

Upon reviewing this patch, @mforns raised concerns about the information we store about the mentors in this schema. It likely makes it possible to connect mentor and mentee, so the suggestion is to bucket edit counts. Thinking about this, we might want to bucket "time since last activity" as well to make both pieces of information less identifying.

Wed, Jun 12, 4:38 PM · Analytics, Product-Analytics, Growth-Team (Current Sprint)

Tue, Jun 11

nettrom_WMF moved T225471: Homepage: add schemas to EventLogging whitelist from Incoming to In Progress on the Growth-Team (Current Sprint) board.
Tue, Jun 11, 6:37 PM · Analytics, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T225471: Homepage: add schemas to EventLogging whitelist from Triage to Doing on the Product-Analytics board.
Tue, Jun 11, 6:37 PM · Analytics, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF updated subscribers of T225471: Homepage: add schemas to EventLogging whitelist.
Tue, Jun 11, 4:57 PM · Analytics, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T225328: Homepage: link from Special:Contributions.

@MMiller_WMF : Sure! I've added counts of the number of users who had events on desktop and mobile in EditorJourney, to account for any users who might've switched between sites. Then the table and the percentages work out like this:

Tue, Jun 11, 4:09 PM · Mobile, Desktop, GrowthExperiments- HomepageDiscovery, Growth Design, GrowthExperiments-Homepage, Growth-Team (Current Sprint)

Mon, Jun 10

nettrom_WMF added a parent task for T225471: Homepage: add schemas to EventLogging whitelist: T219252: Homepage: specify purging strategy.
Mon, Jun 10, 11:39 PM · Analytics, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF added a subtask for T219252: Homepage: specify purging strategy: T225471: Homepage: add schemas to EventLogging whitelist.
Mon, Jun 10, 11:39 PM · Analytics, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF created T225471: Homepage: add schemas to EventLogging whitelist.
Mon, Jun 10, 11:39 PM · Analytics, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T225328: Homepage: link from Special:Contributions.

I restricted the query to visits to their contributions specifically. This uses the 90 days worth of data from EditorJourney, and is limited to non-autocreated accounts that were not registered through the API, and that were not known test accounts by Growth Team members. A user can be counted twice by visiting the page on both desktop and mobile, but the number of users who do so might be rather small.

Mon, Jun 10, 9:41 PM · Mobile, Desktop, GrowthExperiments- HomepageDiscovery, Growth Design, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF closed T222101: Update ServerSideAccountCreation schema whitelist as Resolved.

Verified that this is now deployed and data is retained/purged as expected.

Mon, Jun 10, 5:10 PM · Growth-Team (Current Sprint), Analytics, Product-Analytics

Fri, Jun 7

nettrom_WMF added a comment to T222836: Mobile Homepage: instrumentation.

A lot depend on the module names in the code so I would prefer to not suffix them.

Fri, Jun 7, 5:54 PM · MW-1.34-notes (1.34.0-wmf.10; 2019-06-18), GrowthExperiments-Homepage, GrowthExperiments, Mobile, Growth-Team (Current Sprint)

Thu, Jun 6

nettrom_WMF added a comment to T222836: Mobile Homepage: instrumentation.

I think we can add information to the "impression" action of HomepageModule to specify whether the impression is

  • "desktop" (like it is now)
  • "mobile summary" (the shorter preview shown on the mobile homepage)
  • "mobile details" (the full mobile version, shown in a subpage currently but will be in a mobile overlay when T222833 is done)
Thu, Jun 6, 5:06 PM · MW-1.34-notes (1.34.0-wmf.10; 2019-06-18), GrowthExperiments-Homepage, GrowthExperiments, Mobile, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T216220: Homepage: design for indicating that the user should check back for pageviews.

@nettrom_WMF, @MMiller_WMF any thoughts about where you want to see the event logging data for opening and closing that tooltip? I was thinking we could stuff it as new actions in HomepageModule. Something like open-nopageviews-tooltip and close-nopageviews-tooltip (or anything else you would prefer). The mobile flag would tell us if the open/close are caused by taps (mobile) or hover (desktop).

Thu, Jun 6, 4:34 PM · MW-1.34-notes (1.34.0-wmf.10; 2019-06-18), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)

Mon, Jun 3

nettrom_WMF added a comment to T222836: Mobile Homepage: instrumentation.

@MMiller_WMF : I've drafted some suggested addition to the measurement specification, could you take a look at those and let me know if further changes are needed? Also, I updated this task's description so it mentions that we'd also like to capture clicks on the "back" buttons in the modules on the mobile site, because that kind of interaction isn't available on the desktop site.

Mon, Jun 3, 10:34 PM · MW-1.34-notes (1.34.0-wmf.10; 2019-06-18), GrowthExperiments-Homepage, GrowthExperiments, Mobile, Growth-Team (Current Sprint)
nettrom_WMF updated the task description for T222836: Mobile Homepage: instrumentation.
Mon, Jun 3, 10:31 PM · MW-1.34-notes (1.34.0-wmf.10; 2019-06-18), GrowthExperiments-Homepage, GrowthExperiments, Mobile, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T222836: Mobile Homepage: instrumentation.

All sounds good to me. Apart from hovers, I think the biggest difference for the mobile version of the homepage would be to measure the interactions with the modules preview, so thanks for adding that!

Mon, Jun 3, 10:14 PM · MW-1.34-notes (1.34.0-wmf.10; 2019-06-18), GrowthExperiments-Homepage, GrowthExperiments, Mobile, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T222836: Mobile Homepage: instrumentation.

@nettrom_WMF -- the reason I included the instrumentation for the tabs (even though we're not doing it on desktop) is because we're making this special effort to implement the tabs on mobile, and because Readers Web might be interested in it. But now that I'm thinking about it, we better skip it. We're not looking at it on desktop, and it will be confusing to have it for one platform and not the other. I'll change it in the description.

Mon, Jun 3, 9:57 PM · MW-1.34-notes (1.34.0-wmf.10; 2019-06-18), GrowthExperiments-Homepage, GrowthExperiments, Mobile, Growth-Team (Current Sprint)

Thu, May 30

nettrom_WMF added a comment to T222836: Mobile Homepage: instrumentation.

@MMiller_WMF : The list in the description seems fine to me. I'm wondering about the need for capturing the user browsing behavior to their user and/or user talk page, though. We don't do that for desktop, so why capture that specifically for mobile? Is there a particular question we're interested in answering here? (In the measurement plan, we're mainly interested in how users get to the homepage, not how they leave it)

Thu, May 30, 11:43 PM · MW-1.34-notes (1.34.0-wmf.10; 2019-06-18), GrowthExperiments-Homepage, GrowthExperiments, Mobile, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T222101: Update ServerSideAccountCreation schema whitelist.

The patch for this has been reviewed and +2'ed. Could it be merged in time for next week's deployment? Let me know if something on my end is needed to make that happen.

Thu, May 30, 6:33 PM · Growth-Team (Current Sprint), Analytics, Product-Analytics
nettrom_WMF added a comment to T206720: Help panel: experiments.

Documenting that we've published the results in mediawiki.org, see this analytics update. As mentioned there, we're working on further analysis grouping data from all wikis together.

Thu, May 30, 6:12 PM · Growth-Team (Current Sprint), Product-Analytics

Tue, May 28

nettrom_WMF updated subscribers of T220410: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list.

Adding the other analysts as this might affect schemas they work with.

Tue, May 28, 9:38 PM · Product-Analytics, Analytics
nettrom_WMF added a comment to T220410: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list.

This is unlikely to be an issue for the schemas that the Growth Team works with, as those only store short-term tokens.

Tue, May 28, 9:36 PM · Product-Analytics, Analytics
nettrom_WMF added a comment to T224039: HelpPanel schema: log actual error message in event_data for 'submit-failure' action.

As far as I'm concerned, we don't need to store the specific rule since I can look that up in the log in the MW database. It's perhaps more important to understand why the post failed (e.g. because the user was blocked, or AbuseFilter triggered), rather than specifically what caused it? So for me, knowing the error message works, but that's largely because I can look the specifics up when needed.

Tue, May 28, 9:28 PM · MW-1.34-notes (1.34.0-wmf.8; 2019-06-04), GrowthExperiments-Homepage, GrowthExperiments-Help panel, Growth-Team (Current Sprint)

May 24 2019

nettrom_WMF added a comment to T224315: Homepage: potential missing data for email module.

We moved the state of the modules in the HomepageModule schema into the state field, which for the email module is either "noemail", "unconfirmed", or "confirmed".

May 24 2019, 8:42 PM · GrowthExperiments, Growth-Team (Current Sprint)

May 22 2019

nettrom_WMF closed T219255: Homepage: post-deployment QA of instrumentation as Resolved.

I've used the data for all the leading indicator measurements and done a bit of digging into the data around that. Found no apparent issues with the instrumentation. Closing this as resolved, and if we find specific issues during further analysis we'll open specific tasks for those.

May 22 2019, 5:48 PM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF closed T219255: Homepage: post-deployment QA of instrumentation, a subtask of T216586: Homepage: instrumentation, as Resolved.
May 22 2019, 5:48 PM · Analytics, GrowthExperiments, Product-Analytics, Growth-Team (Current Sprint)

May 14 2019

nettrom_WMF updated subscribers of T222101: Update ServerSideAccountCreation schema whitelist.

@JAllemandou and @Ottomata : you're both listed as reviewers on https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/508626/ A chance one of you could get that reviewed soon?

May 14 2019, 9:31 PM · Growth-Team (Current Sprint), Analytics, Product-Analytics

May 9 2019

nettrom_WMF created T222928: Homepage: Start module prompts to create userpage in place of global userpage.
May 9 2019, 10:54 PM · GrowthExperiments-Homepage, Growth-Team

May 8 2019

nettrom_WMF moved T222101: Update ServerSideAccountCreation schema whitelist from Triage to Doing on the Product-Analytics board.
May 8 2019, 7:16 PM · Growth-Team (Current Sprint), Analytics, Product-Analytics
nettrom_WMF moved T222101: Update ServerSideAccountCreation schema whitelist from Incoming to In Progress on the Growth-Team (Current Sprint) board.
May 8 2019, 7:15 PM · Growth-Team (Current Sprint), Analytics, Product-Analytics
nettrom_WMF edited projects for T222101: Update ServerSideAccountCreation schema whitelist, added: Growth-Team (Current Sprint); removed Growth-Team.
May 8 2019, 7:15 PM · Growth-Team (Current Sprint), Analytics, Product-Analytics
nettrom_WMF added a comment to T222101: Update ServerSideAccountCreation schema whitelist.

The patch was submitted for review yesterday morning (Pacific time). I'll update this task and move it to "In progress" on the current sprint board.

May 8 2019, 7:15 PM · Growth-Team (Current Sprint), Analytics, Product-Analytics
nettrom_WMF moved T219255: Homepage: post-deployment QA of instrumentation from In Progress to Ready for Development on the Growth-Team (Current Sprint) board.
May 8 2019, 4:28 PM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T219255: Homepage: post-deployment QA of instrumentation from Ready for Development to In Progress on the Growth-Team (Current Sprint) board.
May 8 2019, 4:28 PM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T219254: Homepage: report on leading indicators from Ready for Development to In Progress on the Growth-Team (Current Sprint) board.
May 8 2019, 4:28 PM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T216672: EditorJourney and Welcome Survey: second report from Ready for Development to In Progress on the Growth-Team (Current Sprint) board.
May 8 2019, 4:28 PM · Growth-Team (Current Sprint), Product-Analytics

May 7 2019

nettrom_WMF triaged T222101: Update ServerSideAccountCreation schema whitelist as Normal priority.
May 7 2019, 4:54 PM · Growth-Team (Current Sprint), Analytics, Product-Analytics

May 2 2019

nettrom_WMF added a comment to T211950: Add partial blocks to mediawiki history tables.

@JAllemandou : Thanks for your patience while this has been stuck in my backlog! I think this looks great as a first step and will enable me to answer the remaining questions around Partial Blocks that the Anti-Harassment Tools Team is interested in.

May 2 2019, 5:44 PM · Analytics-Kanban, Product-Analytics, Anti-Harassment, Analytics
nettrom_WMF moved T216668: Welcome survey: investigate Vietnamese abandonment rate from Triage to Backlog on the Product-Analytics board.
May 2 2019, 3:54 PM · Product-Analytics, Growth-Team, CommRel-Specialists-Support (Jan-Mar-2019)
nettrom_WMF added a project to T216668: Welcome survey: investigate Vietnamese abandonment rate: Product-Analytics.
May 2 2019, 3:53 PM · Product-Analytics, Growth-Team, CommRel-Specialists-Support (Jan-Mar-2019)
nettrom_WMF moved T216668: Welcome survey: investigate Vietnamese abandonment rate from Inbox to FY 2019-20 on the Growth-Team board.
May 2 2019, 3:53 PM · Product-Analytics, Growth-Team, CommRel-Specialists-Support (Jan-Mar-2019)
nettrom_WMF edited projects for T216668: Welcome survey: investigate Vietnamese abandonment rate, added: Growth-Team; removed Growth-Team (Current Sprint).

We've so far not found any significant differences and at the moment other things take priority. Moving this to Q1 so that we can pick it up again then.

May 2 2019, 3:53 PM · Product-Analytics, Growth-Team, CommRel-Specialists-Support (Jan-Mar-2019)

May 1 2019

nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

I've again updated P8308 with the current state of the schema, to reflect changes to referer_route, referer_namespace, and impact_module_state.

May 1 2019, 11:20 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF edited P8308 HomepageVisit schema.
May 1 2019, 11:18 PM · Growth-Team, GrowthExperiments-Homepage
nettrom_WMF moved T219253: Homepage: publish measurement specifications and experiment plan from Triage to Doing on the Product-Analytics board.
May 1 2019, 6:19 PM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF reassigned T219253: Homepage: publish measurement specifications and experiment plan from nettrom_WMF to MMiller_WMF.

A draft has been written and sent to @MMiller_WMF for review, so this ends up in his column for now.

May 1 2019, 6:19 PM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T222239: Homepage: no "enter-question-text" event recorded.

So, "review" in this context should be treated as "open". The reason it's called "review" is because it is basically identical to to the "review" part of the help panel (it's using the same code). If it's important to change, we could look into it.

May 1 2019, 4:17 PM · MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), Patch-For-Review, GrowthExperiments, Growth-Team (Current Sprint)

Apr 30 2019

nettrom_WMF added a comment to T222239: Homepage: no "enter-question-text" event recorded.

Based on the measurement specification, if I open one of the Help or Mentor modules from the homepage, enter some text and submit it, then close the module, I'd expect the following events to be recorded in the HelpPanel schema:

Apr 30 2019, 11:38 PM · MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), Patch-For-Review, GrowthExperiments, Growth-Team (Current Sprint)

Apr 29 2019

nettrom_WMF created T222101: Update ServerSideAccountCreation schema whitelist.
Apr 29 2019, 7:40 PM · Growth-Team (Current Sprint), Analytics, Product-Analytics

Apr 25 2019

nettrom_WMF closed T220033: Add HelpPanel schema to the EventLogging whitelist as Resolved.

I have verified that HelpPanel data is now being sanitized and stored correctly, so it's time to close this task.

Apr 25 2019, 6:17 PM · Patch-For-Review, Analytics, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF closed T220033: Add HelpPanel schema to the EventLogging whitelist, a subtask of T206719: Help panel: instrumentation, as Resolved.
Apr 25 2019, 6:17 PM · Growth-Team (Current Sprint), Product-Analytics

Apr 19 2019

nettrom_WMF added a comment to T216668: Welcome survey: investigate Vietnamese abandonment rate.

The previous results of analyzing the abandonment rate between the survey and control groups can be found in T206380#5045440. That analysis includes a split between desktop and mobile registrations, and we found that on the mobile side there was no significant difference in abandonment, while on the desktop side the difference was statistically significant. Because of this difference in effect I've also chosen to split further analysis in the same way.

Apr 19 2019, 11:49 PM · Product-Analytics, Growth-Team, CommRel-Specialists-Support (Jan-Mar-2019)
nettrom_WMF closed T212464: Help panel: decide on a purging strategy for the HelpPanel schema as Resolved.

We've decided on a data retention strategy and added the schema to the whitelist in T220033. I don't see a need to keep this task open.

Apr 19 2019, 5:24 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF closed T212464: Help panel: decide on a purging strategy for the HelpPanel schema, a subtask of T206719: Help panel: instrumentation, as Resolved.
Apr 19 2019, 5:23 PM · Growth-Team (Current Sprint), Product-Analytics

Apr 12 2019

nettrom_WMF added a comment to T220507: Decide: start_timestamp for mediawiki history.

@JAllemandou : Thanks for clarifying that, very much appreciated!

Apr 12 2019, 5:44 PM · Analytics-Kanban, Analytics

Apr 11 2019

nettrom_WMF added a comment to T220507: Decide: start_timestamp for mediawiki history.

From reading this, it sounds to me like it'll be possible to identify these events for pages because of the difference between the creation and first edit timestamps. What I'm wondering is if this means that we'll have two fields in mediawiki_page_history? In other words, that we'll have page_creation_timestamp, which reflects the creation timestamp per the logging table, and page_first_edit_timestamp, which has the timestamp of the earliest edit for that page?

Apr 11 2019, 4:31 PM · Analytics-Kanban, Analytics

Apr 10 2019

nettrom_WMF added a comment to T219435: Homepage: HomepageModule schema.

Nothing that I updated P8309 with the current state of the schema, so that's also reflected here.

Apr 10 2019, 11:29 PM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Growth-Team (Current Sprint)
nettrom_WMF edited P8309 HomepageModule schema.
Apr 10 2019, 11:27 PM · Growth-Team, GrowthExperiments-Homepage
nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

Noting that I've updated P8308 with the current state of the schema, so that's reflected here as well.

Apr 10 2019, 11:26 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF edited P8308 HomepageVisit schema.
Apr 10 2019, 11:25 PM · Growth-Team, GrowthExperiments-Homepage
nettrom_WMF added a comment to T219435: Homepage: HomepageModule schema.

Agreed, it looks like we'll stay well below the limit. The team discussed this today, and I've added to our list of leading indicators to also measure the volume of hover events, to make sure that we stay below the limits. That will then also provide us with an indication of how this will scale when adding additional wikis.

Apr 10 2019, 8:01 PM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Growth-Team (Current Sprint)

Apr 9 2019

nettrom_WMF added a comment to T219435: Homepage: HomepageModule schema.

Both @MMiller_WMF and I did some back-of-the-envelope estimation of active users today, and reach roughly the same conclusions, that we'll be well below thousands of active users at a given time. I based mine on the number of users who register on these wikis and used a generous estimate, and find that we might be looking at 10 visits per minute.

Apr 9 2019, 10:54 PM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Growth-Team (Current Sprint)

Apr 8 2019

nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.
Apr 8 2019, 9:46 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)

Apr 6 2019

nettrom_WMF added a comment to T219435: Homepage: HomepageModule schema.

Similarly to my comment on T219432, doesn't the email module have three states? You can need to add your email, need to confirm your email, or be complete.

Commented in T219432, but since we treat the UX of email module as "complete" or "incomplete" that's what the representation in the schema shows. We can definitely change this, just let me know what you prefer to have in the data.

Apr 6 2019, 12:53 AM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

I just glanced over this, and I think it looks good. The only questions I have are:

  • @nettrom_WMF -- I think we probably discussed this and I forgot exactly -- we are recording the states of the modules in the schema so that it is easy to answer the question of "what state was the homepage in when the user visited?" Is that right? To which of our research questions it that related?
Apr 6 2019, 12:43 AM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)

Apr 3 2019

nettrom_WMF updated the task description for T220033: Add HelpPanel schema to the EventLogging whitelist.
Apr 3 2019, 7:41 PM · Patch-For-Review, Analytics, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF created T220033: Add HelpPanel schema to the EventLogging whitelist.
Apr 3 2019, 7:41 PM · Patch-For-Review, Analytics, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T219252: Homepage: specify purging strategy from Triage to Doing on the Product-Analytics board.
Apr 3 2019, 5:15 PM · Analytics, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T219254: Homepage: report on leading indicators from Triage to Next Up on the Product-Analytics board.
Apr 3 2019, 5:15 PM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T219255: Homepage: post-deployment QA of instrumentation from Triage to Next Up on the Product-Analytics board.
Apr 3 2019, 5:14 PM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T220017: Write a blog post about monitoring EventLogging deployment from Triage to Next Up on the Product-Analytics board.
Apr 3 2019, 5:14 PM · Product-Analytics
nettrom_WMF created T220017: Write a blog post about monitoring EventLogging deployment.
Apr 3 2019, 5:14 PM · Product-Analytics
nettrom_WMF triaged T217717: Emailability: basic measurement as Normal priority.
Apr 3 2019, 5:10 PM · Product-Analytics, Growth-Team
nettrom_WMF moved T217717: Emailability: basic measurement from Triage to Backlog on the Product-Analytics board.
Apr 3 2019, 5:10 PM · Product-Analytics, Growth-Team

Apr 2 2019

nettrom_WMF added a comment to T209403: 👩‍👧‍👦 Measure the effectiveness of blocks.

@Niharika : I'm not sure who on the Analytics Engineering team is responsible, and I noticed that neither of the tasks are assigned to anyone. My current understanding is that these changes are likely to arrive with the next snapshot, which should be available in a few days, or the one after that (in early May).

Apr 2 2019, 7:22 PM · Anti-Harassment, Product-Analytics, MediaWiki-User-management

Apr 1 2019

nettrom_WMF updated the task description for T215442: Make Refine use JSONSchemas of event data to support Map types and proper types for integers vs decimals.
Apr 1 2019, 9:33 PM · MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), Patch-For-Review, Analytics-Kanban, EventBus, Analytics
nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

More questions:

  1. Do you just care about reading vs editing, or do you want to know if the user was looking at e.g. page info (action=info), or page history (action=history), etc.
  2. referer_namespace for now this is set based on the assumption that the user has clicked on 1) homepage tab from one of user page or user talk page or 2) personal tools link from any page. There is another scenario, where someone has [[Special:Homepage]] on some arbitary page; do we want to try to attempt to get the namespace from visits when a user clicks on a link on some random page? I assume we don't need that right now, but if we do please let me know.
Apr 1 2019, 6:02 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

In other words, would a key/value approach work better than separate fields?

Do you mean doing something like `"module_states": "impact=unactivated;start_tutorial=complete;start_userpage=incomplete"? That could work althoughiIt seems like that would be cumbersome to parse, while adding a new property each time a new module is added doesn't seem like too much work. I'll defer to you on this one.

Apr 1 2019, 5:45 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF updated subscribers of T209403: 👩‍👧‍👦 Measure the effectiveness of blocks.

@Niharika : I picked this up again last week. At this point, I'd like to wait until partial block data is in the Data Lake to continue the work, because then I'll get block duration and edit revert detection for free rather than handle those myself. It would also be great to have IP blocks in the Data Lake, because so far a lot of the partial blocks are of IPs.

Apr 1 2019, 4:09 PM · Anti-Harassment, Product-Analytics, MediaWiki-User-management

Mar 29 2019

nettrom_WMF added a comment to T219431: Homepage: Mentorship and Help module schema usage.

The measurement specification already listed the HelpPanel schema as an associated schema. I've now also updated the spec to mention how the helppanel_session_id and homepage_pageview_token fields will match for Homepage-related events.

Mar 29 2019, 4:44 AM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219435: Homepage: HomepageModule schema.

@kostajh : This looks good to me, and I've gone through the measurement specification and updated it to reflect the current state of what we're capturing.

Mar 29 2019, 4:39 AM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

@MMiller_WMF @nettrom_WMF I updated the schema per our meeting earlier today. I'm not sure how you'd like to have the start module and impact module states logged @nettrom_WMF, so please have a look at those fields in particular and if you'd prefer a different format let me know.

Mar 29 2019, 4:21 AM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)

Mar 28 2019

nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

I'm a bit confused at this point about what we're proposing to do here, so here's a question to try to clear that up: is the suggestion here to use EditorJourney for capturing this, rather than have an event in the proposed HomePage schema with action=impression?

Yes. The proposal is to log to EditorJourney schema rather than create a new one.

Mar 28 2019, 7:10 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

I'm a bit confused at this point about what we're proposing to do here, so here's a question to try to clear that up: is the suggestion here to use EditorJourney for capturing this, rather than have an event in the proposed HomePage schema with action=impression?

Mar 28 2019, 6:33 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219431: Homepage: Mentorship and Help module schema usage.

Just noting the discussion we had in T219387, as mentioned there this change is OK with me! I've also updated the measurement specification so it mentions the HelpPanel schema and how we'll be using it.

Mar 28 2019, 4:17 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219387: Homepage: modify Schema:HelpPanel to accept events from Special:Homepage mentorship and help modules.

In theory I would favor splitting out the question asking part, as that's how the code has evolved over time – there's now a generic question posting API and a generic question poster JS widget that's then used by three different implementations. That said, it will be more work to revise this (both creating a new schema, and updating the code) so unless it's a big help to your analysis then I'd say we leave it as is.

Mar 28 2019, 3:50 PM · GrowthExperiments, Growth-Team (Current Sprint)

Mar 27 2019

nettrom_WMF added a comment to T217803: Homepage: experiment.

@MMiller_WMF : this looks good to go to me!

Mar 27 2019, 8:58 PM · Product-Analytics, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF closed T216185: Analyze use of mute preferences as Resolved.
Mar 27 2019, 8:26 PM · Chinese-Sites, Anti-Harassment, Product-Analytics
nettrom_WMF added a comment to T219387: Homepage: modify Schema:HelpPanel to accept events from Special:Homepage mentorship and help modules.

Should we consider renaming the schema since it's becoming a more generic schema for posting questions to pages? Or split out the question-asking part from the search & links part (in other words: de-normalize the schema)?

Mar 27 2019, 5:56 PM · GrowthExperiments, Growth-Team (Current Sprint)

Mar 26 2019

nettrom_WMF closed T214125: Help panel: report on leading indicators as Resolved.

We've added table with the findings as a new section in the Help Panel experiment plan. There's also a description of the affected indicators and which ones we've acted upon.

Mar 26 2019, 4:14 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF closed T214125: Help panel: report on leading indicators, a subtask of T206719: Help panel: instrumentation, as Resolved.
Mar 26 2019, 4:14 PM · Growth-Team (Current Sprint), Product-Analytics

Mar 22 2019

nettrom_WMF added a comment to T216185: Analyze use of mute preferences.

I went back and dug into the data around the spike in Echo blacklist usage in October 2017 a bit more. From what I can tell, there doesn't seem to be a reason to suspect the data is invalid. Since the spike does make interpretation of the remainder of the data very difficult, and we're in this case mainly interested in development over time with a higher weight to more recent developments, I changed the start date of the Echo blacklist usage graphs to 2018-01-01 as that removes the spike.

Mar 22 2019, 9:22 PM · Chinese-Sites, Anti-Harassment, Product-Analytics