Page MenuHomePhabricator

nettrom_WMF (Morten Warncke-Wang)
Data Analyst, Product Analytics

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Aug 21 2018, 8:23 PM (34 w, 5 d)
Availability
Available
IRC Nick
Nettrom
LDAP User
Unknown
MediaWiki User
MWang (WMF) [ Global Accounts ]

Recent Activity

Fri, Apr 19

nettrom_WMF added a comment to T216668: Welcome survey: investigate Vietnamese abandonment rate.

The previous results of analyzing the abandonment rate between the survey and control groups can be found in T206380#5045440. That analysis includes a split between desktop and mobile registrations, and we found that on the mobile side there was no significant difference in abandonment, while on the desktop side the difference was statistically significant. Because of this difference in effect I've also chosen to split further analysis in the same way.

Fri, Apr 19, 11:49 PM · Growth-Team (Current Sprint), CommRel-Specialists-Support (Jan-Mar-2019)
nettrom_WMF closed T212464: Help panel: decide on a purging strategy for the HelpPanel schema as Resolved.

We've decided on a data retention strategy and added the schema to the whitelist in T220033. I don't see a need to keep this task open.

Fri, Apr 19, 5:24 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF closed T212464: Help panel: decide on a purging strategy for the HelpPanel schema, a subtask of T206719: Help panel: instrumentation, as Resolved.
Fri, Apr 19, 5:23 PM · Growth-Team (Current Sprint), Product-Analytics

Fri, Apr 12

nettrom_WMF added a comment to T220507: Decide: start_timestamp for mediawiki history.

@JAllemandou : Thanks for clarifying that, very much appreciated!

Fri, Apr 12, 5:44 PM · Analytics-Kanban, Analytics

Thu, Apr 11

nettrom_WMF added a comment to T220507: Decide: start_timestamp for mediawiki history.

From reading this, it sounds to me like it'll be possible to identify these events for pages because of the difference between the creation and first edit timestamps. What I'm wondering is if this means that we'll have two fields in mediawiki_page_history? In other words, that we'll have page_creation_timestamp, which reflects the creation timestamp per the logging table, and page_first_edit_timestamp, which has the timestamp of the earliest edit for that page?

Thu, Apr 11, 4:31 PM · Analytics-Kanban, Analytics

Wed, Apr 10

nettrom_WMF added a comment to T219435: Homepage: HomepageModule schema.

Nothing that I updated P8309 with the current state of the schema, so that's also reflected here.

Wed, Apr 10, 11:29 PM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Growth-Team (Current Sprint)
nettrom_WMF edited P8309 HomepageModule schema.
Wed, Apr 10, 11:27 PM · Growth-Team, GrowthExperiments-Homepage
nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

Noting that I've updated P8308 with the current state of the schema, so that's reflected here as well.

Wed, Apr 10, 11:26 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF edited P8308 HomepageVisit schema.
Wed, Apr 10, 11:25 PM · Growth-Team, GrowthExperiments-Homepage
nettrom_WMF added a comment to T219435: Homepage: HomepageModule schema.

Agreed, it looks like we'll stay well below the limit. The team discussed this today, and I've added to our list of leading indicators to also measure the volume of hover events, to make sure that we stay below the limits. That will then also provide us with an indication of how this will scale when adding additional wikis.

Wed, Apr 10, 8:01 PM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Growth-Team (Current Sprint)

Tue, Apr 9

nettrom_WMF added a comment to T219435: Homepage: HomepageModule schema.

Both @MMiller_WMF and I did some back-of-the-envelope estimation of active users today, and reach roughly the same conclusions, that we'll be well below thousands of active users at a given time. I based mine on the number of users who register on these wikis and used a generous estimate, and find that we might be looking at 10 visits per minute.

Tue, Apr 9, 10:54 PM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Growth-Team (Current Sprint)

Mon, Apr 8

nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.
Mon, Apr 8, 9:46 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)

Sat, Apr 6

nettrom_WMF added a comment to T219435: Homepage: HomepageModule schema.

Similarly to my comment on T219432, doesn't the email module have three states? You can need to add your email, need to confirm your email, or be complete.

Commented in T219432, but since we treat the UX of email module as "complete" or "incomplete" that's what the representation in the schema shows. We can definitely change this, just let me know what you prefer to have in the data.

Sat, Apr 6, 12:53 AM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

I just glanced over this, and I think it looks good. The only questions I have are:

  • @nettrom_WMF -- I think we probably discussed this and I forgot exactly -- we are recording the states of the modules in the schema so that it is easy to answer the question of "what state was the homepage in when the user visited?" Is that right? To which of our research questions it that related?
Sat, Apr 6, 12:43 AM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)

Wed, Apr 3

nettrom_WMF updated the task description for T220033: Add HelpPanel schema to the EventLogging whitelist.
Wed, Apr 3, 7:41 PM · Patch-For-Review, Analytics, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF created T220033: Add HelpPanel schema to the EventLogging whitelist.
Wed, Apr 3, 7:41 PM · Patch-For-Review, Analytics, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T219252: Homepage: specify purging strategy from Triage to Doing on the Product-Analytics board.
Wed, Apr 3, 5:15 PM · Analytics, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T219254: Homepage: report on leading indicators from Triage to Next Up on the Product-Analytics board.
Wed, Apr 3, 5:15 PM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T219255: Homepage: post-deployment QA of instrumentation from Triage to Next Up on the Product-Analytics board.
Wed, Apr 3, 5:14 PM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T220017: Write a blog post about monitoring EventLogging deployment from Triage to Next Up on the Product-Analytics board.
Wed, Apr 3, 5:14 PM · Product-Analytics
nettrom_WMF created T220017: Write a blog post about monitoring EventLogging deployment.
Wed, Apr 3, 5:14 PM · Product-Analytics
nettrom_WMF triaged T217717: Emailability: basic measurement as Normal priority.
Wed, Apr 3, 5:10 PM · Product-Analytics, Growth-Team
nettrom_WMF moved T217717: Emailability: basic measurement from Triage to Backlog on the Product-Analytics board.
Wed, Apr 3, 5:10 PM · Product-Analytics, Growth-Team

Tue, Apr 2

nettrom_WMF added a comment to T209403: 👩‍👧‍👦 Measure the effectiveness of blocks.

@Niharika : I'm not sure who on the Analytics Engineering team is responsible, and I noticed that neither of the tasks are assigned to anyone. My current understanding is that these changes are likely to arrive with the next snapshot, which should be available in a few days, or the one after that (in early May).

Tue, Apr 2, 7:22 PM · Anti-Harassment, Product-Analytics, MediaWiki-User-management

Mon, Apr 1

nettrom_WMF updated the task description for T215442: Make Refine use JSONSchemas of event data to support Map types and proper types for integers vs decimals.
Mon, Apr 1, 9:33 PM · MW-1.33-notes (1.33.0-wmf.22; 2019-03-19), Patch-For-Review, Analytics-Kanban, EventBus, Analytics
nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

More questions:

  1. Do you just care about reading vs editing, or do you want to know if the user was looking at e.g. page info (action=info), or page history (action=history), etc.
  2. referer_namespace for now this is set based on the assumption that the user has clicked on 1) homepage tab from one of user page or user talk page or 2) personal tools link from any page. There is another scenario, where someone has [[Special:Homepage]] on some arbitary page; do we want to try to attempt to get the namespace from visits when a user clicks on a link on some random page? I assume we don't need that right now, but if we do please let me know.
Mon, Apr 1, 6:02 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

In other words, would a key/value approach work better than separate fields?

Do you mean doing something like `"module_states": "impact=unactivated;start_tutorial=complete;start_userpage=incomplete"? That could work althoughiIt seems like that would be cumbersome to parse, while adding a new property each time a new module is added doesn't seem like too much work. I'll defer to you on this one.

Mon, Apr 1, 5:45 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF updated subscribers of T209403: 👩‍👧‍👦 Measure the effectiveness of blocks.

@Niharika : I picked this up again last week. At this point, I'd like to wait until partial block data is in the Data Lake to continue the work, because then I'll get block duration and edit revert detection for free rather than handle those myself. It would also be great to have IP blocks in the Data Lake, because so far a lot of the partial blocks are of IPs.

Mon, Apr 1, 4:09 PM · Anti-Harassment, Product-Analytics, MediaWiki-User-management

Fri, Mar 29

nettrom_WMF added a comment to T219431: Homepage: Mentorship and Help module schema usage.

The measurement specification already listed the HelpPanel schema as an associated schema. I've now also updated the spec to mention how the helppanel_session_id and homepage_pageview_token fields will match for Homepage-related events.

Fri, Mar 29, 4:44 AM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219435: Homepage: HomepageModule schema.

@kostajh : This looks good to me, and I've gone through the measurement specification and updated it to reflect the current state of what we're capturing.

Fri, Mar 29, 4:39 AM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

@MMiller_WMF @nettrom_WMF I updated the schema per our meeting earlier today. I'm not sure how you'd like to have the start module and impact module states logged @nettrom_WMF, so please have a look at those fields in particular and if you'd prefer a different format let me know.

Fri, Mar 29, 4:21 AM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)

Thu, Mar 28

nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

I'm a bit confused at this point about what we're proposing to do here, so here's a question to try to clear that up: is the suggestion here to use EditorJourney for capturing this, rather than have an event in the proposed HomePage schema with action=impression?

Yes. The proposal is to log to EditorJourney schema rather than create a new one.

Thu, Mar 28, 7:10 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219432: Homepage: HomepageVisit schema specification.

I'm a bit confused at this point about what we're proposing to do here, so here's a question to try to clear that up: is the suggestion here to use EditorJourney for capturing this, rather than have an event in the proposed HomePage schema with action=impression?

Thu, Mar 28, 6:33 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219431: Homepage: Mentorship and Help module schema usage.

Just noting the discussion we had in T219387, as mentioned there this change is OK with me! I've also updated the measurement specification so it mentions the HelpPanel schema and how we'll be using it.

Thu, Mar 28, 4:17 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Patch-For-Review, GrowthExperiments-Homepage, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T219387: Homepage: modify Schema:HelpPanel to accept events from Special:Homepage mentorship and help modules.

In theory I would favor splitting out the question asking part, as that's how the code has evolved over time – there's now a generic question posting API and a generic question poster JS widget that's then used by three different implementations. That said, it will be more work to revise this (both creating a new schema, and updating the code) so unless it's a big help to your analysis then I'd say we leave it as is.

Thu, Mar 28, 3:50 PM · Patch-For-Review, MediaWiki-extensions-GrowthExperiments, Growth-Team (Current Sprint)

Wed, Mar 27

nettrom_WMF added a comment to T217803: Homepage: experiment.

@MMiller_WMF : this looks good to go to me!

Wed, Mar 27, 8:58 PM · Growth-Team (Current Sprint)
nettrom_WMF closed T216185: Analyze use of mute preferences as Resolved.
Wed, Mar 27, 8:26 PM · Chinese-Sites, Anti-Harassment, Product-Analytics
nettrom_WMF added a comment to T219387: Homepage: modify Schema:HelpPanel to accept events from Special:Homepage mentorship and help modules.

Should we consider renaming the schema since it's becoming a more generic schema for posting questions to pages? Or split out the question-asking part from the search & links part (in other words: de-normalize the schema)?

Wed, Mar 27, 5:56 PM · Patch-For-Review, MediaWiki-extensions-GrowthExperiments, Growth-Team (Current Sprint)

Tue, Mar 26

nettrom_WMF closed T214125: Help panel: report on leading indicators as Resolved.

We've added table with the findings as a new section in the Help Panel experiment plan. There's also a description of the affected indicators and which ones we've acted upon.

Tue, Mar 26, 4:14 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF closed T214125: Help panel: report on leading indicators, a subtask of T206719: Help panel: instrumentation, as Resolved.
Tue, Mar 26, 4:14 PM · Growth-Team (Current Sprint), Product-Analytics

Mar 22 2019

nettrom_WMF added a comment to T216185: Analyze use of mute preferences.

I went back and dug into the data around the spike in Echo blacklist usage in October 2017 a bit more. From what I can tell, there doesn't seem to be a reason to suspect the data is invalid. Since the spike does make interpretation of the remainder of the data very difficult, and we're in this case mainly interested in development over time with a higher weight to more recent developments, I changed the start date of the Echo blacklist usage graphs to 2018-01-01 as that removes the spike.

Mar 22 2019, 9:22 PM · Chinese-Sites, Anti-Harassment, Product-Analytics
nettrom_WMF updated the task description for T218920: Welcome survey: turn off for Vietnamese Wikipedia.
Mar 22 2019, 4:11 PM · Patch-For-Review, Growth-Team (Current Sprint)

Mar 21 2019

nettrom_WMF added a comment to T218920: Welcome survey: turn off for Vietnamese Wikipedia.

@Trizek-WMF : I just posted T206380#5045440 with the results of the analysis, in case you'd like to refer to that.

Mar 21 2019, 6:25 PM · Patch-For-Review, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T206380: Personalized first day: experiments (Variation A).

I started this analysis on 2019-03-18, at which point we had 3,624 non-autocreated registrations since switching on the survey/control A/B test. Using the week of data prior to deployment I had earlier estimated the overall abandonment rate at 17.2%. A power analysis indicated that if the control group's abandonment rate equalled the estimate, we would be able to determine a significant change if the survey group's abandonment rate was outside the [13%,21%] range.

Mar 21 2019, 6:22 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF added a comment to T218835: prefUpdate schema contains multiple identical events for the same preference update.

I used data from this schema in T216185. My experience was similar to what @Tbayer mentions in that some preferences appear to have issues with duplication, and some do not. In this case the echo-notifications-blacklist and email-blacklist preferences were affected. It also seems that the issue changed over time, in other words that some users logged lots of preference changes during certain periods, and not during others.

Mar 21 2019, 4:20 PM · Product-Analytics, Analytics

Mar 20 2019

nettrom_WMF added a comment to T213974: EditorJourney records HTML tags in page_title field.

@Etonkovidova : Searched through the data from EditAttemptStep and HelpPanel in the Data Lake starting from 2019-01-01, and I didn't find any indications of HTML in the page_title field in either of those.

Mar 20 2019, 5:55 PM · MW-1.33-notes (1.33.0-wmf.19; 2019-02-26), MediaWiki-extensions-GrowthExperiments, Patch-For-Review, Growth-Team (Current Sprint)

Mar 18 2019

nettrom_WMF reassigned T217624: Homepage: analysis of visits to user and user talk pages from nettrom_WMF to MMiller_WMF.

Using all available data from EditorJourney, excluding known test users, and allowing "visit" to be either viewing the page or opening the editor, the results are as follows:

Mar 18 2019, 11:54 PM · Growth-Team (Current Sprint), Product-Analytics

Mar 13 2019

nettrom_WMF moved T217624: Homepage: analysis of visits to user and user talk pages from Triage to Next Up on the Product-Analytics board.
Mar 13 2019, 4:53 PM · Growth-Team (Current Sprint), Product-Analytics

Mar 12 2019

nettrom_WMF added a comment to T216185: Analyze use of mute preferences.

I have completed the snapshot analyses as well as some data cleanup to remove the graph spike in email usage in July 2017. Will do a little bit more work to look into the Echo notification usage spike in October 2017 but I suspect that could be initial adoption and that we might prefer to omit the first few months and start graph on January 2018.

Mar 12 2019, 4:58 PM · Chinese-Sites, Anti-Harassment, Product-Analytics

Mar 11 2019

nettrom_WMF added a comment to T215986: Homepage: help module.

@nettrom_WMF we discussed this in T215986#4970275. IIRC the reasoning was as you described – it somewhat duplicates existing search, and we didn't have a clear spec for how the search should be implemented in the UI.

Mar 11 2019, 6:11 PM · MW-1.33-notes (1.33.0-wmf.23; 2019-03-26), Patch-For-Review, MediaWiki-extensions-GrowthExperiments, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T215986: Homepage: help module.

What's the current state of search in this module? I went back and read through the comment above about the result of the deep dive, but didn't see much about it since then (and it's not present in the mockup). Curious about that, as it'll inform the measurement specification.

Mar 11 2019, 6:01 PM · MW-1.33-notes (1.33.0-wmf.23; 2019-03-26), Patch-For-Review, MediaWiki-extensions-GrowthExperiments, Growth-Team (Current Sprint)

Mar 8 2019

nettrom_WMF updated the task description for T216185: Analyze use of mute preferences.
Mar 8 2019, 11:09 PM · Chinese-Sites, Anti-Harassment, Product-Analytics
nettrom_WMF added a comment to T216185: Analyze use of mute preferences.

@TBolliger : Thanks again for providing those tasks and dates, much appreciated, and I'll make sure to incorporate those as I continue this work.

Mar 8 2019, 11:07 PM · Chinese-Sites, Anti-Harassment, Product-Analytics

Mar 7 2019

nettrom_WMF added a comment to T211950: Add partial blocks to mediawiki history tables.

However I think that logging table doesn't contain detailed historical information on partial blocks. This will prevent us to rebuild historical (and therefore more interesting) information on partial-blocks.
Can you talk to your team see if more detailed logging could be developped?

Mar 7 2019, 7:05 PM · Analytics-Kanban, Product-Analytics, Anti-Harassment, Analytics
nettrom_WMF added a comment to T206380: Personalized first day: experiments (Variation A).

We've completed our initial experiment and found no obvious detrimental effect from the survey. We've also run a second experiment against variation C, and found that Var A is preferable. Currently, we are running an experiment on Vietnamese Wikipedia with Var A and a control group, to learn more about the abandonment rate on that wiki (ref T216668 and T216669).

Mar 7 2019, 1:47 AM · Growth-Team (Current Sprint), Product-Analytics

Mar 6 2019

nettrom_WMF added a comment to T215664: Help panel: show in additional contexts.

Analysis by @MMiller_WMF and some digging by me indicates that events are not recorded correctly if the Visual Editor is used. Those events are instead recorded as reading events.

Mar 6 2019, 9:31 PM · MW-1.33-notes (1.33.0-wmf.20; 2019-03-05), MediaWiki-extensions-GrowthExperiments, Patch-For-Review, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T215664: Help panel: show in additional contexts.

We monitored the SWAT deployment of the patch yesterday and confirmed through Kafka that reading events were flowing in. I've now also confirmed that reading events are present in the Data Lake.

Mar 6 2019, 6:31 PM · MW-1.33-notes (1.33.0-wmf.20; 2019-03-05), MediaWiki-extensions-GrowthExperiments, Patch-For-Review, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T216185: Analyze use of mute preferences.

During the analysis of usage of the email and Echo notification block lists, an issue with spikes in usage has come up. One key question about analyzing historical data of features like these is when these features were available, and in what capacity. For example, the email block list might've been available as a beta feature for a while. Secondly, there might've been announcements of the feature being available that would affect usage. Having that kind of information available aids analysis as it removes guesswork around why certain patterns emerge, so I'm documenting that here for future reference in other analysis tasks.

Mar 6 2019, 5:40 PM · Chinese-Sites, Anti-Harassment, Product-Analytics

Mar 5 2019

nettrom_WMF added a comment to T215664: Help panel: show in additional contexts.

I couldn't find any reading-related events in the Data Lake, nor was I able to see any flow in when I monitored the events in Kafka. Instead, I find that my browser throws the following error to the JavaScript console:

Mar 5 2019, 6:03 PM · MW-1.33-notes (1.33.0-wmf.20; 2019-03-05), MediaWiki-extensions-GrowthExperiments, Patch-For-Review, Growth-Team (Current Sprint)

Mar 1 2019

nettrom_WMF added a comment to T217391: Help panel: improve links in Czech help panel.

@nettrom_WMF -- do you think it will be better to manage the change to the "guide" link if the name in the configuration also changes? Or should we leave it as "guide" because its destination hasn't changed? We may want to look at the difference in traffic before and after the change was made.

Mar 1 2019, 9:24 PM · Patch-For-Review, Growth-Team (Current Sprint), User-Urbanecm, MediaWiki-extensions-GrowthExperiments

Feb 28 2019

nettrom_WMF closed T217377: Blank the event.page_title column in the editorjourney table in the Data Lake as Declined.

We're fine with letting this data get purged as it otherwise would, so I'm closing this.

Feb 28 2019, 11:33 PM · Growth-Team, Analytics
nettrom_WMF added a comment to T215664: Help panel: show in additional contexts.

Just adding a comment to document that since we're also planning to soon add User & User talk, I'll regard the timestamp of that going live as the start of experimentation with additional contexts.

Feb 28 2019, 9:55 PM · MW-1.33-notes (1.33.0-wmf.20; 2019-03-05), MediaWiki-extensions-GrowthExperiments, Patch-For-Review, Growth-Team (Current Sprint)
nettrom_WMF created T217377: Blank the event.page_title column in the editorjourney table in the Data Lake.
Feb 28 2019, 9:45 PM · Growth-Team, Analytics
nettrom_WMF reassigned T216669: Welcome survey: change experiment to include control group from nettrom_WMF to MMiller_WMF.

It's been about 7 hours since this went live, and using data from the replicated database, I get the following overview:

Feb 28 2019, 12:43 AM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T215986: Homepage: help module.

Regarding instrumentation, I'll ping @nettrom_WMF for his thoughts on whether this should go into the HelpPanel schema or not. My thought is that given that we are writing separate code for this help module, and that it will behave somewhat differently than the help panel, it might give us more flexibility for it to store data separately.

Feb 28 2019, 12:20 AM · MW-1.33-notes (1.33.0-wmf.23; 2019-03-26), Patch-For-Review, MediaWiki-extensions-GrowthExperiments, Growth-Team (Current Sprint)

Feb 27 2019

nettrom_WMF moved T216672: EditorJourney and Welcome Survey: second report from Triage to Next Up on the Product-Analytics board.
Feb 27 2019, 5:27 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF moved T216672: EditorJourney and Welcome Survey: second report from Incoming to Ready for Development on the Growth-Team (Current Sprint) board.
Feb 27 2019, 5:27 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF edited projects for T216672: EditorJourney and Welcome Survey: second report, added: Growth-Team (Current Sprint); removed Growth-Team.
Feb 27 2019, 5:26 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF moved T216586: Homepage: instrumentation from Triage to Doing on the Product-Analytics board.
Feb 27 2019, 5:25 PM · Analytics, MediaWiki-extensions-GrowthExperiments, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T216586: Homepage: instrumentation from Incoming to In Progress on the Growth-Team (Current Sprint) board.
Feb 27 2019, 5:25 PM · Analytics, MediaWiki-extensions-GrowthExperiments, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF edited projects for T216586: Homepage: instrumentation, added: Growth-Team (Current Sprint); removed Growth-Team.
Feb 27 2019, 5:25 PM · Analytics, MediaWiki-extensions-GrowthExperiments, Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF moved T210868: Personalized first day: experiments (Variation C) from Triage to Doing on the Product-Analytics board.
Feb 27 2019, 5:12 PM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF added a project to T210868: Personalized first day: experiments (Variation C): Product-Analytics.
Feb 27 2019, 5:12 PM · Product-Analytics, Growth-Team (Current Sprint)

Feb 26 2019

nettrom_WMF updated the task description for T215666: Help panel: deploy on Vietnamese Wikipedia.
Feb 26 2019, 5:41 PM · Growth-Team (Current Sprint), Patch-For-Review
nettrom_WMF added a comment to T215666: Help panel: deploy on Vietnamese Wikipedia.
  • @nettrom_WMF can look to see if events are flowing to the data lake.

    We would like all three of these things to be complete by Wednesday, so that we can start the 50/50 experiment on Thursday.
Feb 26 2019, 5:41 PM · Growth-Team (Current Sprint), Patch-For-Review

Feb 25 2019

nettrom_WMF moved T216669: Welcome survey: change experiment to include control group from Triage to Doing on the Product-Analytics board.
Feb 25 2019, 9:32 PM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF reassigned T216669: Welcome survey: change experiment to include control group from nettrom_WMF to MMiller_WMF.
Feb 25 2019, 9:31 PM · Product-Analytics, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T216669: Welcome survey: change experiment to include control group.

I've completed a quick analysis of the abandonment rate using the eight days of of data we have between the deployment of EditorJourney (on 2019-01-16) and the Welcome Survey (on 2019-01-24). In this analysis, I used the same approach that I used for the control group during the initial Welcome Survey experiment: does the user have more than three events[1] logged in the EditorJourney data? Split by whether the account was created on the mobile or desktop site, for Vietnamese the result is:

Feb 25 2019, 9:31 PM · Product-Analytics, Growth-Team (Current Sprint)

Feb 22 2019

nettrom_WMF added a comment to T215664: Help panel: show in additional contexts.

@nettrom_WMF just noting that, when the help panel is loaded in the context of creating a new article, page ID will be 0. Of course you can filter these out by looking at page_id=0 AND editor_interface='reading' but just wanted to note it. For reading context, I could set page ID to -2 or some numeric value you'll know means it was redacted, just let me know.

Feb 22 2019, 6:46 PM · MW-1.33-notes (1.33.0-wmf.20; 2019-03-05), MediaWiki-extensions-GrowthExperiments, Patch-For-Review, Growth-Team (Current Sprint)
nettrom_WMF added a comment to T215664: Help panel: show in additional contexts.

@nettrom_WMF here's an example event:

{
  "action": "impression",
  "action_data": "",
  "user_id": 1,
  "user_editcount": 127,
  "editor_interface": "reading",
  "is_mobile": false,
  "page_id": 0,
  "page_title": "",
  "page_ns": 12,
  "user_can_edit": true,
  "page_protection": "",
  "session_token": "ac0bc60ed5b559989816",
  "help_panel_session_id": "97487760b5c4b4519e76"
}

Note that page ID has to be an integer, not omitted entirely, so I've set it to 0. Let me know if that should be something else. For page title, I set it to an empty string, but I could make it some other value if you prefer.

Feb 22 2019, 12:58 AM · MW-1.33-notes (1.33.0-wmf.20; 2019-03-05), MediaWiki-extensions-GrowthExperiments, Patch-For-Review, Growth-Team (Current Sprint)

Feb 21 2019

nettrom_WMF closed T212821: Help panel: post-deployment QA of event logging as Resolved.

The data captured by the HelpPanel schema has been QA'ed in various fashions since deployment. We monitored and verified data during deployment, have been investigating it afterwards, and also have checked it during analysis. Closing this task as resolved.

Feb 21 2019, 8:32 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF closed T212821: Help panel: post-deployment QA of event logging, a subtask of T206719: Help panel: instrumentation, as Resolved.
Feb 21 2019, 8:32 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF closed T206719: Help panel: instrumentation as Resolved.

The instrumentation document has gone through the DACI and received approval. We've put the instrumentation in production and monitored the data periodically, and things seem to be going well. Am therefore closing this task.

Feb 21 2019, 8:20 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF closed T206719: Help panel: instrumentation, a subtask of T206711: [EPIC] Growth: Focus on help desk (in-context link), as Resolved.
Feb 21 2019, 8:20 PM · MediaWiki-extensions-GrowthExperiments, Epic, Growth-Team
nettrom_WMF added a comment to T216669: Welcome survey: change experiment to include control group.

@nettrom_WMF -- EditorJourney was deployed in Vietnamese on Jan 16 and the welcome survey was deployed on Jan 24. Could we use the period in between to get a sense of the abandonment rate of users who don't get a survey (even though it's not a real A/B test because of the temporal change?)

Feb 21 2019, 6:12 PM · Product-Analytics, Growth-Team (Current Sprint)

Feb 20 2019

nettrom_WMF created T216658: Timestamp column in EventLogging tables have incompatible collation.
Feb 20 2019, 9:48 PM · Analytics, Product-Analytics
nettrom_WMF moved T216185: Analyze use of mute preferences from Triage to Doing on the Product-Analytics board.
Feb 20 2019, 8:12 PM · Chinese-Sites, Anti-Harassment, Product-Analytics

Feb 14 2019

nettrom_WMF added a comment to T215664: Help panel: show in additional contexts.

Storing reading there seems OK but not great. We could consider a new property on the schema like impression_mode and set that to "edit" or "view". @nettrom_WMF do you have a preference?

Feb 14 2019, 9:17 PM · MW-1.33-notes (1.33.0-wmf.20; 2019-03-05), MediaWiki-extensions-GrowthExperiments, Patch-For-Review, Growth-Team (Current Sprint)
nettrom_WMF created T216185: Analyze use of mute preferences.
Feb 14 2019, 9:08 PM · Chinese-Sites, Anti-Harassment, Product-Analytics

Feb 13 2019

nettrom_WMF added a comment to T215664: Help panel: show in additional contexts.

@MMiller_WMF : We should document the questions we have around this variation, similarly as we did for search. I think at this point, my main interest is behaviour across namespaces, in other words: how do users who see the help panel in the Help & Wikipedia namespaces differ in behaviour from those who see it when editing? We'll be able to answer that because we track the page namespace. While it might be interesting to know if certain pages are more likely to lead to questions being asked, I suspect we might also learn about that from the questions that are asked? It could also be something we consider looking into if we start seeing questions flow in.

Feb 13 2019, 9:07 PM · MW-1.33-notes (1.33.0-wmf.20; 2019-03-05), MediaWiki-extensions-GrowthExperiments, Patch-For-Review, Growth-Team (Current Sprint)
nettrom_WMF changed Due Date from Jan 9 2019, 12:00 AM to Mar 2 2019, 12:00 AM on T212464: Help panel: decide on a purging strategy for the HelpPanel schema.
Feb 13 2019, 7:05 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF set Due Date to Jul 11 2019, 7:00 AM on T206720: Help panel: experiments.
Feb 13 2019, 7:04 PM · Growth-Team (Current Sprint), Product-Analytics
nettrom_WMF moved T201656: Growth: 2018-2019 annual plan metrics report from Inbox to Q4 2018-19 on the Growth-Team board.
Feb 13 2019, 7:04 PM · Growth-Team, Product-Analytics
nettrom_WMF edited projects for T201656: Growth: 2018-2019 annual plan metrics report, added: Growth-Team; removed Growth-Team (Current Sprint).
Feb 13 2019, 7:04 PM · Growth-Team, Product-Analytics
nettrom_WMF moved T201656: Growth: 2018-2019 annual plan metrics report from Ready for Development to Incoming on the Growth-Team (Current Sprint) board.
Feb 13 2019, 7:02 PM · Growth-Team, Product-Analytics
nettrom_WMF changed Due Date from Nov 30 2018, 8:00 AM to Jun 1 2019, 7:00 AM on T201656: Growth: 2018-2019 annual plan metrics report.
Feb 13 2019, 7:02 PM · Growth-Team, Product-Analytics

Feb 11 2019

nettrom_WMF added a comment to T214967: Homepage: comparative review of personalized user pages and dashboards.

Apart from "recommendations work", I'm not sure which parts of SuggestBot are relevant :) Maybe that at we did attempt to simplify the process of getting suggestions for newcomers through WP:Teahouse/Suggestions. It's still complicated, though, partly because we were operating within the constraints of MediaWiki and SuggestBot's infrastructure.

Feb 11 2019, 6:04 PM · Growth-Team (Current Sprint)

Feb 8 2019

nettrom_WMF moved T210434: Understanding first day: analysis caveats from Backlog to Tracking on the Product-Analytics board.
Feb 8 2019, 9:38 PM · Product-Analytics, Growth-Team
nettrom_WMF updated the task description for T210434: Understanding first day: analysis caveats.
Feb 8 2019, 9:37 PM · Product-Analytics, Growth-Team
nettrom_WMF added a comment to T213974: EditorJourney records HTML tags in page_title field.

I assume you prefer to just have [obfuscated page title string] and not [obfuscated surrounding elements and page title string]. In other words, don't include the surrounding elements in the string that's going to be obfuscated, but purge them entirely. Is that right?

Feb 8 2019, 9:35 PM · MW-1.33-notes (1.33.0-wmf.19; 2019-02-26), MediaWiki-extensions-GrowthExperiments, Patch-For-Review, Growth-Team (Current Sprint)