Page MenuHomePhabricator

Understanding first day: analysis caveats
Closed, ResolvedPublic

Description

When analyzing the data related to the Editor Journey project (aka "Understanding first day"), we'll run into challenges and caveats. This task tracks those.

  • The Growth Team has a bunch of test accounts that have been used for this survey. I'll have to put together a list of those and filter them out.
  • Accounts created through the API, which is mainly app accounts, are assigned a survey group and therefore appears to have gotten the survey. We don't support the apps at the moment, so those need to be filtered out (they can be identified through the ServerSideAccountCreation schema's data).
  • T210003 results in inconsistent hashing for the first few events for some users. We'll be able to collect a list of affected user IDs and exclude those from analysis if necessary.
  • T210004 results in the CreateAccount event in the EditorJourney schema being out of order, it shows up as the second event but should always be first. Might require me to ignore those.
  • T210417 is worth noting, but should not be an issue for us as we're not interested in what specifically users we're reading/editing prior to creating their account.
  • T213974 is worth noting. It is currently not an issue for us as we do analysis based on namespace and title. If we're doing analysis where we're examining the full page title, it is necessary to either strip the HTML out or figure a way around it.

Event Timeline

nettrom_WMF updated the task description. (Show Details)Nov 26 2018, 7:19 PM
nettrom_WMF updated the task description. (Show Details)Nov 26 2018, 7:27 PM
kostajh moved this task from Inbox to FY 2019-20 on the Growth-Team board.Nov 27 2018, 2:36 PM
nettrom_WMF moved this task from Triage to Backlog on the Product-Analytics board.Dec 20 2018, 6:34 PM
JTannerWMF moved this task from FY 2019-20 to Q2 2019-20 on the Growth-Team board.Jan 2 2019, 6:01 PM
nettrom_WMF updated the task description. (Show Details)Feb 8 2019, 9:37 PM
nettrom_WMF moved this task from Backlog to Tracking on the Product-Analytics board.
JTannerWMF moved this task from Q2 2019-20 to Q1 2019-20 on the Growth-Team board.Jun 17 2019, 6:52 PM

Hey @nettrom_WMF are you still using this task? Should we leave it open or is it okay to close?

nettrom_WMF closed this task as Resolved.Jul 8 2019, 5:08 PM
nettrom_WMF triaged this task as Normal priority.

@JTannerWMF : thanks for the ping! I think was an interesting experiment with regards to documenting issues, and while our data gathering and analysis continues, I don't see a strong reason to keep this open.