Page MenuHomePhabricator

[REQUEST] Update QA process to account for expected differences in event counts for iOS/Android
Closed, DeclinedPublic

Description

Name for main point of contact and contact preference
@DAbad - tag in Phab or Slack

What teams or departments is this for?
Product Data Engineering

What are the details of your request? Include relevant timelines or deadlines
Following investigations in https://phabricator.wikimedia.org/T281001 and https://phabricator.wikimedia.org/T281002, the QA process should be updated to indicate that we can expect event counts to differ.
For additional context, see https://phabricator.wikimedia.org/T281002#7107821 and notes from 2021-05-24 PDE Technical Workshop.

Is this request urgent or time sensitive?
No

Event Timeline

Thanks, @DAbad ! The Product Analytics team will prioritize this request in our next board refinement meeting (June 1).

ldelench_wmf triaged this task as Medium priority.

Looked into the discrepancies and work arounds that have been discussed in the tasks listed in the description. Before we update the QA process I wanted to make sure we check to see if this impacts the overall insights taken away from the data as recommended in this comment .

Additionally (Product Analytics) could check to see if this impacts the overall insights taken away from the data. Look at a higher volume dataset like (session ticks) to analyze.

Spoke to @SNowick_WMF about this. She is looking into comparing an analysis done using the legacy schema

android_image_recommendation_interaction

that has now been migrated to MEP (see T281730) .

Update: @SNowick_WMF has completed the data-QA for android_image_recommendation_interaction and found many discrepancies between the legacy and modern schema . see T281730#7144107
Since our focus with migrating legacy schemas to MEP is currently on hold for the release of Metrics Platform I think we should revisit the discussion around changing the QA process at a later time, after carefully understanding the impact of changes in data collection on analyses.

Mayakp.wiki lowered the priority of this task from Medium to Low.Jun 8 2021, 10:11 PM
Aklapper changed the subtype of this task from "Deadline" to "Task".Apr 26 2023, 8:43 AM

Hi @SNowick_WMF : do you think this task is still relevant? do the counts differ between legacy and MEP for mobile apps? can you pls decline this task if thats not the case. and if it is, then reassign to @nettrom_WMF ?

SNowick_WMF closed this task as Declined.EditedSep 22 2023, 7:43 PM
SNowick_WMF removed Mayakp.wiki as the assignee of this task.
SNowick_WMF added a subscriber: Mayakp.wiki.

Declining task - we did find discrepancies but differences were within the acceptable variance range https://phabricator.wikimedia.org/T264193#6625426.