Page MenuHomePhabricator
Create Task
Maniphest T133733

Portal Dashboard: Add de-duplication step to data collection
Closed, InvalidPublic2 Estimated Story Points
Actions

Description

@JGirault noticed a discrepancy between the clickthrough rate on the Portal dashboard and the rate reported in the most recent Portal A/B test analysis.

The only difference is that the data used in the A/B test report underwent an additional cleaning step wherein duplicated events were removed. Therefore the clickthrough rate that is surfaced on the Portal dashboard is calculated using faulty data that has A LOT of duplicate events (any session should have at most 1 landing event and 1 clickthrough event).

While work is being done on T133730 to correct this on the event logging side, we should fix how the action/clickthrough data is collected and processed for the Portal dashboards.

We will be able to backfill about a month of "corrected data" (~57% new vs ~34% old) and this change should be annotated on the dashboard.

Related Objects
Search...

Event Timeline

mpopov created this task.Apr 26 2016, 7:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 26 2016, 7:34 PM
mpopov set the point value for this task to 2.Apr 26 2016, 7:35 PM
debt renamed this task from Add de-duplication step to Portal dashboard data collection to Portal Dashboard: Add de-duplication step to data collection.Apr 26 2016, 7:47 PM
debt triaged this task as High priority.
debt added a project: Discovery-ARCHIVED.
debt updated the task description. (Show Details)
debt added a subtask: T133730: Portal Event logging: Figure out why Schema:WikipediaPortal has duplicate events & fix it.
debt added subscribers: • JGirault, Jdrewniak.
• Deskana edited projects, added Discovery-Analysis (Current work); removed Discovery-Analysis.Apr 26 2016, 8:10 PM
mpopov added a comment.May 2 2016, 5:17 PM

I don't think this task is valid anymore since there's actually no bug in the EL and it's actually working as intended. The only problem is that we're currently not accounting for first visits or subsequent visits. The CTR on the dashboard is the overall CTR. If someone goes the Portal 50 times but only clicks through 40 times, those 50 and 40 events will be used in the CTR calculation.

We might want to, I think, cancel this task or repurpose it to instead set up a new metric to track: first impression CTR (which is what I've been using to assess our A/B tests). Just a thought.

• Deskana closed this task as Invalid.May 2 2016, 8:12 PM
• Deskana subscribed.
In T133733#2257281, @mpopov wrote:

I don't think this task is valid anymore since there's actually no bug in the EL and it's actually working as intended. The only problem is that we're currently not accounting for first visits or subsequent visits. The CTR on the dashboard is the overall CTR. If someone goes the Portal 50 times but only clicks through 40 times, those 50 and 40 events will be used in the CTR calculation.

Closing as invalid based on this.

We might want to, I think, cancel this task or repurpose it to instead set up a new metric to track: first impression CTR (which is what I've been using to assess our A/B tests). Just a thought.

Can you file a new task for that one? We can prioritise it in our meeting tomorrow.

mpopov added a comment.May 3 2016, 6:01 PM
This comment was removed by mpopov.
debt subscribed.May 3 2016, 6:01 PM

The new task is here: T134199

debt closed subtask T133730: Portal Event logging: Figure out why Schema:WikipediaPortal has duplicate events & fix it as Invalid.May 3 2016, 6:02 PM
debt moved this task from Backlog to Resolved on the Discovery-Analysis (Current work) board.Jun 7 2016, 7:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL