Request Status: New Request
Request Type: project support request
Related OKRs: Metrics that matter, Knowledge Equity - Regional views

Request Title: "Source of truth" dataset for pageviews

• Request Description: The pageview data we have available in pageviews_daily and pageviews_hourly has known data issues that make using those datasets difficult, and the risk of drawing faulty conclusions is high:
  • Highest priority: Between June 2021 and January 2022, there was a data loss that impacted pageviews, particularly in certain regions like the Americas (see T300164). When we explore the data using Wikistats or Superset, it's easy to forget about the data loss and draw conclusions about the trends that are incorrect and even directionally inaccurate. Some people in the Foundation and our communities don't know about this data loss in the first place. We also don't have a standardized calculations (yet) for correcting the data in that time period, and continuing to do calculations on an ad hoc basis introduces more room for error.
  • In late 2019 and early 2020, there was a large amount of (suspected) bot activity that skewed pageviews, particularly in the United States (see T239811). This happened before Data Engineering implemented a flag for automated traffic. We frequently use 2019 as a pre-pandemic baseline in analysis but need to correct for the bot activity when looking at US traffic and none referrers.
  • There are several other similar issues that we'll also want to consider, to be added later.
• Indicate Priority Level: high
• Main Requestors: @kzimmerman
• Ideal Delivery Date: FY22-23
• Stakeholders:

Request Documentation