@Iflorez ,
Can you let me know a little more about the issue you are facing?

Drafted the announcement and shared with @EChetty on 9/12 to review and provide feedback.

Per discussion in today's Board Refinement meeting moving this task to Tracking for Product Analytics as the scope of this task has been changed.
Assigning the task to data-engineering to fulfill the request.

@ldelench_wmf : Can you please let us know if there is any further support needed from Product Analytics on this task?

Added this to data engineering's radar.

First draft of Data Loss Estimation Recommendations is ready to be shared out with GDI and Research.
I will also share it with Product Analytics during team sharing meeting tomorrow.

Discussed during our 1:1. We have a couple of approaches in mind -

• In favor of providing a warning, on datahub or Wikitech.
• Hive: when you run a query on pageview data tables (webrequest, pageview_hourly, pageview_daily etc.), a header will say “this data is incomplete..” and should be available on all tools using Hive.
• Any local copy of the data will not have that flag
• Remove data from that period
• Provide two columns - Observed data and projected data
• Remove incorrect pageview data from that period from all public datasets
  • API - will have a header, similar to hive that will say “this data is incomplete..”
  • Dumps: community datasets - a separate section which has data loss. Separate flat file with that dump, with the data loss with a warning acknowledging the loss.
  • Separate action you have to take to go access internal data for observed data

Adding here for reference: Impact Analysis: Editors_Daily

We had sessions with Data Engineering and Product Analytics where we discussed the design for the table. On a high level we are going to build a table derived from mediawiki_history and mw_user_history that calculates data in an easier way for product managers and analysts to query and dashboard. The table may not be on Druid (coz Druid has a few limitations as we know).  We will focus on building it in Hive, and accessible via Presto.
The table will provide editors activity on a daily basis and will be refreshed monthly (for now; we will think about making it real time in the future). For the mvp we are not focussing on retention specific use cases, but having editors data on a daily basis would help calculate basic retention. Details are given in the meeting doc.

Completed. Provided in this Document (accessible to WMF-only)

A few more suggestions as I have been using the data in this table for regional breakdown of pageviews for Metrics that Matter T316161

• Missing countries: there are a few countries in pageview_hourly but missing in country_meta_data

Github Readers repo didnt add the folder correctly which means next month I will have to run the Visualization code from @mpopov 's https://gitlab.wikimedia.org/bearloga/dataviz-content-interactions#data-visualization-content-interactions
Or try to add the Viz repo as a git submodule.

I have tried to do this using 2 methods + data from ntsako.country_meta_data : 1. manually from Superset (druid.pageviews_daily) and 2. Spark query (wmf.pageview_hourly)

Hi @ntsako / @JAnstee_WMF
is there a difference between these 2 columns? or are they redundant as well ?

I wanted to note here that Product Analytics relies on the editors_daily table to calculate diversity metrics. While we were aware of the creation of a new aggregated table to calculate Unique Editor metrics by Country, this task doesn't explicitly mention that user_fingerprint_or_id will be changed to user_fingerprint_or_name .
Changes to the source table has impacted global north and global south metrics such as edits, editors, net new content etc. as well as the analytics-wmf-product-jobs that updates the wmf_product.active_editors table which provides data to the editors dashboard.

Right! ok. I will discuss further with the team and get back to you.

Thanks @mpopov !
I made a few changes to the title, as discussed on Slack. I will reach out if I have any other questions.

Thanks @JMando .
To make sure I understand, something like

1. fiscal_year column for all the records in both the tables
2. fr_campaign column which will be populated for the pageview records where the campaign was running.
   • Example: pageviews from Nov 25 - Dec 31 from US, CA, IE, GB, NZ, AU will have a value like 'en6c Campaign'

I also checked if there were any translation events happening on kn.wikipedia and nothing stands out

We will need to meet with GDI to get a consensus on their team's mapping of country codes and various labels and regional groupings implemented in T310712

@JMando : could you perhaps provide the calculated metrics and columns that FR would like to add to the pageview datasets ?

Listing a few themes from the discussion that could be worth pursuing aka creating a task.

• How do our trends compare to global trends?
  • What other groups should we be talking within the Foundation to understand external trends better so we can interpret what we’re seeing in a global context?
  • Are we talking about internet access or engagement? [what are the global trends for both?]
  • How do these numbers compare with other large internet platforms, especially older/mature products used globally?

At first glance, this seems relatively easy to do -

@EChetty, @Milimetric , @MMiller_WMF , @Iflorez and I met today to discuss use cases, existing datasets, limitations and scope of this project.
PA & DE will re group in the coming weeks to discuss the initial table design.

Hi @EYener, @EChetty and I spoke briefly about this and wanted to know the calculated metrics and columns that you would like to add to the existing pageview datasets.
We can decide the best approach once we get an idea of what FR is interested in. Thanks!

Thanks @phuedx for the update! It is super useful to have these QA fields. Pls feel free to let me know whenever you would like to have Product-Analytics represented in these discussions.

I have a feeling we might be blocked by this task T313955

Hi @phuedx, do we have a 'test' or 'debug' field in the Event schema?
Didn't see it listed here - https://wikitech.wikimedia.org/wiki/Metrics_Platform/Event_Schema

@kzimmerman : This is done. May metrics summary has been added to Commons and Mediawiki.

Hi @JArguello-WMF / @EChetty
Can we rename this task to improve discoverability in the future ?
Suggestions:

• Pageview Data loss due to wrong version of package installed on some varnishkafka instances
• Pageview Traffic dropped on some varnishkafka instances between June 2021 and January 2022

Moved subtasks related to "improving" the metrics calculation to a new Parent task T312880
*Inspiration Week reorganization :)