Page MenuHomePhabricator

[MI 3] Monitor and investigate movement trends
Open, HighPublic

Description

The goal:

Wikimedia Foundation leadership is consistently equipped with a clear narrative about threats to, and opportunities within, the Wikimedia ecosystem supported by confident analysis of internal metrics and external trends.

More details in the project overview document (WMF only).

Event Timeline

nshahquinn-wmf triaged this task as High priority.

Last week, I:

  • Worked on the monitoring runbook and dashboard
  • Discussed the trends catalog with Maya
  • Dug back into data on the fundraising issues
  • Produced updated charts of traffic from Google
nshahquinn-wmf moved this task from Backlog to Epics on the Movement-Insights (FY25-26 H2) board.

Last week, I:

  • Continued discussions about reporting venue, audience, and frequency
  • Worked through several different Superset issues
  • Dug into new Comscore data and compared it with our unique device data
  • Answered lots of data questions from Suman

Last week, I:

  • Drafted and shared first “official” trends brief
  • Responded to follow-up questions on brief
  • Discussed plans with Sam NeSmith

Last week, I:

Last week, I:

  • Drafted the trends brief
  • Discussed needs and concerns around monthly reporting with Maya and Sam
  • Improved the Google Search Console data workflow and monitoring notebook
  • Documented monitoring notebooks and walked Maya through them
  • Helped Maya get dependencies for the monitoring notebooks installed, dealing with a bunch of stubborn environment issues in the process

This week, I:

  • Paired with Maya to do the weekly monitoring
  • Clarified leadership requirements around distribution of the fortnightly trends brief and the continuation of the monthly metrics report
  • Made visual improvements to the Google Search Console monitoring
  • Set up a basic monitoring notebook for daily editor counts
  • Made a plan for fetching Google Search Console data through an API rather than through tedious manual data exports

This week, I:

  • Did the weekly monitoring
  • Explored and discussed infrastructure needs for this work (e.g. incremental mediawiki_history, general availability of Dbt)
  • Worked on getting service account access to exported search console data in Google Cloud (via round after round of getting permissions and then finding that they weren’t correctly applied or were insufficient)

Last week, I:

  • Drafted the trends brief
  • Improved Google Search monitoring notebook so that it automatically fetches latest data from BigQuery
  • Came up with proposed priorities in preparation for Monday prioritization meeting
  • Started learning about Dbt

This week, I:

  • Discussed priorities, particularly related to infrastructure needs and to convergence with the funnel model and weekly metrics
  • Learned about the content observability project (via the unusual route of investigating who kept deleting my BigQuery service account keys!) and identified shared interest in ingesting Google Search Console data into the Data Lake
  • Continued conversations with DPE about data needs for weekly contributor metrics
  • Shared suggestions about defining “weekly metrics”, particularly that they should be weekly (rather than daily) but calculated on daily basis
  • Did the weekly monitoring

Last week, I:

Last week, I:

  • Wrote the trends brief
  • Organized conversations about the time grain of contributors metrics
  • Gave detailed input on importing Google Search Console data to the Data Lake (T420996)
  • Updated the runbook to fully document the monitoring process
  • Came up with a proposal for the next hypothesis

The week before last, I:

  • Wrote the trends brief
  • Gave extensive input on the incremental mediawiki_history plan (in meeting and in T425573#11929101)
  • Started backfilling revised readership data in the movement-metrics repo

Last week, I:

Last week, I:

  • Wrote the trends brief
  • Added Google Search impressions of Wikipedia to the monitoring and brief
  • Worked on definitions for weekly active editors and weekly active admins with Dan Andreescu
  • Created a daily editor activity Superset dashboard

This week, I:

  • Reviewed Dan’s MR for human pageviews and account registrations metrics
  • Extensively reviewed and discussed Caroline’s monthly active moderators SQL
  • Figured out that the inconsistencies in Google Search impressions data were due to the difference between property-aggregated and page-aggregated data

This week, I:

  • Wrote the trends brief
  • Detected, documented, and fixed a bug in our calculation of emerging wiki active admins
  • Fixed Google Search Console queries to consistently use property-aggregated data
  • Wrote a detailed documentation page on Google Search Console data quirks
  • Reviewed Dan’s weekly active editors MR
  • Dug into data quirks around account registration (one example: T429061)
  • Drafted a Dbt model for account registrations (T422732)

This week, I:

Last week, I:

  • Wrote the trends brief
  • Worked with Aleksandr Mastilovic to finish backfilling base_account_registration and editor_month (T430602)
  • Movement metrics query updates (T431647)
    • Improved the movement metrics SQL machinery to make backfilling easier
    • Rewrote queries
    • Spent a long time investigating why the changes caused 10-20% decreases in the calculated retained newcomers at Wikidata and Commons

This week, I:

  • Movement metrics query updates (T431647)
    • Figured out the cause of the major changes in retained newcomers at Wikidata and Commons
    • Finalized the query changes and reviewed them with Maya
    • Ran the backfill

Last week, I:

  • Wrote the trends brief
  • Finished and documented the impact of the contributors monthly metric backfill (T431647)

This week, I:

  • Wrote and released the June movement metrics report (T430608)
  • Added metric definition pop-ups to the movement metrics report (T430473)
  • Responded to an on-wiki question about why we report on user and automated pageviews, but not spider
  • Collected some documentation about the 2020 introduction of bot detection to pageviews and cleaned up some old bot detection tickets (e.g. T138207#12177265, T425661#12177268)
  • Started cleaning up the heartbeat metrics on the FY metrics dashboard and noting inconsistencies

Last week, I:

  • Investigated recent Google Search trends
  • Wrote the trends brief