Page MenuHomePhabricator

Global proportions outputs from Equity Landscape inputs
Open, Stalled, MediumPublic

Description

Develop annual output table of global proportions from Equity Landscape inputs which have been making appearances in quarterly learning sessions

Output-side global proportions in the form of should be tabled by location and year for:

  • population
  • internet population
  • pageviews
  • unique devices
  • editors
  • active editors
  • grant dollars (weighted and unweighted)
  • grants
  • grantees
  • operating affiliates
  • primarily located affiliates

Each proportion should be based on the location's count over the global count as denominator to produce percentages for each location that sum to 100% with the exception that, for operating affiliates, the denominator should be total affiliates (rather than total operating hits) and the sum across locations will exceed 100%.

Source tables and data:
population_data_input_metrics

  • population
  • internet population (Note: for this we need to multiply total population_annual_signal by (internet_percent_annual signal/100) for each location first to produce the internet population count and then take each over the sum of those internet population outputs)

georeadership_input_metrics

  • pageviews
  • unique devices

geoeditor_input_metrics

  • editors
  • active editors

grants_leadership_input_metrics

  • historical grants count
  • historical unique grantee count
  • historical grants dollars unweighted
  • historical grants dollars weighted
  • annual grants count
  • annual unique grantee count
  • annual grants dollars unweighted
  • annual grants dollars weighted

affiliate_data_input_metrics

  • operating affiliates (using operating_countries)
  • primarily located affiliates (using country_code)

Event Timeline

JAnstee_WMF renamed this task from Develop annual output table of global proportions from EL inputs to Develop annual output table of global proportions from Equity Landscape inputs which have been making appearances in quarterly learning sessions .May 24 2023, 4:44 PM
JAnstee_WMF assigned this task to KCVelaga_WMF.
JAnstee_WMF added a project: Product-Analytics.
JAnstee_WMF updated the task description. (Show Details)
JAnstee_WMF renamed this task from Develop annual output table of global proportions from Equity Landscape inputs which have been making appearances in quarterly learning sessions to Global proportions outputs from Equity Landscape inputs.May 24 2023, 5:23 PM
JAnstee_WMF updated the task description. (Show Details)
mpopov triaged this task as Medium priority.May 30 2023, 2:28 PM
mpopov edited projects, added: Product-Analytics (Kanban); removed: Product-Analytics.

Clarification per discussion with Jaime: this is related to the data that was compiled for the Equity Landscape Dashboard, but is not actually intended for the Dashboard. Instead, it's responding to the fact that we keep querying the same regional views for quarterly learning discussions, staff meetings, etc.

It's possible the request could be satisfied by having easily accessible queries or putting it in Superset.

KCVelaga_WMF moved this task from Doing to Needs Review on the Product-Analytics (Kanban) board.

@JAnstee_WMF I have added the initial data to this workbook. Please review.

The data doesn't have everything that is mentioned in the description, due to the following reasons:

  • grants_leadership_input_metrics
    • unique grantee count (both historical & annual) are not present in the input metrics file
  • affiliate_data_input_metrics
    • primary located affiliates metric is not present in the input metrics file
      • due to this, the operating affiliates proportion might be off (red marked on the sheet)

Also to note, I have used a mix of Spark and SQL in a notebook on JupyterLab because:

  • the interim tables for input metrics do not include regional views (sub_continent, continent etc.) and their aggregations
  • the final input metrics data is stored as a parquet file (because input metrics tend to change)
  • the input data only changes yearly

Thanks for the update, @KCVelaga_WMF Yes, as these were not our EL metric inputs these data are only available in the root files rather than our interim country level-aggregate tables for input.

  • Primary location has been based on affiliate_country (string) available in the root file (affiliate_data_input_metrics) but not in the country-level aggregate affiliate leadership input data table (affiliate_leadership_input_metrics)
  • Unique grant id (unique_grantee_id (string)) is available in the grants data root file (grants_input_metrics) rather than the aggregated input data table (grants_leadership_input_metrics)
KCVelaga_WMF claimed this task.
KCVelaga_WMF added a subscriber: KCVelaga.
KCVelaga_WMF unsubscribed.

@KCVelega_WMF I did some quick comps to those we QAed for our April presentation - seems I need to see the code in your Jupyter notebook.

@JAnstee_WMF Sure, I will share the notebook with you.

Also, a quick thought about the metrics for affiliates primary location and unique grantees, the root tables do not have the regional aggregations (sub_continents and continents). This comes at a later step in the pipeline. But @ntsako already has the code for it, it might be easier to include these metrics in final parquet file. That we can just have one way of calculating the proportions for all metrics.

mpopov added subscribers: OSefu-WMF, KCVelaga_WMF, mpopov.

Un-assigning @KCVelaga_WMF per discussion with @OSefu-WMF. It's possible we'll ask KC to either help with this or transfer knowledge of the technical backend to someone closer to Movement Insights, but if/when that happens (pending some decisions about the dashboard's future) it would need to be done in coordination with KC's other stakeholders as it will require making room for this work in the context of his commitments to Moderator Tools, Language Engineering, and Community-Tech teams.

@KCVelaga_WMF: From what you've shared with me it sounds like this work will be useful for regional learnings in the Quarterly Learning Sessions. Can you please confirm if I understood correctly? And are you aware of any other dependencies that pausing this work will block?

@mpopov Yes, as of now, the main use case for this data will be the Quarterly Learning Sessions.

nshahquinn-wmf subscribed.

KC is not planning to work on this any more, so it doesn't make sense for it to be in "watching".