There are 4 parts to this
example of old pageview definition in Vital Signs
https://metrics.wmflabs.org/static/public/dash/#projects=enwiki/metrics=DailyPageviews (webstatscollector)
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Duplicate | None | T86141 Vital Signs user clicks "Add Metric" then Daily Pageviews (New Definition) | |||
| Duplicate | None | T88125 Analytics Engineer has an oozie job to aggreate page views by time | |||
| Duplicate | None | T88127 Analytics Engineer has a job to aggregate pageviews by project | |||
| Duplicate | None | T88128 Vital Signs user has access to the new page view data |
There are 4 different steps:
#1 Pageview data goes into refined tables (done)
#2 oozie job to compute pageview agreggtaes based on time
#3 python aggregation per project (800 files will be output from this step)
#4 files get checked into a depot, dashiki will sourcethese files
Some oozie info: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Oozie
Code for other oozie jobs: https://github.com/wikimedia/analytics-refinery/tree/master/oozie
Agreggated counts for (older) webstats collector pageviews: https://github.com/wikimedia/analytics-aggregator-data
More docs: https://wikitech.wikimedia.org/wiki/Analytics/Pagecounts-all-sites
Code for step #3 (python)
https://github.com/wikimedia/analytics-aggregator