Page MenuHomePhabricator
Create Task
Maniphest T120036

Vital Signs: Please make the data for enwiki and other big wikis less sad, and not just be missing for most days
Closed, ResolvedPublic
Actions

Description

See e.g. https://vital-signs.wmflabs.org/#projects=all,ruwiki,itwiki,dewiki,frwiki,enwiki,eswiki,jawiki/metrics=RollingSurvivingNewActiveEditor where the data is missing for enwiki on most days, and on most of the other wikis for at least some.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedmfornsT134790 Extract edit oriented data from MySQL for simplewiki
 ResolvedNoneT120036 Vital Signs: Please make the data for enwiki and other big wikis less sad, and not just be missing for most days
 ResolvedodimitrijevicT130256 Wikistats 2.0.
 ResolvedNoneT131779 Implement Pages Created & Count of Edits full vertical slice
 ResolvedmfornsT131782 Put data needed for edits metrics through Event Bus into HDFS
 ResolvedNoneT131786 Load edit history data into Druid
DuplicateNoneT141472 Research spike: load enwiki data into Druid to study lookup table performance
 InvalidmfornsT141479 Reportupdater calculations for Pages Created and Edit counts
ResolvedJAllemandouT131783 Examine wikistats reports, make a summary of the most granular data needed that would serve all reports
 DuplicatemfornsT134793 User history in hadoop
 ResolvedOttomataT134502 Propose evolution of Mediawiki EventBus schemas to match needed data for Analytics need
 ResolvedOttomataT137287 Update MediaWiki hooks to generate data for new event-bus schemas
 DeclinedJAllemandouT134792 Spike - Slowly Changing Dimensions on Druid
 DuplicateNoneT138268 Mediawiki changes to publish data for analytics schemas
DuplicateNoneT138269 solving issues with scalability , querying and high cardinality dimensions (maybe hosting in Druid, clickhouse or other)
 Resolved• NuriaT134791 Scale MySQL edit history reconstruction data extraction
ResolvedmfornsT141468 User history: rewrite the user history script to use the new algorithm
 ResolvedJAllemandouT141548 Scale scala algorithms using graph partitioning
 ResolvedJAllemandouT141778 Use scalable algo on enwiki
 ResolvedJAllemandouT142552 Productionize scala code for edit reconstruction
 ResolvedmfornsT142724 Page History: Add unit tests to PageHistoryDataExtractors and PageHistoryBuilder
 DuplicateMilimetricT141536 Compare early results of Wikistats 2.0 with Wikistats 1.0
 ResolvedmfornsT143321 Create clean simplewiki output from edit history reconstruction
 ResolvedmfornsT143322 Edit History: Review scala code functionality and make page and user output uniform
 ResolvedNoneT140000 Design new UI for Wikistats 2.0
 ResolvedNoneT152033 Visual Language for http://stats.wikimedia.org replacement
ResolvedashgrigasT152438 Navigation model concept development
 ResolvedashgrigasT153466 Develop site map and overview and detail page wireframes
 ResolvedMilimetricT156287 Involve the community in Wikistats 2.0 design. Stage 1.
 ResolvedMilimetricT157827 Visual prototype for community feedback for Wikistats 2.0 iteration 1.
 ResolvedMilimetricT157986 Simple json model of the site
 ResolvedMilimetricT162155 Create and monitor Round2 consultation page
 ResolvedmfornsT141774 User history: Adapt the user history reconstruction to use scaling by clustering
 ResolvedNoneT143924 Replacing standard edit metrics in dashiki with data from new edit data depot
 ResolvedNoneT146489 Calculate edit metrics from history reconmstruction in hadoop intermediate table
ResolvedmfornsT150023 Run Standard metrics on denormalized history and compare with wikistats
 ResolvedmfornsT150024 (subtask) Marcel's standard metrics
 ResolvedMilimetricT150025 (subtask) Dan's standard metrics
 ResolvedMilimetricT152034 Create 1-off tsv files that dashiki would source with standard metrics from datalake
ResolvedJAllemandouT152035 Productionize Edit History Reconstruction and Extraction
 ResolvedMilimetricT141476 Productionize edit history extraction for all wikis using Sqoop
 ResolvedJAllemandouT141473 Productionize loading of edit data into Druid (contingent on success of research spike)
 DeclinedNoneT153923 vet edit data on the data lake
ResolvedJAllemandouT178591 Feedback on hive table mediawiki_history by Erik Z
 ResolvedJAllemandouT219177 Add user_is_bot_by to MediaWiki history
 ResolvedJAllemandouT144717 Extract edit history denormalized data from intermediate data
 ResolvedJAllemandouT160074 Create oozie job for mediawiki edit history job
 ResolvedOttomataT160083 Create cron job in puppet sqooping prod and labs DBs
 ResolvedJAllemandouT160151 Productionise standard metrics from mediawiki denormalized history
 ResolvedJAllemandouT160152 Update sqoop job to add infra and version partition
 ResolvedJAllemandouT160153 Provide 2 static files to differenciate prod and labs projects to sqoop in
ResolvedJAllemandouT160154 Synchronise changes for productionisation of mediawiki history jobs
 ResolvedJAllemandouT160155 Create hive tables and queries for metrics computation out of mediawiki denormalized history
 DeclinedNoneT145091 Redact data so it can be public
ResolvedNoneT160370 Initial Launch of new Wikistats 2.0 website
 Resolved• fdansT163814 Initial FE code for Wikistats 2.0. Dashboard skeleton
ResolvedNoneT163817 Implement pageview metric in Wikistats UI
 DuplicateNoneT167672 Routing
 DuplicateNoneT167673 Address design feedback from Volker
 ResolvedMilimetricT167674 Re-read Round 2 feedback on wikistats on mediawiki and make any critical items into tasks
 ResolvedMilimetricT167675 Implement Wiki Selector Widget
 ResolvedMilimetricT167676 Implement Topic Selector Widget
 Resolved• fdansT167677 Build Dashboard on top of dynamic data
 Resolved• fdansT167678 Data Model using Crossfilter, called DimensionalData
 Resolved• fdansT167679 Interface from Graphs to DimensionalData
 Resolved• fdansT167680 Interface from Detail page to DimensionalData
 ResolvedMilimetricT167681 AQS Api works with DimensionalData
 ResolvedMilimetricT167684 Deploy new Wikistats to stats.wikimedia.org/v2
 Resolved• mmodellT168549 Separate Diffusion's rANWS from gerrit's analytics-wikistats
 ResolvedMilimetricT156384 Backend for wikistats 2.0
ResolvedJAllemandouT163356 Finalize list of metrics, breakdowns, and filters for Wikistats 2.0 backend
 ResolvedMilimetricT162817 Design document for wikistats prototype backend
ResolvedMilimetricT170882 Implement some example metrics as Druid queries
 DuplicateJAllemandouT174174 Add edits endpoint to AQS using druid as a backend
 DeclinedJAllemandouT175299 Punch hole so AQS can access druid hosts
ResolvedelukeyT176223 Create Druid public cluster such AQS can query druid public data
 ResolvedOttomataT177511 LVS for Druid
 DuplicateJAllemandouT176234 Add basic authentication abilities to restbase endpoint to be able to connect to druid authentication proxy
 Resolved• fdansT166387 Create yaml UI configuration files for Standard Metrics
 ResolvedMilimetricT166388 Create yaml UI configuration files for Wikistats metrics
 ResolvedmfornsT170457 Define, Document (and test) Desktop and Mobile browser support for wikistats 2.0
 Resolved• fdansT170458 Set up continuous integration for wikistats 2.0 UI
 ResolvedmfornsT170459 Cleanup Routing code
Resolved• fdansT163229 Review by legal department of text on wikistats site
ResolvedNoneT90759 Create Daily & Monthly pageview dump with country data and Visualize on UI
DeclinedNoneT118839 Productionize Pageview_sanitization hive code with Oozie job and refinery inclusion {hawk}
 ResolvedJAllemandouT118838 Write hive code doing pageview data anonimisation with two tables {hawk}
 Resolved• fdansT175422 Design map visualization on UI
Resolved• fdansT181520 Add "Pageviews by Country" AQS endpoint
 Resolved• fdansT181521 Add pageviews by country cassandra oozie job
 Duplicate• fdansT184748 Add ISO code to AQS data per country
Resolved• fdansT181529 Add map component to Wikistats 2
 Resolved• fdansT188928 Intervals/buckets for data arround pageviews per country in wikistats maps
 ResolvedNoneT170460 Wikistats 2.0 UI second deployment/iteration
 Resolved• fdansT170461 Addition of Unique Devices metric
Resolved• fdansT170936 Wikistats2 bugs (2/4) - Wiki selector
 Resolved• fdansT171766 Productionise line graph
 DeclinedNoneT170463 Addition of (mock) Active Editors metric
 Resolved• fdansT170933 Wikistats2 bugs (1/4) - Dashboard and general UI
 Resolved• fdansT170937 Wikistats2 bugs (3/4) - Data issues
 Resolved• fdansT170940 Wikistats2 bugs (4/4) - Detail page
 ResolvedmfornsT186120 Wikistats Beta
 ResolvedAmitjokiT186335 Wikistat Beta: expand topic explorer by default
DuplicateErik_ZachteT191555 stats.wikimedia.org home page should link to wikistats 2
 ResolvedNoneT188550 Wikistats 2.0: allow to view stats for all language versions (a.k.a. Project families)
ResolvedMilimetricT201177 Underreporting WikiSource edits?
 ResolvedJAllemandouT203258 Add project-families to AQS for additive-metrics
 ResolvedNoneT143927 Per Family Unique Devices Counts
Resolved• emaT138027 Add global last-access cookie for top domain (*.wikipedia.org)
 ResolvedJAllemandouT143928 Count project-wide unique devices (like *.wikipedia.org)
 Resolved• TbayerT169550 Final Vetting of Family Wide unique devices data
Resolved• fdansT167539 Final steps to expose project family unique devices data
Resolved• fdansT168477 Update html language for per-domain uniques
 Resolved• fdansT164201 AQS unique devices api should report offset/underestimate separately
DuplicateNoneT157981 Serve global unique device counts externally
Resolved• fdansT205665 Make wikistats UI family aware: you should be able to select a family in drop down menu and request available metrics for it
 OpenNoneT205730 Group pageview data per family in AQS so we can surface it in wikistats per-family pageview metrics
 OpenNoneT188265 Active Editors metric per project family
OpenNoneT181703 Implement digest-only mediawiki_history_reduced dataset in spark
 OpenNoneT249751 Define reduce calculations needed to compute active editors per project family
 ResolvedJAllemandouT212928 [Spike] Spark job for digests-only mediawiki-history-reduced
ResolvedNoneT188552 Wikistats 2.0.: display the mean value for Total Page Views
Resolved• fdansT188859 Wikistats 2.0: Add statistics for the geographical origin of the contributors
 ResolvedmfornsT238365 Add editors per country data to AQS API (geoeditors)
 ResolvedmfornsT244597 Create intermediate table that holds public data for geoeditors dataset so it can be used to load cassandra
 Resolved• fdansT267026 Add mutually exclusive (radio button-style) dimension values
 DuplicateNoneT205769 Wikistats 2.0: add filters on the "Top editors" view
 DuplicateNoneT205770 Wikistats 2.0: add filters on the "Top edited pages" view

Event Timeline

Jdforrester-WMF created this task.Dec 1 2015, 11:28 PM
Jdforrester-WMF raised the priority of this task from to Needs Triage.
Jdforrester-WMF updated the task description. (Show Details)
Jdforrester-WMF added a project: Analytics-Backlog.
Jdforrester-WMF subscribed.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 1 2015, 11:28 PM
Milimetric triaged this task as Medium priority.Dec 14 2015, 6:44 PM
Milimetric moved this task from Incoming to Backlog on the Analytics-Backlog board.
Milimetric set Security to None.
• ggellerman edited projects, added Analytics; removed Analytics-Backlog.Jan 12 2016, 7:41 PM
Milimetric moved this task from Incoming to Event Platform on the Analytics board.Jan 12 2016, 7:41 PM
Milimetric closed this task as Declined.Aug 29 2016, 3:49 PM
Milimetric subscribed.

We're going stop vital signs metric creation on labs and replace it with our hadoop pipeline. Labs was just never performant enough to work with the default schema.

Jdforrester-WMF added a comment.Aug 29 2016, 4:15 PM

So… This is not Declined, just being done in another way? I'm confused.

Jdforrester-WMF mentioned this in T120037: Vital Signs: Please provide an "all languages" de-duplicated stream for the Community/Content groups of metrics.Aug 29 2016, 7:08 PM
• Nuria subscribed.EditedAug 29 2016, 7:14 PM

Please see: https://phabricator.wikimedia.org/T143924 and note that it fits within this project: https://phabricator.wikimedia.org/T130256

Jdforrester-WMF reopened this task as Stalled.Aug 29 2016, 7:20 PM
Jdforrester-WMF added a subtask: T130256: Wikistats 2.0..
Milimetric moved this task from Event Platform to Backlog (Later) on the Analytics board.Sep 19 2016, 3:54 PM
• Nuria added a comment.Dec 22 2016, 12:02 AM

Preliminary data is temporarily at: https://analytics.wikimedia.org/dashboards/standard-metrics/#projects=eswiki,itwiki,enwiki,jawiki,dewiki,ruwiki,frwiki/metrics=(Beta)%20Monthly%20New%20Editors

Note that there are no gaps for big wikis

Milimetric moved this task from Backlog (Later) to Operational Excellence Future on the Analytics board.Feb 9 2017, 5:27 PM
• Nuria moved this task from Operational Excellence Future to Wikistats on the Analytics board.Mar 23 2017, 4:08 PM
• Nuria moved this task from Wikistats to Dashiki on the Analytics board.Jun 26 2017, 4:15 PM
• Nuria moved this task from Dashiki to Backlog (Later) on the Analytics board.Jul 6 2017, 4:36 PM
• fdans added a project: Data-Engineering-Wikistats.Oct 26 2017, 4:22 PM
• fdans subscribed.

The new Wikistats has gapless data for big (and small) wikis from the beginning of time.

Wikistats 2.0.png (2×3 px, 578 KB)

• Nuria added a comment.Dec 13 2017, 6:03 PM

@Jdforrester-WMF Closing this ticket, consensus among research team is not to use rolling metrics but there are many other metrics available in wikistats 2.0, please take a look for all projects: https://stats.wikimedia.org/v2/#/all-projects

And, as an example of a specific project, spanish wikipedia active editor metrics: https://stats.wikimedia.org/v2/#/es.wikipedia.org/contributing/editors

We will be anouncing the site at some point tomorrow on e-mail first and on social and others later.

Please note that not all metrics can be computed for all projects but many can be so.

• Nuria closed this task as Resolved.Dec 13 2017, 6:03 PM
odimitrijevic closed subtask T130256: Wikistats 2.0. as Resolved.Aug 11 2021, 9:08 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL