Page MenuHomePhabricator
Create Task
Maniphest T220456

Many small wikis missing from mediawiki_history dataset
Closed, ResolvedPublic
Actions

Description

The Wikimedia clusters contains about 940 wikis (all.dblist contains 934, and the enwiki sites table contains 940).

However, mediawiki_history contains data on only 752 wikis. The missing ones (compared to the sites table) are listed in P8372.

Various types of wikis are excluded, from private coordination wikis (e.g. arbcom_enwiki), to closed content wikis (e.g. akwikibooks), to small, open content wikis (e.g. roa_rupwiki, hiwikivoyage).

All of these wikis should be included; the biggest problem are the missing open content wikis, but even for closed or private wikis we should at least have the data available.

Details

SubjectRepoBranchLines +/-
Add 122 wikis to prod sqoop listanalytics/refinerymaster+122 -0
Add 122 wikis that haven't been sqooped so faranalytics/refinerymaster+122 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

nshahquinn-wmf created this task.Apr 8 2019, 9:43 PM
nshahquinn-wmf mentioned this in T218819: Investigate discrepancies in editor metrics between Data Lake and MediaWiki replica pipelines .
nshahquinn-wmf added a parent task: T204953: Data quality issues in the mediawiki_history data stream.
nshahquinn-wmf moved this task from Triage to Tracking on the Product-Analytics board.Apr 9 2019, 2:05 AM
• fdans triaged this task as Low priority.Apr 11 2019, 4:36 PM
• fdans moved this task from Incoming to Analytics Query Service on the Analytics board.
• fdans subscribed.Apr 11 2019, 4:38 PM

A lot of the wikis there are either private or closed. The rest can be added via a patch to the sqoop load list

https://github.com/wikimedia/analytics-refinery/blob/master/static_data/mediawiki/grouped_wikis/labs_grouped_wikis.csv

Generally whenever we get pageviews for a new site we add the site to both the pageview whitelist and the sqoop list, but this hasn't always been consistent.

• Nuria subscribed.Apr 23 2019, 4:32 PM

@Neil_P._Quinn_WMF the private wikis are not included on the labs replicas and that is intentional, if you notice we also do not report pageview data for those wikis either. This is at the request of users of those wikis and seems like the right policy as there are wikis of delicate nature there, these wikis will continue not to be included.

Any small open content wiki should be included and we should work to make sure all are. I am assigning this ticket to @fdans so the small wikis missing can be added. The best measure here would be to source our labs_grouped_wikis.csv is at least as current as the pageview list which is always updated upon the creation of a new wiki.

• Nuria assigned this task to • fdans.Apr 23 2019, 4:33 PM
• Nuria raised the priority of this task from Low to Medium.
• Nuria added a project: cloud-services-team (Kanban).
• Nuria edited projects, added Analytics-Kanban; removed cloud-services-team (Kanban).
nshahquinn-wmf added a comment.Apr 23 2019, 5:38 PM
In T220456#5131546, @Nuria wrote:

@Neil_P._Quinn_WMF the private wikis are not included on the labs replicas and that is intentional, if you notice we also do not report pageview data for those wikis either. This is at the request of users of those wikis and seems like the right policy as there are wikis of delicate nature there, these wikis will continue not to be included.

Yeah, on reflection I agree with that. So I'm fine with just making sure all the public wikis are included (closed or not). There are more than a hundred of them, although the overall effect isn't huge since they're pretty much all small.

The best measure here would be to source our labs_grouped_wikis.csv is at least as current as the pageview list which is always updated upon the creation of a new wiki.

Sorry, what do you mean by source?

JAllemandou added a parent task: T221824: Mediawiki History Release - 2019-04 snapshot.Apr 25 2019, 8:08 AM
• fdans moved this task from Next Up to In Progress on the Analytics-Kanban board.Apr 30 2019, 3:50 PM
gerritbot added a comment.Apr 30 2019, 4:00 PM

Change 507355 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/refinery@master] Add 137 wikis that haven't been sqooped so far

https://gerrit.wikimedia.org/r/507355

gerritbot added a project: Patch-For-Review.Apr 30 2019, 4:01 PM
gerritbot added a comment.May 6 2019, 9:09 PM

Change 507355 merged by Nuria:
[analytics/refinery@master] Add 122 wikis that haven't been sqooped so far

https://gerrit.wikimedia.org/r/507355

• Nuria added a comment.May 7 2019, 4:47 PM

ping @fdans let's make sure the prod list and lab list match as every snapshot scoops from both

gerritbot added a comment.May 13 2019, 8:39 PM

Change 509938 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/refinery@master] Add 122 wikis to prod sqoop list

https://gerrit.wikimedia.org/r/509938

gerritbot added a comment.May 13 2019, 8:50 PM

Change 509938 merged by Fdans:
[analytics/refinery@master] Add 122 wikis to prod sqoop list

https://gerrit.wikimedia.org/r/509938

JAllemandou removed a parent task: T221824: Mediawiki History Release - 2019-04 snapshot.May 16 2019, 7:46 AM
JAllemandou added a parent task: T221825: Mediawiki-history release - Snapshot 2019-06.
• fdans moved this task from In Progress to Ready to Deploy on the Analytics-Kanban board.May 17 2019, 1:28 PM
mforns moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Jun 6 2019, 2:30 PM
• Nuria added a comment.Jul 1 2019, 10:51 PM

closing as after scooping looks like the only wiki failed was hiwikisource so rest existed.

• Nuria closed this task as Resolved.Jul 1 2019, 10:52 PM
nshahquinn-wmf mentioned this in T294653: Update the wiki comparison tool.Jan 19 2022, 4:22 PM
nshahquinn-wmf mentioned this in T299548: 22 small wikis missing from the mediawiki_history dataset.Jan 19 2022, 5:23 PM
nshahquinn-wmf mentioned this in T349743: NEW BUG REPORT 12 new wikis missing from the mediawiki_history dataset.Oct 25 2023, 5:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL