Page MenuHomePhabricator
Create Task
Maniphest T187414

Wikistats 2.0: "aa.wikipedia.org" exists and has data available, but marked "Invalid"
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

But it seems that on https://stats.wikimedia.org/v2/ one cannot look for aa.wikipedia.org through the UI. Typing it regardless seems to force use of the first suggestion, viewing data from the unrelated kaa.wikipedia.org domain.

I tried working around it by manually changing the url and reloading does work at https://stats.wikimedia.org/v2/#/aa.wikipedia.org and produces the correct data from the backends, but the interface marks the input field with "INVALID".

There are a few other (closed) wikis that exist, have page view data available, but are not suggested in the interface, including https://za.wiktionary.org (zawiktionary).

On the other hand, there some closed projects that are listed in Wikistats 2.0, but actually do not have pageview data at the moment. Such as https://stats.wikimedia.org/v2/#/usability.wikimedia.org which is suggested as valid target, but results in "Something went wrong" displayed instead of graphs. (For this wiki, it seems RESTBase/metrics has no data, not sure why.)

Details

SubjectRepoBranchLines +/-
Add strategy, usability and advisory sites to pageview definitionanalytics/refinery/sourcemaster+9 -4
Add usability.wikimedia to pageview whitelistanalytics/refinerymaster+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Krinkle created this task.Feb 15 2018, 1:15 AM
Restricted Application added a project: Analytics. · View Herald TranscriptFeb 15 2018, 1:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle updated the task description. (Show Details)Feb 15 2018, 1:18 AM
Krinkle updated the task description. (Show Details)
• fdans subscribed.EditedFeb 22 2018, 5:46 PM

This wiki needs to be added to the sqoop list for it to appear as valid.

• fdans edited projects, added Analytics-Kanban; removed Analytics.Feb 22 2018, 5:47 PM
• Nuria edited projects, added Analytics; removed Analytics-Kanban.Mar 22 2018, 4:19 PM
Krinkle added a comment.Mar 22 2018, 7:03 PM

@fdans Where is this list maintained, and what is the process for updating it? Given this is not a new wiki, it suggests that this list is intentionally a subset of the actual wikis we have. If so, what is the reason for that?

• fdans triaged this task as Medium priority.Mar 26 2018, 4:40 PM
• fdans moved this task from Incoming to Smart Tools for Better Data on the Analytics board.
• Nuria subscribed.Jul 16 2018, 7:02 PM

@Krinkle sqooping is different per wiki size, thus it requires a whilelist to manage it. See similar addition: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/445445/

• Nuria assigned this task to • fdans.Jul 16 2018, 7:03 PM
• Nuria raised the priority of this task from Medium to High.
• Nuria added a project: Analytics-Kanban.
• Nuria moved this task from Smart Tools for Better Data to Wikistats Beta on the Analytics board.
Milimetric moved this task from Next Up to In Progress on the Analytics-Kanban board.Jul 31 2018, 3:12 PM
• Nuria added a comment.EditedAug 28 2018, 6:29 PM

@Krinkle: usability.wikimedia.org does not have pageviews cause i suspect we have a bug on our end and we are not parsing this domain, if we were we should have received an alert as we do for every domain for which we see pageviews but not process them. This is the list for wikis for which we process pageviews (we need a whitelist as we do not want to report data for wikis that are private): https://github.com/wikimedia/analytics-refinery/blob/master/static_data/pageview/whitelist/whitelist.tsv

gerritbot subscribed.Aug 28 2018, 6:59 PM

Change 455894 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/refinery@master] Add usability.wikimedia to pageview whitelist

https://gerrit.wikimedia.org/r/455894

gerritbot added a project: Patch-For-Review.Aug 28 2018, 6:59 PM
gerritbot added a comment.Aug 28 2018, 7:01 PM

Change 455894 merged by Nuria:
[analytics/refinery@master] Add usability.wikimedia to pageview whitelist

https://gerrit.wikimedia.org/r/455894

• fdans added a comment.Aug 28 2018, 8:29 PM

@Nuria I've been checking sites listed under "special" in the sitematrix that aren't private. As far as I can tell, the sites that are slipping through the cracks are:

  • usability.wikimedia.org
  • strategy.wikimedia.org
  • wikimania.wikimedia.org

Plus all the yearly wikimania sites. Do we want to add these? In which case they all follow the same url structure, so it would be easy to integrate them in the regex.

• fdans added a comment.Aug 28 2018, 8:34 PM

In the case of aa.wikimedia.org, we should probably change the "INVALID" label to "CLOSED". Will open a separate task for this.

• Nuria added a comment.Aug 28 2018, 8:38 PM

@fdans : we should probably open tickets to refactor all regexes but we can do that at a later time. Since we have not had any requests ever for the wikimania sites , let's add the other two. Thanks for going through the list.

gerritbot added a comment.Aug 28 2018, 9:14 PM

Change 456022 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/refinery/source@master] Fix strategy and usability sites not being counted as pageviews

https://gerrit.wikimedia.org/r/456022

• fdans added a comment.Aug 31 2018, 1:33 PM

The issue with aa.wikipedia.org is fixed in https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/456612/

gerritbot added a comment.Sep 4 2018, 12:25 PM

Change 456022 merged by Fdans:
[analytics/refinery/source@master] Add strategy, usability and advisory sites to pageview definition

https://gerrit.wikimedia.org/r/456022

• fdans moved this task from In Progress to Ready to Deploy on the Analytics-Kanban board.Sep 4 2018, 12:39 PM
• fdans moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Sep 5 2018, 11:50 AM
• Nuria moved this task from Done to Ready to Deploy on the Analytics-Kanban board.Sep 5 2018, 4:02 PM
• Nuria moved this task from Ready to Deploy to Done on the Analytics-Kanban board.
• Nuria added a comment.Sep 14 2018, 1:10 PM

usability data is still not showing on wikistats

• Nuria added a comment.Sep 14 2018, 2:12 PM

I will live this ticket open until next month, we have started computing pageviews for usability.wikimedia and you can see there are pageviews available for the daily range here: https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/strategy.wikimedia.org/all-access/user/daily/2018090100/2018091400

However, there is still not 1 month full of pageviews and, until that happens the wikistats UI (which agreggates monthly data for both pageviews and unique devices) will not work

mforns set the point value for this task to 5.Oct 8 2018, 4:03 PM
• Nuria added a comment.Oct 19 2018, 2:29 AM

za.wiktionary.org , usability.wikimedia.org and aa.wikipedia.org are now selectable options on the drop-down menu, there is data (for pageviews) for the last month: https://stats.wikimedia.org/v2/#/usability.wikimedia.org/reading/total-page-views/normal|bar|3-Month|~total

https://stats.wikimedia.org/v2/#/za.wiktionary.org

• Nuria closed this task as Resolved.Oct 19 2018, 2:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits