T102178: Fix RESTBase support for wikitech.wikimedia.org

@Milimetric, you sure that's a duplicate?

Oh, no, my fault, I got confused by the assumption around https://wikitech.wikimedia.org/api/rest_v1/ not working. That part is what we'd address in T119094. As for pageviews for wikitech, those should be in the api. The reason they're not is because wikitech is not included in this whitelist:

https://github.com/wikimedia/analytics-refinery-source/blob/bf6221db8623e705782cd2ab4abdbd23976c2589/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/PageviewDefinition.java#L51

So it's an easy change to add it, and I vote we should add it. Sorry about the accidental close.

Are they even being collected given that wikitech is not behind varnish?

@Krenair if wikitech is not behing varnish pageviews cannot be collected. Correct. Seems that we can close ticket?

No, we should leave it open and blocked on wikitech being set up properly. We could of course collect pageviews via some other route and stick them into kafka but that seems extreme. Are there plans on putting wikitech behind varnish?

In T153821#2965325, @Milimetric wrote:

Are there plans on putting wikitech behind varnish?

Indirectly, yes via T161859: Make Wikitech an SUL wiki. At the conclusion of T161859 wikitech will be a "normal" wiki in that it will not have OpenStackManager (T161553) or Semantic MediaWiki (T53642) installed. When we reach that point it will make good sense to move the labswiki database off of silver and serve wikitech via the general MediaWiki server pool. There is currently no official goal or timeline to complete this work, but unofficially I hope that we can have it done by the end of the 2017 calendar year.

In T153821#2962822, @Nuria wrote:

@Krenair if wikitech is not behing varnish pageviews cannot be collected. Correct. Seems that we can close ticket?

Be that as it may - we do actually have data in the webrequest table for Wikitech. Using a somewhat simplistic pageview definition, here are the 100 most viewed pages for September 2018 (without spiders) according to that data. Looks quite plausible.

Data via

SELECT CONCAT('https://wikitech.wikimedia.org',uri_path) AS desktopurl, COUNT(*) AS views
FROM wmf.webrequest 
WHERE year = 2018 AND month = 9
AND uri_host = 'wikitech.wikimedia.org' AND uri_path LIKE '/wiki/%' 
AND agent_type = 'user'
GROUP BY uri_path
ORDER BY views DESC LIMIT 100;

wikitech is not part of the projects to account for in PageviewDefinition code (https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/PageviewDefinition.java#L77). Easy to change if needed.

In T153821#4651184, @Tbayer wrote:

In T153821#2962822, @Nuria wrote:

@Krenair if wikitech is not behing varnish pageviews cannot be collected. Correct. Seems that we can close ticket?

Be that as it may - we do actually have data in the webrequest table for Wikitech. Using a somewhat simplistic pageview definition, here are the 100 most viewed pages for September 2018 (without spiders) according to that data. Looks quite plausible.

I think at some point since I wrote that, the setup was changed and wikitech went behind varnish.

In T153821#4651186, @JAllemandou wrote:

wikitech is not part of the projects to account for in PageviewDefinition code (https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/PageviewDefinition.java#L77). Easy to change if needed.

Why is this not documented on the wiki creation page? And why is it in a source file instead of a config file?

Why is this not documented on the wiki creation page?

I don't underdstand what the 'wiki creation page' is, but I think the current doc is correct about wikitech not being a pageview (https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters).

And why is it in a source file instead of a config file?

Technical debt ... It's been implemented this way originally, and never changed since.

In T153821#4651201, @JAllemandou wrote:

Why is this not documented on the wiki creation page?

I don't underdstand what the 'wiki creation page' is

https://wikitech.wikimedia.org/wiki/Add_a_wiki

Right, I get it now :)
We discussed withbthe team and our plan is to change how we detect/filter pageviews from a domain perspective. We'll update the doc as needed when we're done refactoring.
In the meantime, we're going to add wikitech.wikimedia.org to the regex, providing pageviews for wikitech site.

Adding @Harej and @srodlund as subscribers as I think they will be interested in the outcome here.

Clarifying:

• wikitech pageviews can now be computed as now wikitech wiki is behind varnish, webrequest table gets all data (indirectly) from varnish.

• The process of adding a new wiki includes documentation as to what to do for analytics since quite far back: https://wikitech.wikimedia.org/wiki/Add_a_wiki#Analytics

This is all all third party users need to do. For all purposes here wikitech is a "new wiki" so it should be included on this list (a whitelist exists cause we do not want to surface pageviews for wikis that are private)

If there is any need to modify the PageviewDefinition in any way , the analytics team will take care of that.

the two steps needed to add this to the pageview definition are:

1. What nuria mentioned, adding it to the whitelist (https://wikitech.wikimedia.org/wiki/Add_a_wiki#Analytics)
2. Adding wikitech to this regex that includes it in the pageview definition: https://phabricator.wikimedia.org/diffusion/ANRS/browse/master/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/PageviewDefinition.java$77 (I've added this to the link above for future clarity)

Change 481223 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[analytics/refinery@master] Add wikitech to whitelist

https://gerrit.wikimedia.org/r/481223

Change 481224 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[analytics/refinery/source@master] Add wikitech to pageview definition

https://gerrit.wikimedia.org/r/481224

Change 481224 merged by jenkins-bot:
[analytics/refinery/source@master] Add wikitech to pageview definition

https://gerrit.wikimedia.org/r/481224

Change 481223 merged by Joal:
[analytics/refinery@master] Add wikitech to whitelist

https://gerrit.wikimedia.org/r/481223

Thanks for the merge, @JAllemandou ! What are the next steps to complete this task?
IIUC the 2 steps that Milimetric described above have now been completed, but pageviews are not yet visible for wikitech (example1, example2). Perhaps it just needs to wait for the deployment train to do a circuit? (There's no rush, I just want to make sure it's moving forward :)

This is exactly it @Quiddity : A deploy on our side should unlock the thing.

Wikitech pageviews are now available as of yesterday, closing: https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/wikitech.wikimedia.org/all-access/user/daily/2019010700/2019010800