There is a bit of traffic that comes to the wikipedia.org portal that has the HTTP referrer 'wikipedia.org'. This implies that the page is somehow redirecting traffic to itself.
@ksmith noted these lines in the source code of the portal page might be the reason and it'd be interesting to know if it really is:
<!-- Search form -->
<div class="search-container">
<form class="pure-form" id="search-form" action="//www.wikipedia.org/search-redirect.php" data-el-section="search">
<fieldset>
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | mpopov | T138411 Wikipedia Portal Dashboard: filter out requests for 'search-redirect.php' | |||
| Resolved | mpopov | T130769 Determine if wikipedia.org portal is redirecting to itself |
Regarding @ksmith's observation:
The URL www.wikipedia.org/search-redirect.php is the end-point for the search form. So when someone
makes a search on the portal page, they visit that URL. The flow looks like this:
*internet* -> www.wikipedia.org -> www.wikipedia.org/search-redirect.php -> *results page or article*
^ ^
| |
referrer: *internet* referrer: www.wikipedia.orgSo the referrer of www.wikipedia.org/search-redirect.php is www.wikipedia.org.
I don't know how we count visits to the wikipedia.org portal, but if we count users visiting www.wikipedia.org/search-redirect.php
as visitors to www.wikipedia.org, that could explain why it seems that the portal is referring traffic to itself.
I'm thinking that this now belongs on the Discovery-Analysis instead of Discovery-Portal-Sprint. Plus, I merged in https://phabricator.wikimedia.org/T130388 which was essentially a duplicate of this.
Okay, looks like a vast majority of the traffic to Wikipedia Portal (from Wikipedia Portal) is specifically to search-redirect.php. @debt I suggest creating a ticket to adjust the Portal dashboard data collection scripts to filter out requests to search-redirect.php
| date | request | referrer | requests | proportion |
| 2016-06-20 | other | other | 197775 | 0.216 |
| 2016-06-20 | other | search-redirect.php | 224 | 0.000 |
| 2016-06-20 | other | Wikipedia Portal | 138185 | 0.151 |
| 2016-06-20 | search-redirect.php | other | 10457 | 0.011 |
| 2016-06-20 | search-redirect.php | search-redirect.php | 738 | 0.001 |
| 2016-06-20 | search-redirect.php | Wikipedia Portal | 569928 | 0.621 |
Here's the Hive query I used (for future reference):
ADD JAR /home/bearloga/Code/analytics-refinery-jars/refinery-hive.jar;
CREATE TEMPORARY FUNCTION classify_referrer AS 'org.wikimedia.analytics.refinery.hive.SmartReferrerClassifierUDF';
USE wmf;
SELECT request, referrer, COUNT(1) AS requests
FROM (
SELECT
CASE WHEN referer RLIKE('^(https?://www\.)?wikipedia\.org/+search-redirect\.php\??.*') THEN 'search-redirect.php'
WHEN referer RLIKE('^(https?://(www\.)?)?wikipedia\.org.*$') THEN 'Wikipedia Portal'
ELSE 'other' END AS referrer,
CASE WHEN uri_path = '/search-redirect.php' THEN 'search-redirect.php'
ELSE 'other' END AS request
FROM webrequest
WHERE year = 2016 AND month = 06 AND day = 20
AND webrequest_source = 'text'
AND content_type RLIKE('^text/html')
AND uri_host RLIKE('^(www\.)?wikipedia.org/*$')
AND classify_referrer(referer) IN ('internal', 'external', 'unknown')
AND NOT (referer RLIKE('^http://localhost'))
) AS refined_webrequests
GROUP BY request, referrer;