Background

When we analyze referrals in pageview data, we tend to omit referer_class=internal on the understanding that any given reading session must started from something that is not internally referred.

That something may be a different website (e.g. someone's blog, or a search engine like Google), a native app outside the browser (especially mobile apps like iMessage, Gmail, WhatsApp, Discord, etc), or a browser action (such as your browser default homepage, typing in the address bar, or revisiting something from your bookmarks or browser history).

But, if we see a pageview with an internal referrer, then it is generally safe to assume this is the second navigation in their browser session on a Wikimedia domain, and whatever the first one was, had either an external referer or no referrer, but definitely not the internal referrer.

Referrer aggregation was introduced in 2015 (change 189980) using a simple list of WMF domain wildcards, (e.g. *.wikipedia.org and *.wikimedia.org).

Problem

As part of an on-going investigation, I found myself assuming that internally-referred pageviews are the continuation of a reading session that started from an earlier "external" or "none" referred pageview that started the session. The https://www.wikipedia.org portal currently breaks this assumption.

If you open the Wikipedia portal first (through a Google search, or a bookmark, or however) and then browse Wikipedia from there, your first counted pageview has an internal referrer.

Now, the webrequest for the the Wikipedia portal will have had a referrer classified as "none", "external (search engine)" or some other non-internal value. But, this is not a pageview.

The problem also applies to our other www-portals: https://www.wikimedia.org, https://www.wikivoyage.org, https://www.wiktionary.org, etc.

The problem also applies to all the non-wiki sites and microsites we host under wikimedia.org such as blogs (https://diff.wikimedia.org, https://wikimediafoundation.org/news), and even Phabricator tasks and Gerrit code review. Though these are negligible compared to our www-portals.

Impact

Using 1-7 April 2026 as example:

• 3.1 billion pageviews (Source: stats.wikimedia.org, Turnilo pageviews_daily)
  • 1.3 billion with a "external search engines" referrer
  • 1.1 billion with an "internal" referrer
  • 700 million with a "none" referrer
• 536 million visits to https://www.wikipedia.org
  • 10 million pageviews referred by www.wikipedia.org
• 1 million visits to https://www.wiktionary.org
  • 400K pageviews referred by www.wiktionary.org
• 67K visits to https://www.wikivoyage.org
  • 24K pageviews referred by www.wikivoyage.org

SELECT uri_host, access_method, COUNT(*) AS _count FROM wmf.webrequest WHERE year=2026 AND month=4 AND day IN (1,2,3,4,5,6,7) AND agent_type='user' AND http_status in (200,304) AND uri_host LIKE 'www.wik%' AND uri_path='/' GROUP BY uri_host, access_method ORDER BY _count DESC LIMIT 30;
uri_host                access_method   _count
www.wikipedia.org       desktop         460072100
www.wikipedia.org       mobile web      76694650
www.wikimedia.org       desktop         2028466
www.wikimedia.org       mobile web      2003517
www.wiktionary.org      desktop         724864
www.wiktionary.org      mobile web      323684
www.wikivoyage.org      desktop         47281
www.wikivoyage.org      mobile web      19666
www.wikiquote.org       desktop         29334
www.wikiquote.org       mobile web      25427
www.wikibooks.org       desktop         25234
www.wikibooks.org       mobile web      18470
www.wikiversity.org     desktop         17622
www.wikiversity.org     mobile web      12979
www.wikinews.org        desktop         14478
www.wikinews.org        mobile web      9585
…

SELECT referer, referer_data, COUNT(*) FROM wmf.webrequest WHERE year=2026 AND month=4 AND day IN (1,2,3,4,5,6,7) AND agent_type='user' AND is_pageview AND referer IN ('https://www.wikipedia.org/','https://www.wiktionary.org/','https://www.wikimedia.org/','https://www.wikibooks.org/','https://www.wikivoyage.org/') GROUP BY referer, referer_data;
referer referer_data    count(1)
https://www.wikivoyage.org/     {"referer_class":"internal","referer_name":"none"}      23595
https://www.wikipedia.org/      {"referer_class":"internal","referer_name":"none"}      9562250
https://www.wiktionary.org/     {"referer_class":"internal","referer_name":"none"}      392915
https://www.wikibooks.org/      {"referer_class":"internal","referer_name":"none"}      17718
https://www.wikimedia.org/      {"referer_class":"internal","referer_name":"none"}      246682
Time taken: 597.09 seconds, Fetched 5 row(s)

Proposal

We already have PageviewDefinition.java#isPageviewHostname, which is a narrower and wiki-specific. We already use this in our pipeline to decide whether to include a webrequest in the pageview dataset.

I propose we:

1. Update the referer_class=internal assignment to use PageviewDefinition.java#isPageviewHostname, thus limiting it to our wiki domains.
2. Add a "Wikimedia" entry to our shortlist of aggregated and named external referrers ("Google", "Youtube", etc), which can keep using the simple wildcard approach (*.wikipedia.org, *.wikimedia.org, etc). This matches the way we treat other organizations with a large number of domain names. This would translate to referer_class=external (media sites) with referer_name=Wikimedia.