Impact
New articles on English Wikipedia often take weeks or months to surface in Google Search. DuckDuckGo and other search engines seem unaffected. Most (all?) other Wikipedias and non-Wikipedias also seem unaffected.
These new articles can be relevant to breaking news, and present an important opportunity to share knowledge and draw a reader to Wikipedia. New articles should be indexed and surfaced by Google Search as quickly as possible.
Google often does index new articles within minutes of their creation. Likewise, edits to existing articles often reflect in excerpts on their SERP within minutes. However, some articles consistently remain unindexed for many months which is an unacceptably long period of time.
Problem
On 10 Feb 2026, it was reported in WP:VPT that an article created in July 2025 was still not returned in Google search results more than six months later, yet is indexed just fine in DuckDuckGo. The article in question is Nova Scotia Guard, created 1 July 2025 and patrolled 6 September 2025 (log).
On 25 Feb, @tstarling confirmed that the patrol action updates page_touched, which updates the last-modified tag in the sitemap, signalling that the page is ready to be recrawled and indexed.
My current theory is is that it is caused by a combination of these factors:
- The page is created by a user who is not autopatrolled.
- Google first crawls the page creation within pages of its creation (i.e. via RCFeed), which is generally a good thing. It sees the page in the noindex and honours that, which is good per the spam issue described below.
- Google retains this information and thus ignores future edit events from RCFeed, and approximately never recrawls it passively to see if the page contents (or robots tag) changed.
- Google does not currently use a sitemap for en.wikipedia.org.
Background: How stuff works
New articles sometimes require review from a patroller before they are indexable by search engines. This works via Extension:PageTriage ($wgPageTriageNoIndexUnreviewedNewArticles = true; $wgPageTriageMaxAge = 90;) and is enabled on English Wikipedia. The extension adds a "noindex" meta tag to the HTML head when we render web pages for articles that are unreviewed and less than 90 days old. Articles created by autpatrolled are considered reviewed from the get to.
This means articles not created by "autopatrolled" editors are initially kept outside external search engines, until they are reviewed or survive 90 days without deletion. This is a good thing as it protects our domain reputation from spam and vandalism in the most valuable SEO position: the page title; and discourages such actions at scale in the first place.
The sitemaps we have are dynamically served on-demand and include all indexable pages with their URL and last-modified timestamp. Sitemaps are cached for upto one hour. When a page is edited or patrolled, this sets the page's last-modified timestamp (aka page_touched).
For example:
- 31 March 2026 00:23: A new user creates Sports Riot (Newspaper).
- 31 March 2026 01:30: One hour later, it has not yet been patrolled. And indeed, when we view the page and check the source of the HTML web page, there is a "noindex" meta tag. It is also omitted from the relevant sitemap chunk.
Another example:
- 31 March 2026 00:16: An established user in the autopatrolled user group creates https://en.wikipedia.org/wiki/Doroth%C3%A9e_Aim%C3%A9e_Malenzapa Dorothée Aimée Malenzapa
- 31 March 2026 01:30: One hour later, when we view the page, there is not a "noindex" instruction in the page's HTML source. And unlike the other page, it is included and discoverable in that same sitemap chunk.
Note that just because an inexperienced editor initially created a page, says nothing about its current content or reliability. Within a few minutes or hours, experienced editors tend to write, fix, extend the content as they see fit.
We know that Google subscribes to our RCFeed and may decide (based on heuristics and priorities) for a subset to crawl a page within a minutes of a page being created or edited. We also know that Google crawls the sitemap once a day to discover new or changed pages as a reliable fallback for stuff not otherwise prioritized. The feeds, pipelines, caching, crawling is all there. This usually works correcty and efficiently.
There are 14 million pages on en.wikipedia.org currently indexed by Google according to its Search Console. It has also found over 80 million links that it decided not to index, such as:
- Random gargage pointing to a 404 Not Found, e.g. /w/does_not_exist.html
- non-canonical URL, e.g. /wiki/Foo?something=bogus123
- <meta> robots "noindex", including:
- old revisions, /w/index.php?oldid=123
- non-view action, /w/index.php?title=Einstein&action=history, /w/index.php?title=Einstein&action=edit
- local search, /w/index.php?title=Special:Search&s=foo+123
- new unpatrolled article, /wiki/Something_New
While Google does re-crawl a small trickle of these (perhaps based on inbound links or sitemaps?), generally speaking a search engine does not (and should not) proritize re-crawl the infinitely many URLs it has ever found in the wild. Doing so would take away crawl quota it could spend on more relevant and higher value pages instead. For most of the web, this rarely changes, and it is even less likely for it to be relevant/urgent.
However, there are cases outside Wikipedia as well. For example, I can see a company website creating new product pages, blog posts, or press releases as noindex while the page is being drafted or publicly tested before wider announcement. Given that most websites don't have an equivalent to RCFeed, these pages are not likely to be linked or discovered from anywhere until after the "noindex" tag is removed. But, it's still possible that such page may be leaked on Reddit, social media, where Google stumbles upon it. It would be annoying if such page then remains un-indexed for weeks or months despite high relevance to current events.
Expectation
We hope the signals should suffice to prioritize re-crawling in search engines:
- Included in the sitemap. This strongly implies it can be indexed, because sites generally do not include noindexed pages in an indexing sitemap.
- Not previously seen in the sitemap. This rules out strange cases of a site listing a noindexed page in their sitemap for a no reason (MediaWIki doesn't do this, but I'm sure search engines encounter bad sitemaps).
- The sitemap entry has a last-mod timestamp after the last crawl attempt.
Beyond that:
- The last crawl included a <link rel=canonical> pointing to self, reflecting it isn't wild bogus.
- First-party inbound links, reflecting it isn't wild bogus.
- A past crawl was initiated by RCFeed, reflecting it wasn't wild bogus.
In terms of RCFeed, it seems fair to ignore a page there once seen as noindex until something else flips it. There are many namespaces and page patterns that are essentially permanely no-indexed and so would be wasteful (e.g. user pages, talk pages, project pages, various meta discussions and archives etc).
See also
- "Indexed article omitted from Google" on WP:VPT, Feb 2026
- Google Search Partnerships <> Wikimedia, public meeting notes, Google Docs.
