For context plwiki uses FlaggedRevs.

Basically T147544: Unreviewed new articles on English Wikipedia should be marked as noindex but for wikis using FlaggedRevs instead of PageTriage? Noting that there's T58526: Google uses unsighted version of the page for existing articles, it would indeed be preferable if new articles are marked with NOINDEX if they haven't been reviewed yet.

Adding SEO tag since noindexing stuff would affect SEO.

We could gate this behind a feature flag ($wgFlaggedRevsNoIndexUnreviewedNewPages) and then turn it on just for plwiki, then expand rollout after that if there is interest.

We probably wouldn't want to noindex every unreviewed page. Just unreviewed pages that have never been reviewed before (i.e. new pages). Turning on and off noindex on previously reviewed pages wouldn't be good for SEO. The algorithm for this would be to check every revision of the page and make sure they are all unreviewed.

PageTriage does noindexing by using the onArticleViewFooter Hook and setting this code: $article->getContext()->getOutput()->setRobotPolicy( 'noindex,nofollow' );

onArticleViewFooter runs on every page, so the code would need to be performant.

We would only want this feature for when the wiki is configured to $wgFlaggedRevsOverride = true; $wgFlaggedRevsProtection = false;

I was asking Gemini in Google AI Mode. He give a reference to this page on EN Wiki:

https://en.wikipedia.org/wiki/Wikipedia:Controlling_search_engine_indexing

It states, "The default behavior is that articles older than 90 days are indexed". Is this how it really works?

I'm not sure how to find the last added page to EN Wiki.

I just checked the latest edits on PL Wiki, and it looks like today there was created a new page:

https://pl.wikipedia.org/w/index.php?title=Rezydencja_mi%C4%99tne&action=edit&redlink=1

https://www.google.com/search?hl=pl&q=Rezydencja%20mi%C4%99tne

It's already in Google. It was created at 22:49 and deleted a few minutes afterward.

90 days is true for enwiki (provided by the PageTriage extension), but not for other wikis. Reading docs on mediawiki.org would probably get you a doc page that isn't so enwiki-specific.

This is also of interest for dewiki.

Hi all, I would strongly recommend first having a local community conversation about the tradeoffs of implementing any search engine indexing restrictions on new content on your local wikis, and thinking through your page review processes. The important negative impacts to consider are:

• A blanket no-index policy will mean that legitimate new articles on important breaking news topics will also be in a no-index state until someone reviews them or some time passes, leading to the possibility of Wikipedia content on these topics not being discoverable in search at exactly the moment when everyone is searching online for reliable information on the topic. (And the unfortunate corollary of that is that lower quality sources – e.g., social media, certain encyclopedias rhyming with "rock" – will fill the void if there's not a Wikipedia result to display.)
• On English Wikipedia, there have been recent issues with Google not indexing some new pages at all, even long after no-index tags were removed (https://phabricator.wikimedia.org/T423068). My sense is that the explosion of AI content on the web has caught in the gears of and slowed down Google's crawling infrastructure (they've been pushing many spam prevention updates lately, e.g.: https://status.search.google.com/incidents/a7Aainy6E9rZsmfq82xt), meaning that Googlebot may be less likely to revisit a page that has a no-index tag on it and index it properly in a timely fashion now than just a few years ago. (We've been working to try to resolve the issue by manually submitting a link to the sitemap of English Wikipedia and saw some, but not 100%, improvement, so this is still very much a problem and may not be one we can resolve through technical changes on our end.)

On English Wikipedia, the no-index policy was created at a time when the expectation was that new pages would generally be reviewed within about a week (2012, those were the days! https://en.wikipedia.org/wiki/Wikipedia:Requests_for_comment/NOINDEX) at the latest, and back then Wikipedia was reliably at the top of search results, so the community decided the possible downsides were worth it. But today it can take months for new enwiki pages to clear the review queue, and AI in search results is already making Wikipedia content less visible. Delays in indexing and downstream indexing issues caused by no-index tags may further reduce the chances that people searching for information in your language will see results from your wiki. These aren't trivial downsides, and it's worth considering whether the number and severity of spam incidents on your wiki outweighs them.

I started a discussion before creating a ticket. There was not a lot of interest. This is a very technical problem; we don't have a lot of technical people that could say anything.

https://pl.wikipedia.org/w/index.php?oldid=79868325#Dodawanie_nowych_stron_przez_IPki

Translation:

https://translate.google.com/translate?hl=en&sl=pl&tl=en&u=https://pl.wikipedia.org/w/index.php?oldid=79868325%23Dodawanie_nowych_stron_przez_IPki

I can start another discussion where more people can say what they think. Something less technical so it's easier to digest.

As for the problem of not indexing important topics, I don't think this is an issue. If something is not verified, IMHO, it's potential spam, and should not be indexed at all, even if this is something very important.

If there is a new topic that is very important, it will most likely be reviewed very quickly. But creating a hoax on Wikipedia or fake news and being indexed instantly, even when deleted in minutes, is unacceptable.

But I can start a discussion on Discord to get a chance for more people to say what they think.

Just had this idea. What about not adding noindex? But instead, show some kind of placeholder instead of the real page.

The real problem that we have is that the new article is visible before it's reviewed, and the root cause is that this article doesn't have a history entry that is verified, so there is nothing to show for the unverified page. We can add a fake page that will be shown on the page that has never been reviewed.

If the page will be empty, Google will be able to index it, but it will never rank it high. Just the title and placeholder will probably not be enough. When the page is reviewed, Google could index it the same as when any page changes.

On Git, I usually add a blank, empty commit with an empty message as the first commit. MediaWiki can do the same. The entry can be NULL, not an empty string, so it can be distinguished from an empty page when something deletes everything and saves. The NULL page can be auto-verified.

The NULL page can show some graphics like the Wikipedia logo and maybe some information for the new person about the status of this page.

The text could be controlled by the wiki instance as some page that can be controlled by an admin.

On MediaWiki itself, it can be the MediaWiki logo and message this page is not yet reviewed.

The only problem that I see is that the page can be recognized as a duplicate. I'm not sure how Google handles duplicated content, how fast the page that was marked as duplicate will rank in Google when it gets real content.

We could create an experiment to see.

In T420414#11982323, @Jcubic wrote:

Just had this idea. What about not adding noindex? But instead, show some kind of placeholder instead of the real page.

but then anonymous users also see that same placeholder ?

Yes, it would work the same as with the current under-review changes. The placeholder will be like a first edit in history.

On Wiki, the placeholder can explain that the article needs review, and maybe encourage people to create an account. It will be up to the wiki what to put there.

The empty page could just insert a template:

{{empty}}

and on Wiki we would just edit Template:Empty (on PL wiki we could use {{pusty}} that redactors would be able to edit) and the {{empty}} string will be changed by the admin.

It also could be useful on En Wiki, where new pages have noindex; the new pages can always have this state no matter if the Wiki has a review process or not. This will fix the problem with indexing new pages (just an idea; not sure if it makes sense).