Page MenuHomePhabricator

Some special pages on the mobile version are indexed by search engines even with noindex rule (because robots can't access the pages)
Open, Needs TriagePublic

Description

Look for "jmtc63" on Google.

Event Timeline

Aklapper renamed this task from Some special pages on the mobile version are indexed even with noindex rule because robots can't access the pages to Some special pages on the mobile version are indexed by search engines even with noindex rule (because robots can't access the pages).Feb 3 2021, 12:47 PM

Hi @Lofhi, thanks for taking the time to report this! For future reference, please include what you see where and what you expect instead, so people looking at this ticket a while later don't have to guess what was the situation at the time of reporting... I guess this is about a mobile page on French Wikipedia?

Screenshot from 2021-02-03 13-45-00.png (305×647 px, 34 KB)

Where is that "noindex rule" defined (in case you know)?

Ah, sorry, I'll think about it next time. Special pages should not be accessed by robots (robots.txt disallow it), and the expected result is that these pages are not referenced (<meta name="robots" content="noindex,nofollow">). So I think the problem is this: the pages can't be accessed and Google can't take the noindex into account. For some reason, these pages have been indexed before.

I think this may be primarily Google's problem, but I'll also note that this doesn't seem to be exclusive to the mobile site; see this enwiki VPT section for another instance of a page with NOINDEX and in robots.txt appearing in Google search results.