robots.txt (tracking)
Open, LowPublicFeature
Actions

Assigned To

None

Authored By

	Danny_B
	Jul 4 2008, 1:10 AM

Description

NOTE: According to T102500#2025332, we won't create a tag to replace this task.

Tracking bug for robots.txt settings on different wikis and related issues.

Version: unspecified
Severity: enhancement

Details

Reference: bz14720

Related Objects
Search...

Status	Subtype	Assigned	Task
Open	Feature	None	T16720 robots.txt (tracking)
Declined		None	T7582 Disallow ia_archiver on user and user_talk pages (robots.txt)
Resolved		None	T16793 Addition of en:Wikipedia:Changing_username to robots.txt
Invalid		None	T13432 robots.txt for no.wikipedia.org
Declined		None	T15439 Concealing search results that are listed in robots.txt
Resolved		None	T15844 Adding Danish language pages to robots
Resolved		None	T15890 add user talk namespace of En.Wikipedia to Robots.txt
Resolved		None	T15961 Add Wikipedia:Spam-Blacklist-Log to robots.txt for dewiki
Declined		None	T12648 Exclusion to robots.txt file for en.wikipedia.org
Invalid		None	T17878 MediaWiki:Robots.txt and $wgCapitalLinks = true on pl.wiktionary
Resolved		None	T16076 Exclude bot-generated spam reports on meta from indexing via robots.txt
Declined		None	T16393 robots.txt
Declined		None	T16419 Expansion of robots.txt
Resolved		None	T24932 Add charset to robots.txt
Resolved		None	T30229 MediaWiki:Robots.txt has no effect on pl.wikimedia.org
Resolved		None	T24040 MediaWiki:Robots.txt has no effect on Commons
Declined		Krinkle	T31162 Automatically add encoded URL lines for entries in MediaWiki:robots.txt
Declined		Mdann52	T64494 Add exception in robots.txt to allow the Internet Archiver to index action=raw
Open		None	T104251 Move wiki-specific robots.txt out of the global file to Mediawiki:Robots.txt on specific wikis
Resolved		Mdann52	T104949 Fix or remove robots.txt code for Internet Archive exclusion of user pages
Resolved		Jdlrobson	T108443 Google doesn't honor canonical URLs of zh.wiki
Resolved		Jdlrobson	T54429 Canonical URL should include language variant
Resolved		Smalyshev	T126898 query.wikidata.org should have robots.txt
Resolved		None	T7239 Add entries of VFDs to robots.txt for jawiki
Invalid		None	T47982 Blacklist all access to special pages in robots.txt
Resolved		Krenair	T105662 quarry.wmflabs.org/robots.txt gives 502
Resolved		• GWicke	T119786 rest.wikimedia.org is crawled by search engines
Duplicate		None	T16333 Userspace pages should be added to robots.txt list
Resolved		• mmodell	T110710 Expand phabricator's robots.txt blacklist
Resolved		• mmodell	T109247 Hotfix phabricator robots.txt to exclude /multimeter/
Resolved		bd808	T76915 robots.txt Last Modified header is wrong
Open		None	T90782 API landing page (action=help) should not be indexed by robots
Open		None	T87140 Regularly run GenerateSitemap.php on Farsi Wikinews to improve Google crawling
Declined		None	T95625 craft robots.txt to intelligently exclude dynamic pages with cleaner URLs
Declined		None	T76992 Allow search engines to crawl code hosted by phabricator
Resolved	Feature	None	T58533 Let MeetBot logs be indexed and archived
Open	Feature	None	T64468 Add option to have the Internet Archiver (and/or other robots) retrieve raw wikitext of all pages
Declined		None	T50856 Prevent zero.wikipedia.org from being indexed by search engines
Resolved		kerberizer	T146908 Global robots.txt contains invalid empty line
Open		None	T147170 Implement a PHPUnit test to validate robots.txt against the specs
Resolved		bd808	T127206 provide a more strict robots.txt at Tool Labs
Open		None	T226688 Block web crawlers from accessing Cloud Services
Invalid		Cyberpower678	T168684 "Page cannot be displayed due to robots.txt." from ArchiveBot link
Resolved		Kizule	T199496 Remove frwiki outdated entries in robots.txt
Open		None	T179312 robots.txt prevents indexing of Special:EntityData
Resolved		None	T6275 Robots.txt incomplete
Duplicate		None	T258299 Robots.txt broken on Toolforge.org
Open		None	T298902 Add robots.txt to wbcloud mediawiki deployment
Resolved	BUG REPORT	matmarex	T319124 Url "veaction=edit" should be marked as noindex, nofollow

Event Timeline

• bzimport raised the priority of this task from to Low.Nov 21 2014, 10:15 PM

• bzimport added projects: Wikimedia-Site-requests, Tracking-Neverending.

• bzimport set Reference to bz14720.

• bzimport added a subscriber: Unknown Object (MLST).

Danny_B created this task.Jul 4 2008, 1:10 AM

jeluf wrote:

robots.txt can now be edited on-wiki by editing Mediawiki:robots.txt => closing this bug

Reopening, since there can be random other issues with robots.txt as well as tracking bugs should remain opened.

Adding 'tracking' keyword.

rm shell keyword no current action is required

Glaisher added a subtask: T64494: Add exception in robots.txt to allow the Internet Archiver to index action=raw.Mar 10 2015, 5:42 PM

Dereckson moved this task from Backlog to Tracking on the Wikimedia-Site-requests board.Apr 10 2015, 11:20 AM

Bugreporter mentioned this in T102500: Replace tracking bug T16720 by new project tag "robots.txt".Jun 15 2015, 4:30 PM

Glaisher added a subtask: T104251: Move wiki-specific robots.txt out of the global file to Mediawiki:Robots.txt on specific wikis.Jun 30 2015, 12:44 PM

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJun 30 2015, 12:44 PM

• Tbayer added a subtask: T104949: Fix or remove robots.txt code for Internet Archive exclusion of user pages.Jul 7 2015, 6:31 AM

Krinkle closed subtask T31162: Automatically add encoded URL lines for entries in MediaWiki:robots.txt as Declined.Jul 7 2015, 3:31 PM

Liuxinyu970226 set Security to None.Aug 28 2015, 1:49 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 28 2015, 1:49 AM

Liuxinyu970226 added a subtask: T108443: Google doesn't honor canonical URLs of zh.wiki.Aug 31 2015, 7:14 AM

Mdann52 closed subtask T64494: Add exception in robots.txt to allow the Internet Archiver to index action=raw as Declined.Sep 18 2015, 11:51 AM

Glaisher subscribed.Sep 18 2015, 11:56 AM

Mdann52 closed subtask T104949: Fix or remove robots.txt code for Internet Archive exclusion of user pages as Resolved.Jan 12 2016, 6:10 PM

Restricted Application added a subscriber: JEumerus. · View Herald TranscriptJan 12 2016, 6:10 PM

Danny_B updated the task description. (Show Details)Feb 12 2016, 7:03 PM

Danny_B moved this task from Tag to Transition announced on the Tracking-Neverending board.Feb 14 2016, 2:36 AM

Danny_B added a subtask: T126898: query.wikidata.org should have robots.txt.Feb 14 2016, 4:50 PM

Danny_B added subtasks: T7239: Add entries of VFDs to robots.txt for jawiki, T47982: Blacklist all access to special pages in robots.txt, T105662: quarry.wmflabs.org/robots.txt gives 502, T119786: rest.wikimedia.org is crawled by search engines, T16333: Userspace pages should be added to robots.txt list, T110710: Expand phabricator's robots.txt blacklist, T109247: Hotfix phabricator robots.txt to exclude /multimeter/, T76915: robots.txt Last Modified header is wrong , T90782: API landing page (action=help) should not be indexed by robots, T87140: Regularly run GenerateSitemap.php on Farsi Wikinews to improve Google crawling, T95625: craft robots.txt to intelligently exclude dynamic pages with cleaner URLs, T76992: Allow search engines to crawl code hosted by phabricator, T58533: Let MeetBot logs be indexed and archived, T64468: Add option to have the Internet Archiver (and/or other robots) retrieve raw wikitext of all pages, T50856: Prevent zero.wikipedia.org from being indexed by search engines.Feb 15 2016, 9:59 AM

Smalyshev closed subtask T126898: query.wikidata.org should have robots.txt as Resolved.Feb 17 2016, 7:11 PM

Quiddity removed a parent task: T4007: [DO NOT USE] Tracking bug [superseded by #Tracking].Jul 14 2016, 3:26 AM

kerberizer added a subtask: T146908: Global robots.txt contains invalid empty line.Sep 28 2016, 3:47 PM

kerberizer added a subtask: T147170: Implement a PHPUnit test to validate robots.txt against the specs.Oct 3 2016, 3:18 AM

Urbanecm closed subtask T146908: Global robots.txt contains invalid empty line as Resolved.Oct 3 2016, 5:31 PM

Liuxinyu970226 updated the task description. (Show Details)Dec 17 2016, 2:00 PM

Liuxinyu970226 moved this task from Transition announced to Tag on the Tracking-Neverending board.

Liuxinyu970226 removed a subscriber: • wikibugs-l-list.

Liuxinyu970226 subscribed.

Krinkle closed subtask T95625: craft robots.txt to intelligently exclude dynamic pages with cleaner URLs as Declined.Apr 29 2017, 2:37 AM

Framawiki added a subtask: T127206: provide a more strict robots.txt at Tool Labs.Jun 15 2017, 7:03 PM

Liuxinyu970226 added a subtask: T168684: "Page cannot be displayed due to robots.txt." from ArchiveBot link.Jun 22 2017, 10:52 PM

Cyberpower678 closed subtask T168684: "Page cannot be displayed due to robots.txt." from ArchiveBot link as Invalid.Jun 25 2017, 5:43 PM

• IZoid added a subtask: T49578: Score should output SVG.Dec 9 2017, 6:34 PM

Ebe123 removed a subtask: T49578: Score should output SVG.Dec 9 2017, 6:37 PM

Od1n mentioned this in T199496: Remove frwiki outdated entries in robots.txt.Jul 13 2018, 3:27 AM

Od1n added a subtask: T199496: Remove frwiki outdated entries in robots.txt.

Liuxinyu970226 added a subtask: T179312: robots.txt prevents indexing of Special:EntityData.Jul 13 2018, 2:18 PM

Kizule closed subtask T199496: Remove frwiki outdated entries in robots.txt as Resolved.Jul 16 2018, 11:15 AM

Liuxinyu970226 added a subtask: T191276: InternetArchiveBot adds link to archived version which is not displayed due to robots.txt.Jul 21 2018, 4:42 AM

Liuxinyu970226 added a subtask: T154212: InternetArchiveBot adds unavailable links (robots.txt restricted?).Jul 21 2018, 4:53 AM

Cirdan removed a subtask: T191276: InternetArchiveBot adds link to archived version which is not displayed due to robots.txt.Jul 21 2018, 6:11 AM

Cirdan mentioned this in T191276: InternetArchiveBot adds link to archived version which is not displayed due to robots.txt.

Cirdan removed a subtask: T154212: InternetArchiveBot adds unavailable links (robots.txt restricted?).

Cirdan mentioned this in T154212: InternetArchiveBot adds unavailable links (robots.txt restricted?).

Liuxinyu970226 unsubscribed.Oct 8 2018, 6:33 AM

Liuxinyu970226 mentioned this in T192655: Replace all "tracking" tasks with tags/projects (if unbounded), relabel them to outcome tasks (if bounded), or kill them entirely (if pointless).Oct 13 2018, 3:52 AM

Proposing to decline this "tracking task" as per discussion in T102500. If there are use cases why someone would specifically like to get notification about (sub)tasks which are about robots.txt (or some other use case which isn't covered by [[ https://phabricator.wikimedia.org/maniphest/query/__t2oa6sdQss/#R | searching for robots.txt ]]), please share them! Thanks.

@Aklapper Umm, I little oppose rejection, because edit right to MediaWiki: namespace is nowadays limited to technical adminships only (even you play MediaWiki software on your localhost, you still should grant TA yourself before continue editing), and we can't ensure that all living MWsites are having at least one TA, thus we still need a better-running bulletin for those wikis that don't have that, unless if there's another system (on-wiki? Github? Discord? IRC? FB group? Google+? Phabricator Conpherence? Telegram?...) that in any cases much more better than this task, we should still open this for a long long time.

@Liuxinyu970226: What is "TA"? How are namespaces related to this task about robots.txt?

"TA" is for "technical admin" which I guess stands for "interface admin" (https://www.mediawiki.org/wiki/MediaWiki_1.32/interface-admin). I don't think that it has anything to do with editing robots.txt messages though, the https://en.wikipedia.org/wiki/MediaWiki:Robots.txt message can still be edited by normal admins.

Here's my alternate suggestion: To rewrite this task as an Epic (and keep opening permanently). This can be useful for non-WMF MediaWiki websites users (a famous example: translatewiki.net), as interests will quickly run register-login-debug trains for them.

Liuxinyu970226 moved this task from Tag to Should be Epic instead on the Tracking-Neverending board.Nov 23 2018, 12:34 PM

@Liuxinyu970226: Epics are not meant to be kept open permanently. It is also unclear why "this can be useful for non-WMF MediaWiki website users". I fail to understand the sentence "as interests will quickly run register-login-debug trains for them".

Liuxinyu970226 added a subtask: T212426: Update robots.txt file to disallow POST routes.Dec 23 2018, 11:16 AM

kai.nissen removed a subtask: T212426: Update robots.txt file to disallow POST routes.Jan 3 2019, 11:14 AM

Dzahn closed subtask T50856: Prevent zero.wikipedia.org from being indexed by search engines as Declined.Jan 3 2019, 11:28 PM

Liuxinyu970226 added a subtask: T6275: Robots.txt incomplete.Mar 1 2019, 1:41 PM

< bblack> https://opensource.googleblog.com/2019/07/googles-robotstxt-parser-is-now-open.html

" Google's robots.txt Parser is Now Open Source " <- This might be helpful for this ticket or subtickets of it.

jbond subscribed.Jul 2 2019, 9:41 AM

Aklapper changed the status of subtask T87140: Regularly run GenerateSitemap.php on Farsi Wikinews to improve Google crawling from Stalled to Open.May 25 2020, 3:41 PM

DanielFriesen unsubscribed.May 25 2020, 4:51 PM

Dzahn unsubscribed.May 29 2020, 8:20 AM

Nintendofan885 added a subtask: T258299: Robots.txt broken on Toolforge.org.Jul 17 2020, 9:30 PM

• Bstorm added a subtask: T226688: Block web crawlers from accessing Cloud Services.Aug 11 2020, 4:22 PM