Maniphest T232115

Create a robots.txt
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Samwalton9-WMF
	Sep 5 2019, 3:45 PM

Tags

Referenced Files

None

Subscribers

Description

List Item

We don't have one and Google keeps crawling things it shouldn't.

Related Objects

Mentioned In: T237495: Identify the paths we'd want or not want robots to crawl

Event Timeline

Samwalton9-WMF created this task.Sep 5 2019, 3:45 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 5 2019, 3:45 PM

MJL subscribed.Sep 7 2019, 7:04 PM

Samwalton9-WMF triaged this task as Medium priority.Sep 19 2019, 8:52 AM

Nikkimaria added a parent task: T240148: Infosec.Dec 8 2019, 9:32 PM

Samwalton9-WMF edited parent tasks, added: T191777: Overhaul and improve Library Card information security processes; removed: T240148: Infosec.Dec 9 2019, 10:34 AM

Samwalton9-WMF removed a parent task: T191777: Overhaul and improve Library Card information security processes.May 21 2020, 8:44 AM

Samwalton9-WMF removed a subtask: T237495: Identify the paths we'd want or not want robots to crawl.

Samwalton9-WMF moved this task from Open tasks to Incoming tasks on the Library-Card-Platform board.Oct 1 2020, 8:55 AM

jsn.sherman claimed this task.Mar 29 2021, 3:01 PM

jsn.sherman edited projects, added The-Wikipedia-Library (Kanban); removed Library-Card-Platform.

jsn.sherman updated the task description. (Show Details)

We've been having site reliability issues due to being hammered by bots. I don't see evidence of malice at this point, so I just hotfixed in a robots.txt file that asks bots to wait 10 seconds between requests. It's going through the build pipeline now.

jsn.sherman mentioned this in T237495: Identify the paths we'd want or not want robots to crawl.Mar 29 2021, 3:06 PM

This has been deployed, and is working as I'd expect:
https://github.com/WikipediaLibrary/TWLight/commit/32b4736c65f1a69747449e16b73125c3df8705eb