https://auth.wikimedia.org/robots.txt currently says "Invalid request URI (requestUri=/robots.txt), can't determine language."
We may want a robots.txt that disables any bots from scrapping any pages in this website, since login links in all wikis will soon be redirected to the new auth domain.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | matmarex | T384137 Set up robots.txt in auth.wikimedia.org | |||
| Resolved | RLazarus | T385228 Please review MediaWiki Apache config changes adding new docroot for the auth domain |
Yeah, we should just ban robots unconditionally from that domain (all of the pages that are accessible on it ban robots anyway).
Change #1115103 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[operations/mediawiki-config@master] Add 'auth' docroot with custom files
Change #1115104 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[operations/puppet@production] Add 'auth' docroot with custom files (beta)
Change #1115103 merged by jenkins-bot:
[operations/mediawiki-config@master] Define new 'auth' docroot with custom files for the auth domain
Mentioned in SAL (#wikimedia-operations) [2025-01-30T15:15:08Z] <hashar@deploy2002> Started scap sync-world: Backport for [[gerrit:1115103|Define new 'auth' docroot with custom files for the auth domain (T383952 T384137)]]
Mentioned in SAL (#wikimedia-operations) [2025-01-30T15:18:25Z] <hashar@deploy2002> matmarex, hashar: Backport for [[gerrit:1115103|Define new 'auth' docroot with custom files for the auth domain (T383952 T384137)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
Mentioned in SAL (#wikimedia-operations) [2025-01-30T15:30:03Z] <hashar@deploy2002> Finished scap sync-world: Backport for [[gerrit:1115103|Define new 'auth' docroot with custom files for the auth domain (T383952 T384137)]] (duration: 14m 55s)
Change #1115104 merged by RLazarus:
[operations/puppet@production] Use new 'auth' docroot for the auth domain
Mentioned in SAL (#wikimedia-operations) [2025-02-03T19:46:58Z] <rzl@deploy2002> Started scap sync-world: T383952, T384137
Mentioned in SAL (#wikimedia-operations) [2025-02-03T20:29:56Z] <rzl@deploy2002> Started scap sync-world: T383952, T384137
Mentioned in SAL (#wikimedia-operations) [2025-02-03T20:31:53Z] <rzl@deploy2002> rzl: T383952, T384137 synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
Mentioned in SAL (#wikimedia-operations) [2025-02-03T20:33:35Z] <rzl@deploy2002> Finished scap sync-world: T383952, T384137 (duration: 06m 10s)
Change #1117924 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[operations/puppet@production] Reapply "Use new 'auth' docroot for the auth domain"
Change #1117924 merged by RLazarus:
[operations/puppet@production] Reapply "Use new 'auth' docroot for the auth domain"
Mentioned in SAL (#wikimedia-operations) [2025-02-13T19:35:49Z] <rzl@deploy2002> Started scap sync-world: T383952, T384137
Mentioned in SAL (#wikimedia-operations) [2025-02-13T19:36:58Z] <rzl@deploy2002> rzl: T383952, T384137 synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
Mentioned in SAL (#wikimedia-operations) [2025-02-13T19:44:03Z] <rzl@deploy2002> Finished scap sync-world: T383952, T384137 (duration: 11m 16s)