Page MenuHomePhabricator

Install a robots.txt file for maps.wikimedia.org
Open, Needs TriagePublic

Description

It's my first bug report, so my apologises if I'm doing something wrong...

I saw maps.wikimedia.org map-tiles showing up in the Google Image search of my website. Which means that the Google Crawler has crawled and index your server. I checked your robots.txt file but you seem to not have one. In my mind you could reduce your server load significantly by installing a robots.txt file that disallows all crawlers.
Just like OSM does: https://a.tile.openstreetmap.org/robots.txt
Or OpenTopoMap: https://a.tile.opentopomap.org/robots.txt

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 12 2019, 5:22 PM

Change 619124 had a related patch set uploaded (by TheDJ; owner: TheDJ):
[mediawiki/services/kartotherian@master] Add a robots.txt to maps.wikimedia.org

https://gerrit.wikimedia.org/r/619124

Change 619124 merged by jenkins-bot:
[mediawiki/services/kartotherian@master] Add a robots.txt to maps.wikimedia.org

https://gerrit.wikimedia.org/r/619124

Mentioned in SAL (#wikimedia-operations) [2020-11-09T20:12:57Z] <mbsantos@deploy1001> Started deploy [kartotherian/deploy@0a38bc5]: Add new target for beta environment and clean-up old envs (T223041 T222377 T255932)

Mentioned in SAL (#wikimedia-operations) [2020-11-09T20:24:33Z] <mbsantos@deploy1001> Finished deploy [kartotherian/deploy@0a38bc5]: Add new target for beta environment and clean-up old envs (T223041 T222377 T255932) (duration: 11m 36s)

Mentioned in SAL (#wikimedia-operations) [2020-11-09T20:25:13Z] <mbsantos@deploy1001> Started deploy [kartotherian/deploy@0a38bc5]: Add new target for beta environment and clean-up old envs (T223041 T222377 T255932)

Mentioned in SAL (#wikimedia-operations) [2020-11-09T20:26:22Z] <mbsantos@deploy1001> Finished deploy [kartotherian/deploy@0a38bc5]: Add new target for beta environment and clean-up old envs (T223041 T222377 T255932) (duration: 01m 09s)