Page MenuHomePhabricator

Install a robots.txt file for maps.wikimedia.org
Open, Needs TriagePublic

Description

It's my first bug report, so my apologises if I'm doing something wrong...

I saw maps.wikimedia.org map-tiles showing up in the Google Image search of my website. Which means that the Google Crawler has crawled and index your server. I checked your robots.txt file but you seem to not have one. In my mind you could reduce your server load significantly by installing a robots.txt file that disallows all crawlers.
Just like OSM does: https://a.tile.openstreetmap.org/robots.txt
Or OpenTopoMap: https://a.tile.opentopomap.org/robots.txt

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 12 2019, 5:22 PM