Page MenuHomePhabricator
Create Task
Maniphest T223041

Install a robots.txt file for maps.wikimedia.org
Closed, ResolvedPublic
Actions

Description

It's my first bug report, so my apologises if I'm doing something wrong...

I saw maps.wikimedia.org map-tiles showing up in the Google Image search of my website. Which means that the Google Crawler has crawled and index your server. I checked your robots.txt file but you seem to not have one. In my mind you could reduce your server load significantly by installing a robots.txt file that disallows all crawlers.
Just like OSM does: https://a.tile.openstreetmap.org/robots.txt
Or OpenTopoMap: https://a.tile.opentopomap.org/robots.txt

Details

SubjectRepoBranchLines +/-
Fix robots.txt content-typemediawiki/services/kartotherianmaster+4 -5
Add a robots.txt to maps.wikimedia.orgmediawiki/services/kartotherianmaster+4 -0
Customize query in gerrit

Related Objects

Event Timeline

Humppakarajat created this task.May 12 2019, 5:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 12 2019, 5:22 PM
gerritbot added a comment.Aug 8 2020, 2:44 PM

Change 619124 had a related patch set uploaded (by TheDJ; owner: TheDJ):
[mediawiki/services/kartotherian@master] Add a robots.txt to maps.wikimedia.org

https://gerrit.wikimedia.org/r/619124

gerritbot added a project: Patch-For-Review.Aug 8 2020, 2:44 PM
gerritbot added a comment.Sep 9 2020, 3:26 PM

Change 619124 merged by jenkins-bot:
[mediawiki/services/kartotherian@master] Add a robots.txt to maps.wikimedia.org

https://gerrit.wikimedia.org/r/619124

Maintenance_bot removed a project: Patch-For-Review.Sep 9 2020, 4:10 PM
MSantos added a project: Product-Infrastructure-Team-Backlog-Deprecated (Kanban).Sep 10 2020, 10:53 AM
MSantos moved this task from To Do to To Deploy on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.
MSantos moved this task from To Deploy to Blocked on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Sep 30 2020, 3:27 PM
MSantos subscribed.

Blocked on T260947: [OSM] Setup a beta cluster instance to test imposm3 DB replication

MSantos moved this task from Blocked to To Deploy on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Oct 13 2020, 1:46 PM
Stashbot added a comment.Nov 9 2020, 8:12 PM

Mentioned in SAL (#wikimedia-operations) [2020-11-09T20:12:57Z] <mbsantos@deploy1001> Started deploy [kartotherian/deploy@0a38bc5]: Add new target for beta environment and clean-up old envs (T223041 T222377 T255932)

Stashbot mentioned this in T222377: Move kartotherian/tilerator logging to new logging pipeline.Nov 9 2020, 8:13 PM
Stashbot mentioned this in T255932: Automatic zoom and positioning doesn't work for geomasks in thumbnails.
Stashbot added a comment.Nov 9 2020, 8:24 PM

Mentioned in SAL (#wikimedia-operations) [2020-11-09T20:24:33Z] <mbsantos@deploy1001> Finished deploy [kartotherian/deploy@0a38bc5]: Add new target for beta environment and clean-up old envs (T223041 T222377 T255932) (duration: 11m 36s)

Stashbot added a comment.Nov 9 2020, 8:25 PM

Mentioned in SAL (#wikimedia-operations) [2020-11-09T20:25:13Z] <mbsantos@deploy1001> Started deploy [kartotherian/deploy@0a38bc5]: Add new target for beta environment and clean-up old envs (T223041 T222377 T255932)

Stashbot added a comment.Nov 9 2020, 8:26 PM

Mentioned in SAL (#wikimedia-operations) [2020-11-09T20:26:22Z] <mbsantos@deploy1001> Finished deploy [kartotherian/deploy@0a38bc5]: Add new target for beta environment and clean-up old envs (T223041 T222377 T255932) (duration: 01m 09s)

MSantos moved this task from All map-related tasks to Production Infrastructure on the Maps board.Nov 25 2020, 1:16 PM
Jgiannelos moved this task from To Deploy to Sign off on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Apr 7 2022, 12:56 PM
Jgiannelos closed this task as Resolved.Apr 7 2022, 1:15 PM
Jgiannelos claimed this task.
TheDJ reopened this task as Open.Apr 7 2022, 6:09 PM
TheDJ subscribed.

curl -v "https://maps.wikimedia.org/robots.txt"

< HTTP/2 200
< date: Thu, 07 Apr 2022 09:34:32 GMT
< access-control-allow-origin: *
< access-control-allow-headers: accept, x-requested-with, content-type
< access-control-expose-headers: etag
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< content-security-policy: default-src 'self'; object-src 'none'; media-src 'none'; style-src 'self'; script-src 'self'; frame-ancestors 'self'
< x-content-security-policy: default-src 'self'; object-src 'none'; media-src 'none'; style-src 'self'; script-src 'self'; frame-ancestors 'self'
< x-webkit-csp: default-src 'self'; object-src 'none'; media-src 'none'; style-src 'self'; script-src 'self'; frame-ancestors 'self'
< user-agent: *
< disallow: /
< server: ATS/8.0.8
< age: 30729
< x-cache: cp3061 miss, cp3057 hit/243
< x-cache-status: hit-front
< server-timing: cache;desc="hit-front", host;desc="cp3057"
< strict-transport-security: max-age=106384710; includeSubDomains; preload
< report-to: { "group": "wm_nel", "max_age": 86400, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }
< nel: { "report_to": "wm_nel", "max_age": 86400, "failure_fraction": 0.05, "success_fraction": 0.0}
< accept-ch: Sec-CH-UA-Arch,Sec-CH-UA-Bitness,Sec-CH-UA-Full-Version-List,Sec-CH-UA-Model,Sec-CH-UA-Platform-Version
< permissions-policy: interest-cohort=(),ch-ua-arch=(self "intake-analytics.wikimedia.org"),ch-ua-bitness=(self "intake-analytics.wikimedia.org"),ch-ua-full-version-list=(self "intake-analytics.wikimedia.org"),ch-ua-model=(self "intake-analytics.wikimedia.org"),ch-ua-platform-version=(self "intake-analytics.wikimedia.org")
< x-client-ip: xxxxxxxxxxx
< accept-ranges: bytes
< content-length: 0

gerritbot added a comment.Apr 7 2022, 6:29 PM

Change 778327 had a related patch set uploaded (by MSantos; author: MSantos):

[mediawiki/services/kartotherian@master] Fix robots.txt content-type

https://gerrit.wikimedia.org/r/778327

gerritbot added a project: Patch-For-Review.Apr 7 2022, 6:29 PM
gerritbot added a comment.Jun 21 2022, 5:28 PM

Change 778327 merged by jenkins-bot:

[mediawiki/services/kartotherian@master] Fix robots.txt content-type

https://gerrit.wikimedia.org/r/778327

Jgiannelos mentioned this in rMSKAbaaa6bc79042: Fix robots.txt content-type.Jun 21 2022, 5:29 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 21 2022, 5:30 PM
Aklapper added a comment.May 23 2023, 4:41 PM

@Jgiannelos: Hi! This task has been assigned to you a while ago. Could you maybe share an update?
If this task has been resolved in the meantime: Please update the task status (via Add Action... → Change Status in the dropdown menu).
If this task is not resolved and only if you do not plan to work on this task anymore: Please consider removing yourself as assignee (via Add Action... → Assign / Claim in the dropdown menu): That would allow others to work on this (in theory), as others won't think that someone is already working on this. Thanks!

Jgiannelos closed this task as Resolved.May 23 2023, 4:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits