Page MenuHomePhabricator

Define tile usage policy
Closed, ResolvedPublic

Description

  • For everyone: basic attribution / other legal requirements
  • For core WMF use (deployed extensions/apps)
  • For WMF-related tools (e.g. labs/site scripts)
  • For third parties: do we allow them at all?

Event Timeline

MaxSem created this task.Aug 1 2016, 9:22 PM
Restricted Application added a project: Discovery. · View Herald TranscriptAug 1 2016, 9:22 PM
Restricted Application added subscribers: JEumerus, Aklapper. · View Herald Transcript
BBlack added a subscriber: BBlack.Aug 1 2016, 9:37 PM

In general we'll probably allow third parties, as we've done for all of our other content in the general case (e.g. upload.wm.o images are linked and embedded all over the web). I'm probably not the best person to ask about attribution/legal stuff. We don't want to do anything there that would be burdensome for re-users, but we probably need something.

On the pragmatic level with traffic rates and all of that:

  • We always reserve the right to block abusive/excessive traffic, and the practical definition of that is "if we notice it hurting our service or impacting us financially". Even if everything about some third-party use is "legal" and obeys our other rules, we're probably not going to let all other use of the service fail due to their onslaught of traffic, nor are we obligated to make large new investments in hardware and/or network links to support some random third party's excessive use.
  • If we're writing down some rules somwhere, we should probably ask that anyone planning to send us statistically-significant traffic (e.g. rates that are easiest expressed in reqs/sec) make contact and at least talk to us about their plans and expectations. It's good opportunity to talk about whatever attribution/legal issues there might be anyways.
Legoktm added a subscriber: Legoktm.Aug 1 2016, 9:41 PM
ZhouZ moved this task from Backlog to Assigned on the WMF-Legal board.Aug 1 2016, 10:57 PM
Slaporte claimed this task.Aug 1 2016, 11:33 PM

Hi,

I'm the developper of the app E-walk: https://play.google.com/store/apps/details?id=com.at.ewalk.free

I'd definitively be interested if we could use Wikimedia maps, with a policy close to the OSM's policy (allowing everything, and keeping the right to block abusing software). More specifically, being able to cache (store the tiles for a limited time while the user browses the map, in order to improve performance) and bulk download (select an area of the map, a min and a max zoom levels, and downloading each tiles one after one) the map would be interesting.

I'll keep an eye on the project!

bulk download (select an area of the map, a min and a max zoom levels, and downloading each tiles one after one)

Bulk downloading high zoom image tiles is something almost certain to put an excessive load on the servers as these tiles will not be pre-rendered.

Bulk downloading the vector tiles and rendering client-side is technically feasible.

Attribution-wise, anyone using your tiles would need proper attribution to OSM. It's up to you if you want to require additional attribution for Wikimedia as the server of the tiles or as the author of the style.

For reference, the OSMF tile usage policy is http://wiki.openstreetmap.org/wiki/Tile_usage_policy, and the technical requirements of a valid user-agent or HTTP referer are now being automatically enforced.

timautin added a comment.EditedSep 3 2016, 10:52 AM

Thanks for the input!

Downloading vector tiles and rendering would be really interesting, do you have more information (which tools, which format for the vector tiles, etc)?

No problem for the attribution, I already have an HTML box on top of the map dedicated to it.

Yurik added a subscriber: Yurik.Sep 3 2016, 3:23 PM

@timautin our tiles are stored in Mapbox's vector tile format, and we use this style to convert it to images. It should be possible to set up a local copy of Tilerator to do the rendering from the downloaded tiles, but we do not yet have a bulk vector tile export (T142354)

MaxSem moved this task from Backlog to In progress on the Maps-Sprint board.Sep 14 2016, 8:39 PM
debt added a subscriber: debt.Oct 5 2016, 2:32 PM

Some discussions about the equivalent OSM usage policies: https://github.com/openstreetmap/operations/issues/113

debt triaged this task as High priority.Nov 4 2016, 2:15 PM
debt added a subscriber: Gehel.

@Slaporte - this is interesting and might affect what the Foundation decides...

Some discussions about the equivalent OSM usage policies: https://github.com/openstreetmap/operations/issues/113

@Gehel can you take a look at this comment - is this even feasible for our servers?

Gehel added a comment.Nov 5 2016, 8:49 AM

@debt: as @BBlack pointed out in the start of this thread, we tend to have a fairly liberal view on who can reuse our content / services. Bulk download of all tiles probably falls into the "abusive / excessive use case", but synchronous usage of our tiles is probably not much of an issue. I'm still working on some load testing for our servers, which might give us a better idea of the load they can actually support.

grin added a subscriber: grin.Nov 7 2016, 6:31 PM
grin added a comment.Nov 7 2016, 6:40 PM

@debt: as @BBlack pointed out in the start of this thread, we tend to have a fairly liberal view on who can reuse our content / services

This is a bit larger in magnitude if it starts off; as Firefishy wrote in the quoted ticket the current traffic of the OSM tileservers are around 6800GB/day in over 528 million requests (average of around 13.6KB per request). While this traffic will not just 'come over' to us it'd be probably prudent to plan not to get the servers slashdotted. :-)

We're talking about valid requests in high amounts. An example was Locus Map on Android which was downloaded to a few millions of mobiles and were requesting just the local area in various zooms, which was pretty soon noticed by the server admins and banned with "implement some strict limits please" message. The result is that the app had to slow down display experience to comply.

Similar would be extensive usage of leaflet inserts on webpages showing a simple map with company address or like. Mapquest seems to have been bitten by that and have closed their open servers recently. And it is a really simple and a definitely valid usage (pattern).

I definitely support doing it but I suggest capacity planning, too.

Deskana added a subscriber: Deskana.Nov 7 2016, 6:55 PM

I think this task is conflating two separate issues.

  • Define technical tile usage policy (i.e. what users can do without melting the servers down)
  • Define ethical tile usage policy (i.e. attribution requirements, legal requirements, etc.)

What might be acceptable to one (e.g. "Reuse all you like provided you attribute") might not be acceptable to the other (e.g. "Please don't bulk download all our tiles or you'll overload the service"). The two have to be combined into a single policy in the end, but it may be beneficial to split the discussion out.

Gehel added a comment.Nov 7 2016, 7:17 PM

@Deskana as always, you have the word of wisdom!

In any case, this is a discussion that will take some time, and that will need to be rolled out progressively. We will definitely not advertise our tiles broadly at first and we need to have better visibility on how much traffic our current infrastructure can actually handle before we move forward. I was mainly pointing out that the idea in itself has merit and is inline with what we usually do.

grin added a comment.Nov 8 2016, 7:34 AM

Another sidenote: this decision should have a good visibility to the people planning server resources.
And I try to ask around MapQuest what traffic levels did they observe before throwing it in.

debt moved this task from In progress to Stalled/Waiting on the Maps-Sprint board.Nov 17 2016, 6:16 PM

Moving to 'stalled' as we don't have wording from Legal.

And I try to ask around MapQuest what traffic levels did they observe before throwing it in.

Some statistics from the HOT layer were that when MQ Open closed non-osm.org tile requests went from 2.2M/day to 8.5M/day. I don't have any numbers directly from MQ Open (at least not that I can share), but I wouldn't assume issues MQ Open had are applicable to what we'll have.

grin added a comment.Nov 18 2016, 2:39 PM

Thanks for the reminder, I've got a word back from MQ, and they said, that in 2014 MapQuest served 380 million Open Tiles per day, 9.3 million Open geocodes per day, and 38 million Open reverse geocodes per day (these numbers were readily available).

debt awarded a token.Nov 18 2016, 11:13 PM
debt added a comment.Jan 4 2017, 7:25 PM

Added a meeting to chat about this with Legal and the Interactive team on Thursday, Jan 5, 2017

We clarified the tile usage policy in the Maps Terms of Use:

Using maps in third-party services
If you use the Wikimedia Maps service for your third-party project (that is, outside of the Wikimedia projects), please respect our limited services and resources. We reserve the right to discontinue or change our service, block or limit certain users or applications, or take other measures in cases at our sole discretion at any time without notice. If you use the service too heavily, it could affect the service's stability for others or degrade our quality of service.
If you are developing an application that uses the Wikimedia Maps service, you must provide a valid HTTP User-Agent that includes your application, version, and sufficient information to easily contact you (e.g., your email address).
The following are prohibited on the Wikimedia Maps service:

  • Excessive downloading (such as downloading significant areas of tiles for later offline usage)
  • Accessing the service without a proper HTTP User-Agent or HTTP referer
  • Using the service without compliance with any license or copyright terms

@timautin , this means you are welcome to download or use tiles in your app, but please note the caveats above. The policy may change as we continue to work on the Wikimedia Maps service, so I wouldn't recommend relying on it in a widely used app for now.

debt moved this task from Stalled/Waiting to Done on the Maps-Sprint board.Jan 10 2017, 12:01 AM

Thanks for the update, @Slaporte !

Great, thanks! So this means that caching (storing tiles as the user browses the map to improve performance) is allowed but not bulk downloading (the user selects an area, min and max zoom and the app download the area), right? Or both are allowed as long as there is no excessive usage? E-walk does not set a limit for now but warns the user if he tries to download more than 5000 tiles.

debt added a comment.Jan 17 2017, 5:11 PM

Hi @timautin - yes, these things can be done as long as there isn't excessive usage, as noted by @Slaporte in the above comment.

debt closed this task as Resolved.May 30 2017, 7:22 PM