Page MenuHomePhabricator

Find out who maintains http://hikebikemap.org/
Open, Needs TriagePublic

Description

Traffic to the tile servers hosted in the maps Cloud VPS project seem to be up significantly. The traffic through the shared project-proxy fronting system has reached a point where the operation of other Cloud VPS projects is being affected (e.g. T190218).

I would like to be in better contact with the volunteers who run the tile servers and the long term upstream consumer at http://hikebikemap.org/ (https://wiki.openstreetmap.org/wiki/Hike_%26_Bike_Map) so we can think about changes that can be made to reduce impact on other Cloud VPS users.

Event Timeline

bd808 created this task.Mar 22 2018, 8:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 22 2018, 8:45 PM

Hi, just a quick note - I am the hikebikemap.org guy.

Heaviest hikebike tile users:

root@maps-tiles3:/var/log/apache2# grep /hikebike/ other_vhosts_access.log | awk '{ print $9; }' | grep http | grep -oP '(?<=://)[^/"]*' | sort | uniq -c | sort -rn | head -n 100
1165417 www.gpsies.com
 238723 www.apemap.com
  94155 hikebikemap.org
  55847 m.gpsies.com
  20521 www.hikebikemap.org
   9791 www.mazovia.zamanagroup.pl
   9705 routen.vfdnet.de
   7980 leaflet-extras.github.io
   7149 www.gaiagps.com
   6668 osmapa.pl
   5545 www.mazoviamtb.pl
   5202 rowery.rbgp.pl

Because it seems like it's absolutely not used by anything Wikimedia related, I would say that the very serving of hikebike tiles goes against Labs ToS and needs to be disabled.

TheDJ added a comment.Mar 23 2018, 1:41 PM

I did include it in my list of 'demo's' https://tools.wmflabs.org/maplayers-demo/
Which means I found it in use 'somewhere' on tools.wmflabs.org, but I can't find it right now which tool that was. I thought it was en.wikivoyage, but it seems it uses waymarkedtrails.org for separate hiking and biking layers..

TheDJ added a comment.Mar 23 2018, 2:46 PM

hikebikemap notes: "In lieu of a dedicated tile usage policy on Wikimedia Labs at this time, please respect the Tile usage policy of https://openstreetmap.org."
Which is https://operations.osmfoundation.org/policies/tiles/

Which notes: "Heavy use (e.g. distributing an app that uses tiles from openstreetmap.org) is forbidden without prior permission from the System Administrators"

That's exactly what those first 2 are doing, so I guess those top 2 can immediately be blacklisted if we truly have a problem.

TheDJ added a comment.Mar 23 2018, 2:50 PM

leaflet-extras is likely due to: http://leaflet-extras.github.io/leaflet-providers/preview/ and https://github.com/leaflet-extras/leaflet-providers/search?utf8=✓&q=hikebike Which is possibly where i recognised it from as well, since my maplayers-demo is a derivative from leaflet-extras

Not so quick with Gpsies please - quite some years back we were in contact with first River and later Nosy to discuss whether/how it was acceptable to use these tiles with heavy caching on Gpsies side. (This is with me not having any further relationship to Klaus Bechtold of Gpsies other than that he liked the map style, and me liking Gpsies, so I cannot speak for Klaus.)
Things were fine since. If that is not the case anymore, there should at least be a grace period - Klaus has very much been a good citizen in this game.

I have no idea about apemap.

bd808 added a comment.Mar 23 2018, 3:43 PM

Not so quick with Gpsies please - quite some years back we were in contact with first River and later Nosy to discuss whether/how it was acceptable to use these tiles with heavy caching on Gpsies side.

Is that discussion documented publicly somewhere? The criteria to judge acceptable use (and traffic levels) for toolserver.org may or may not have much relevance now that hosting is provided by Wikimedia Cloud Services.

I really do not want to drive out legitimate usage that furthers the goals of the Wikimedia movement, but I also need to figure out how to keep this particular project from causing undue impact on the other 211 projects that share resources with it. The Cloud Services team and the Wikimedia SRE team invest a non-trivial amount of hardware and support time into keeping a postgres database and a dedicated NFS share available for the maps project. I can tell by scanning log files that the majority of egress traffic from the Cloud VPS cluster is attributable to the tile servers. What I really can't tell is what the benefit and impact of the use of these resources is for the Wikimedia movement. I would like to start by understanding who is actively maintaining the project and what the project goals are.

One such thread was https://lists.wikimedia.org/pipermail/maps-l/2012-January/001103.html - IIRC there were more, but I'm on my phone right now...

Nosy79 added a subscriber: Nosy79.Mar 23 2018, 5:17 PM

I indeed had heavy load on the web servers caused by gpsies, too. Due to the limited hardware ressources the first thing I did was to throttle the regarding traffic via nginx.
When the load/lactency was a problem again I gave them the whole hillshading tiles to host them themselves.
Indeed Klaus was quite cooperative.

bd808 added a comment.Mar 23 2018, 5:21 PM

One such thread was https://lists.wikimedia.org/pipermail/maps-l/2012-January/001103.html - IIRC there were more, but I'm on my phone right now...

That January 2012 message includes "Next time I'm planning to produce own tiles for GPSies.com. The (3 new) servers are running and we must now configure it, but this will take some time." It seems like six years is a more than reasonable time for them to have created their own infrastructure?

@cmarqu, you got permission from the old Toolserver admins. That project had scope that allowed projects related to either Wikimedia or OpenStreetMap. This here is Wikimedia Cloud that is run by a different organization and has different rules. We only allow stuff that that benefits Wikimedia projects and Toolserver rules aren't grandfathered.

I have two points, first the relation between Wikimedia Movement and OSM and second my opinion what we should do with gpsies and apemap as the largest external users.

First, I believe it's wrong to say that it's not the jobs of wmflabs to support OpenStreetMap community. It would be against the work of many local chapters of the WMF.
Examples: At WMDE we have OSM explicit in our support rules ("Förderrichtlinien") [1]. WM Italy is a local chapter of OSMF[2]. And so on.
(In my personal eyes the problem is that the central WMF organisation doesn't find the right way to cooperate with OSM and develop the feature rich map integration inside Wikimedia projects, something the communities dreams from. Where is a maps support inside the mobile Wikimedia app to see the map around an article? Where are multilingual maps? What's the future of kartographer? Where is a strategy and an active maps team? Sorry for my frustrated, hard words.)
If it would be an official WMF statement that OSM is outside the scope of WMFlabs it would violate the discussions we had at the end of the live of toolserver and local WMF-chapters that want to support general free knowledge should start to discus new independent server infrastructure.

Now to my second point gpsies. It seems Gpsies have now a kind of an business model and with 12 servers a relative large infrastructure. So I believe we should give them some acceptable time (1-2 Month) to install a caching system or an render stack and then start to block them.

In the past the plan was to integrate OSM-maps inside Wikipedia and number of requests are than in billions, if so it would be no problem to deliver also some tiles to a small page like gpsies. But I have full respect to tell them that we can not further deliver tiles them.

But we should further support the openstreetmap community and keep the beautiful tiles of hikemap and the hillshading-tiles free for external maps from non-profit volunteers and small pages without significant traffic.

[1] https://de.wikipedia.org/wiki/Wikipedia:Wikimedia_Deutschland/F%C3%B6rderrichtlinien#F%C3%B6rderung
[2] https://wiki.openstreetmap.org/wiki/Foundation/Local_Chapters