Page MenuHomePhabricator

Examine replacing tiles.wmflabs.org with production tile server
Closed, ResolvedPublic

Description

tiles.wmflabs.org provides OSM tiles for some community tools and other projects created and maintained by the Wikimedia technical community. The service it provides is awesome, but it runs in a Cloud VPS VM and uses an NFS share as its rendered tile cache. These things make me a bit nervous about the potential for the service to go down and cause interruptions for the on-wiki users.

@MaxSem knows a bit about use cases, but it would be good to track down and talk to the actual folks who are keeping the current tools alive to find out what the major use cases are and if they are compatible with Kartotherian.

See also:

Event Timeline

Pnorman edited projects, added Maps; removed Maps (Kartotherian).

tiles.wmflabs.org isn't running kartotherian, so we're untagging it. It's mainly a community relations item.

Chris, can you take on letting users of tiles.wmflabs.org know that they can switch to our production tiles if they want to?

CKoerner_WMF subscribed.

I'm sure someone (if not me) on my team can help with communicating, but we'd need a specific task for that request. :) I'm a little booked up at the moment and with Wikimania next week won't be able to be much immediate help.

Some quick feedback to make things go more smoothly, I personally don't know much about the use of tiles.wmflabs.org. Is there any list (or could one be generated) of projects using it? This way we would have a better understanding of who our audience is and how to reach them.

Some quick feedback to make things go more smoothly, I personally don't know much about the use of tiles.wmflabs.org. Is there any list (or could one be generated) of projects using it? This way we would have a better understanding of who our audience is and how to reach them.

We do not have such a list today, but it should be possible to create at least a partial inventory using the access logs from the shared http proxy that the requests pass through.

Well WMA uses a different projection and a different tile scheme, it's hard to move it to our production tiles. I tried a couple of times but with little actual mapping background, I couldn't really figure it out.

Also.. It might be wise to actually talk to the people who built said services before you tell their users to move out. I know from talking to these people they are already rather disappointed about how some of the things have evolved over the years.

Also.. It might be wise to actually talk to the people who built said services before you tell their users to move out. I know from talking to these people they are already rather disappointed about how some of the things have evolved over the years.

@TheDJ, pardon my ignorance, but where's a good place to find these folks? m:Talk:WikiMiniAtlas?

For WMA specifically, it's @dschwen. You might have to email, since he seems rather busy off wiki these days.

Hey all. yes, WMA uses lat lon (plate carree) projection. Tile services are an important part, but people are painfully unaware that tiles are just a small part of the WMA. It had 3D buildings, article markers with text labels, article summaries on hover, area comparison by reprojection, entity highlighting (area shading for the article subject), it showed all coordinates from teh current article (not just the main coordinate - nice in list articles). WMA had support for globes besides earth (moon, mars, mecury, io, etc.), it had client side rendered tiles at high zoom levels, it has user interface translations and article labels from several dozen projects (including thumbnails from commons).

All that was destroyed by not permitting user DBs on the main database servers anymore. It is nice that you guys are discussing "tiles" but like most iterations of this discussion, it misses the point. :-(

FYI. I have now made tiles.wmflabs.org tiles return a static image. I don't feel like dragging this thing on for another 4 years as it is not really something i enjoy. Especially with the unreliability of the OSM database sync for wmflabs and no one invested in fixing that either. This means the following:

  1. Standard multilingual OSM mapnik tiles will no longer be available to the community. Use the standard WMF maps using: https://maps.wikimedia.org/osm-intl/{z}/{x}/{y}{r}.png?lang={lang} or the OSM server directly https://tile.openstreetmap.org/${z}/${x}/${y}.png if you really need that particular style
  2. OSM mapnik nolabels style will no longer be available (WMF maps has a replacement in https://maps.wikimedia.org/osm/{z}/{x}/{y}{r}.png)
  3. The hillshading tiles will not longer be available (no known community alternative)
  4. The hikebike map that we were hosting will no longer be available (no known community alternative)
  5. The OSM black and white tiles are no longer available (no known community alternative)

If anyone is interested in taking it over, i'm willing to show you around the server. If no one volunteers, it will likely be deleted soon in the upcoming purge: https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge

To fully cleanup, someone needs to remove the tile data from the NFS server btw. /data/project/tiles/

tiles.wmflabs.org has never generated tiles for the Wikiminiatlas!

The Wikiminiatlas does access the OSMdb mirror though. So it would be great if that stayed and I'd be happy to try and help maintaining that mirror.

By the way, my rant above is outdated. I rewrote the data extraction for the Wikiminiatlas a year ago, and I just rebuilt the wma VM with Debian 11. Everything should be fully functional.

tiles.wmflabs.org has never generated tiles for the Wikiminiatlas!

Yeah, that's really old info that was already corrected multiple times by ppl, i've update the description of this ticket, seemed about time.

The Wikiminiatlas does access the OSMdb mirror though. So it would be great if that stayed and I'd be happy to try and help maintaining that mirror.

It's not planned to go offline, it's just that it's outdated and bugged and has problems with sync'ing. There's been a proposal to put that database back into the hands of volunteers, but i'm not interested. However, tools like WMA, wiwosm and several other POI layer types of services also depend on it, so getting it back into a more reliable state again would be advantageous I think. See also: T285668

FYI. I have now made tiles.wmflabs.org tiles return a static image. I don't feel like dragging this thing on for another 4 years as it is not really something i enjoy. Especially with the unreliability of the OSM database sync for wmflabs and no one invested in fixing that either. This means the following:

  1. Standard multilingual OSM mapnik tiles will no longer be available to the community. Use the standard WMF maps using: https://maps.wikimedia.org/osm-intl/{z}/{x}/{y}{r}.png?lang={lang} or the OSM server directly https://tile.openstreetmap.org/${z}/${x}/${y}.png if you really need that particular style
  2. The hillshading tiles will not longer be available (i know of no alternative)
  3. The hikebike map that we were hosting will no longer be available
  4. The OMS black and white and nolabels style will no longer be available

If anyone is interested in taking it over, i'm willing to show you around the server. If no one volunteers, it will likely be deleted soon in the upcoming purge: https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge

To fully cleanup, someone needs to remove the tile data from the NFS server btw. /data/project/tiles/

Could we get in touch so you could tell me what maintaining this server entails? @TheDJ

FYI. I have now made tiles.wmflabs.org tiles return a static image. I don't feel like dragging this thing on for another 4 years as it is not really something i enjoy. Especially with the unreliability of the OSM database sync for wmflabs and no one invested in fixing that either. This means the following:

  1. Standard multilingual OSM mapnik tiles will no longer be available to the community. Use the standard WMF maps using: https://maps.wikimedia.org/osm-intl/{z}/{x}/{y}{r}.png?lang={lang} or the OSM server directly https://tile.openstreetmap.org/${z}/${x}/${y}.png if you really need that particular style
  2. OSM mapnik nolabels style will no longer be available (WMF maps has a replacement in https://maps.wikimedia.org/osm/{z}/{x}/{y}{r}.png)
  3. The hillshading tiles will not longer be available (no known community alternative)
  4. The hikebike map that we were hosting will no longer be available (no known community alternative)
  5. The OSM black and white tiles are no longer available (no known community alternative)

If anyone is interested in taking it over, i'm willing to show you around the server. If no one volunteers, it will likely be deleted soon in the upcoming purge: https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge

To fully cleanup, someone needs to remove the tile data from the NFS server btw. /data/project/tiles/

@TheDJ I may be interested in supporting the hikebike map and especially the hillshading overlay tiles. Please contact me for further details.

@DasKomplott I'm not going to be contacting all ppl individually in private.. this is going to be where the discussion is ;)

@Thatent

  1. The server is running within wikimedia cloud services. See also: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction
  2. I or another cloud admin needs to grant you access to the service
  3. the server is currently a Debian Stretch VM, and needs to be replaced with a new VM https://wikitech.wikimedia.org/wiki/News/Stretch_deprecation
  4. This new server needs to be setup similar to the old server (there is no puppet config or anything right now for this unfortunately)
  5. The server probably needs to be expanded to a multi cluster setup. It is currently one node, but the renderer crashed all the time because of various load problems. It should probably be split into multiple render and web host nodes
  6. The stack is mapnik + mod_tile + renderd. This is rather old, maybe someone should eventually switch it to newer kind of stack ?
  7. Tile expiration is broken, as the osmdb runs on a separate host, and we aren't processing the expiration information. See also: T285668#7345301
  8. In general, keep an eye on the statistics, to make sure that things run the way they are supposed to be running and respond to community members when things go wrong.
  9. Update all of it again, in a couple of years when the next version of the server's OS expires ;)

Some details about the current setup are in this page: https://wikitech.wikimedia.org/wiki/OSM_Tileserver

@MSantos BTW. the hill shading tiles are pretty much static tiles that never get refreshed (earth doesn't change that much). I don't think we even have the script any longer that generated them from the NASA data. It might be an idea to just put those on production under /hillshading ?

I'm currently importing a new planet file into a database within the maps project. While all nodes completed import last night the ways import goes a lot slower and I expect it to last a few days (up to two weeks).

Once I enable replication we can expire tiles. If the tiles are on NFS, that will be straightforward. Else we can come up with a custom solution.

Actually, there is a hillserver functional at http://osmscout.karry.cz/hillshade/. But it i not suited for mass usage. Maybe someone wants to support the owner of that server to make it able to bear big loads?

Also, it seems to cover the whole world.

como resolvi

$wgKartographerMapServer = 'https:/';
$wgKartographerDfltStyle = 'tile.openstreetmap.org';

pode conferir https://wikie.com.br/Feira_Nova_do_Maranh%C3%A3o

@TheDJ: Here: https://help.openstreetmap.org/answer_link/83282/ is someone willing to volunteer but does not know how to contact you.

Can you contact that person?

Regards!

@TheDJ: Here: https://help.openstreetmap.org/answer_link/83282/ is someone willing to volunteer but does not know how to contact you.

Can you contact that person?

I'm pretty sure I spoke to them back then via another channel. Anyways, that server now no longer exists, so.

TheDJ claimed this task.

Server no longer exists, closing ticket