Generate location maps dynamically, producing actual images
Open, HighPublic

Description

This is a long-standing and extremely popular feature request, I'm surprised that I don't find a report. Please merge any duplicate.

Template:Location_map (https://www.wikidata.org/wiki/Q5625881) and friends use a manually-crafted SVG map and add a manually-centred point marker to show the location on the map of the page subject. Apart from its huge maintenance cost for editors and imperfect coverage, this system doesn't work reliably outside of desktop browsers (see blocked tasks).

All SVG locator maps (thousands, used in about a million articles) and maps of country subdivisions (over 600k files) should be replaced by maps dynamically generated from a map server / tileserver based on coordinates and perhaps another parameter.

Priority set to high as there is an entire team working on maps.wikimedia.org, presumably for this very purpose.

I discovered on https://lists.wikimedia.org/pipermail/maps-l/2015-November/001443.html that a new extension was created: https://www.mediawiki.org/wiki/Extension:Kartographer . The extension page doesn't clearly mention this use case, which however seems implied by some of the bullets. Maps maintainers are kindly asked to move this from MediaWiki-extension-requests to Maps (Kartographer) once the component is created, if this use case falls within the scope of the extension.

Nemo_bis created this task.Dec 8 2015, 10:23 AM
Nemo_bis updated the task description. (Show Details)
Nemo_bis raised the priority of this task from to High.
Nemo_bis added subscribers: Nemo_bis, TheDJ, Kolossos.
Restricted Application added a project: Discovery. · View Herald TranscriptDec 8 2015, 10:23 AM
Restricted Application added subscribers: Steinsplitter, Aklapper. · View Herald Transcript
Nemo_bis updated the task description. (Show Details)Dec 8 2015, 11:25 AM
Nemo_bis set Security to None.
zhuyifei1999 moved this task from Incoming to Backlog on the Commons board.Dec 8 2015, 11:55 AM
zhuyifei1999 added a subscriber: zhuyifei1999.
Yurik added a subscriber: Yurik.Dec 8 2015, 1:18 PM

Hi, this is indeed an important feature request, and there are two very different approaches to it.

Tile-based map

Use our maps.wikimedia.org server to generate tiles of the area, stitch them together (static image) and overlay the image with extra data (e.g. styled GeoJson).
PROs: map can be switched to interactive mode to allow moving and zooming. Users can easily add more data by editing the geojson in the page.
CONs: map style is always created for a specific usecase, and the current style might not be ideal for this usage. The images above show just the administrative boundaries, very few cities and no roads. Our maps style is much more "crowded", so we might need to make a separate style specifically for this usecase.

Graph-based map

Use <graph> tag to draw the map. Store map data in GeoJson/TopoJson format as wiki pages, preferably on commons. Graph extension allows overlaying maps with extra data, like cities, and produces a server-side image. Soon it will have interactive capability as well.
PROs: very flexible, can draw data in any style and overlay with any extra information, multiple graphs can reuse and clip the same data
CONs: more complex to use initially (until templates are built), requires data to be stored as non-formatted wiki text

For the above usecase, I am inclined to suggest the graph path for now, as it offers far greater control over visualization

Ricordisamoa added a subscriber: Ricordisamoa.

generate tiles of the area, stitch them together (static image) and overlay the image with extra data (e.g. styled GeoJson)

Sorry, I have no idea what this means.

make a separate style specifically for this usecase

Yes, sounds like a suitable subtask to resolve this feature request.

Store map data in GeoJson/TopoJson format as wiki pages, preferably on commons

Doesn't sound like an option. SVG is a much better storage than this scenario, because SVG files can be easily transcluded while everything else can't.

Yurik added a comment.Dec 8 2015, 9:49 PM

generate tiles of the area, stitch them together (static image) and overlay the image with extra data (e.g. styled GeoJson)

Sorry, I have no idea what this means.

Regular map consists of square images (e.g. this). Stitching these squares together on the server produces a static image of the needed size, like this one. Eventually our server will be able to add some user-given data (in GeoJSON format), like highlighting an area or draw a pushpin as part of that static image.

make a separate style specifically for this usecase

Yes, sounds like a suitable subtask to resolve this feature request.

Sure, the instructions on how to edit a style are here - would be great to get some help on this - we don't have any map designers unfortunately.

Store map data in GeoJson/TopoJson format as wiki pages, preferably on commons

Doesn't sound like an option. SVG is a much better storage than this scenario, because SVG files can be easily transcluded while everything else can't.

You are correct that transcluding an image is at this point much easier than transcluding a graph or a template, because images can be reused in all wikis, whereas each wiki must have its own copy of the graph/template. That's one of the biggest limitations in my opinion. Yet, <graph> tag allows you to get the map data from commons, and draw indicators on top of it. Even though <graph> allows you to use SVG as well, SVG by itself is not a good solution, because it is basically an image, which means you won't be able to manipulate graph data to highlight areas. Your goal is to have a small number of graph data pages stored somewhere, and many pages that use that data. So if you have one large geojson file that describes administrative boundaries of all the countries of the world, you can set up a template based on the <graph> that draws an image from that file, centering it in France, clipping everything around it, highlighting Aquitaine region, and drawing the marker for the city of Bordeaux. Obviously all these are template parameters.

Nemo_bis added a comment.EditedDec 9 2015, 9:00 AM

Eventually our server will be able to add some user-given data (in GeoJSON format), like highlighting an area or draw a pushpin as part of that static image.

Can you file a subtask for this? Rather than "eventually", this sounds like one of the very first tasks to work on; otherwise I see little benefit in this whole exercise.

would be great to get some help on this - we don't have any map designers unfortunately.

You could start by documenting the styles used by the locator maps and country subdivision maps, then ask feedback on the requirements so assembled from the users who were most active creating said images.

Yurik added a comment.Dec 15 2015, 1:26 AM

Eventually our server will be able to add some user-given data (in GeoJSON format), like highlighting an area or draw a pushpin as part of that static image.

Can you file a subtask for this? Rather than "eventually", this sounds like one of the very first tasks to work on; otherwise I see little benefit in this whole exercise.

Yep, see T114820, and check out the demo at http://vem.wmflabs.org

would be great to get some help on this - we don't have any map designers unfortunately.

You could start by documenting the styles used by the locator maps and country subdivision maps, then ask feedback on the requirements so assembled from the users who were most active creating said images.

Yep, T113912 is still waiting to be worked on. Also, see future plans page where we place all community ideas and suggestions.

Beside the technical issues, discussed here, I like to note, that there are lot of reasons, why we use hand made maps today for the location maps:

  • Until today there is no global geodata set, that contains up-to-date and correct administrative boundaries for all countries. For lot of countries it is pretty difficult to keep the maps up-to-date and correct. We have to find out these information painfully from some government web sites. Usually the cartographers at de:Wikipedia:Kartenwerkstatt and fr:Wikipédia:Atelier graphique/Cartes are keen to keep these maps up-to-date, if in some country the boundaries changes. There is also an ongoing QS process for location maps.
  • We also have no global geodata set, that has good small to med scale topographic data for all countries. For example the correct course of main rivers is a huge huge problem in datasets like Natural Earth Data.
  • Several location maps use specific map projections (edc, laea etc) suitable for that area. For example for Russia, the US or Antarctica. The areas around the poles cannot handled by tile servers with mercator projection. For some countries as the US the map uses also inset maps (for Hawaii, Alaska)
  • For disputed boundaries today each Wikipedia can decide to use their own set of location maps as they like. From the discussions I noticed, it would be absolutly impossible to force each Wikipedia using the same maps.
  • Beside the political issues, some Wiki Projects insist to use their own color scheme (for example the US and France traditionally use different color schemes)
  • Additionally to the administrative maps we also use relief maps for topographic features like mountains etc.

... hope you get an idea now, that this topic is not much about technical problems, but much more about the absence of good geodata, the lack of tile rendering machines to produce good small scale maps and certain political issues.

Alexrk2 added a comment.EditedDec 19 2015, 1:18 PM

PS: we have to differentiate between location maps and locator maps.

  • Location maps are blank maps used for Template:Location_map ...

Example: Location map of the state of Brandenburg (Germany)
https://commons.wikimedia.org/wiki/File:Brandenburg_location_map.svg

  • Locator maps made out of location maps and highlight a specific area.

Example: Locator map that highlights a district in Brandenburg
https://commons.wikimedia.org/wiki/File:Locator_map_HVL_in_Brandenburg.svg

... location maps are not the problem, their quantity is manageable. A problem though are the locator maps wich exists in great quantities and derived from the location maps (many of them are out dated and not in use).

Yurik added a comment.EditedDec 19 2015, 3:19 PM

@Alexrk2, thanks for the great in-depth reply!

  • Geodata - do you think we should introduce support for the user-contributable geodata sets? Our maps effort pulls all the data from OSM, but they concentrate on higher zoom levels, making their data too detailed. There are some algorithms that reduce its size & quality, but I heard they are not very good. We could support geodata files on commons, and use <graph> to show it in any projection you need, in the colors you choose, with additional area and POI highlighted. The data files could be larger than the displayed area, and clipped to the needed size.
  • Hill shading is not yet supported by either maps or graphs code, but they could use raster images as the base, and overlay them with extra marks.
  • Lastly, @Alexrk2, would you be interested in participating in the hangout chat about maps on Monday, as announced on maps-l?
Alexrk2 added a comment.EditedDec 19 2015, 6:36 PM

Yurik, as you said, I think user-contributed geodata is an essential key to this problems. Because at Wikipedia we all want highly up-to-date maps and we want to have the ability to edit these maps on our needs.

At the moment I see two ways:

  1. Create some base SVG files with special tags inside from where we can automatically derive a set of locator maps.
    • pro: SVG easy to edit, can be uploaded to Commons
    • con: presumably not all editors are aware of these "magic" SVG tags
  1. Set up some kind of versioned geodatabase, where the cartographers can create the same kind of geodata as they did in the SVG files today
    • pro: highly flexible, scalable
    • pro: robust datamodels
    • pro: probably good for concurrent multi user edits?
    • con: doubtful whether there is a good editor (GUI) cartographers are happy with
    • con: doubtful whether there is a framework with a simple checkin/checkout process and easy to understand datamodel

At this point I don't know, what way is the most promising or the most pragmatic one.

Maybe it's worth to have a look on http://geogig.org/ which supposed to be Git for geospatial data.

Maybe it's sufficent to set up a PostGIS server that can be used by multiple users over the net (with QGIS? or a web interface?). Don't know if this scenario works well with concurrent edits.

I also like to note, that the location maps are not the only use case. For example look at the category for highway maps. We also had a discussion recently, whether we can create these maps automatically in some way. And again: the technique to make SVG's is not the biggest problem, but to have good geodata as a source.

OSM: is great for large scale street maps and for navigating. But pretty much useless for generalized small scale maps. I tried it once for the highway overview maps with lots of ESRI voodoo and gave up.

Luckally in Germany as in other countries we have better geodata sources from the public authorities meanwhile (also prepared for smaller scales). But we have to reshape them a little bit, bring them up-to-date (cause the data usually is 1-2 years old) and again there is the question where we can store "our own" geodata.

Yurik added a comment.EditedDec 20 2015, 3:43 PM

@Alexrk2, I looked at the spec, and also got scared by this article. Seems that SVG is not well suited for storing geo coordinates. I wonder if the geojson.io simple editor would suffice for most use cases, or if QGIS would work well enough. There is also github's approach - (bottom revision slider button).
At the end of the day, the question is - who should be the "authoritative source"? Should OSM specialize in that, and possibly update their tech for the small-scale maps? Or should we create a centralized storage where volunteers can collaboratively build better world maps? Or should we simply be the centralized storage, with various data sources like governments supplying up-to-date data?

Deskana moved this task from Needs triage to Maps on the Discovery board.Dec 29 2015, 10:24 PM

There is some overlap here with what @aude, @Jdlrobson and myself worked on 2 years ago in zurich, namely "WikiMaps"
I have quickly updated this extension to work again: https://github.com/hartman/WikiMaps and am running it in my (very slow) demo

This extension did the following things:

  1. Define a Map namespace
  2. A json contenthandler to put GeoJSON data in for this map namespace
  3. A view displaying a GeoJSON layer in leaflet (example)
  4. Provide two editors
  5. A <map> tag that can be used to embed this geojson view on other pages (example)

Now it's all very rough etc, but i still believe that it is a good thing if we could have a Data or Map namespace where we curate and edit definitions like these, and then 'ingest' them into our embedded renderings.

Yurik moved this task from Unsorted to General on the Maps (Kartographer) board.Jan 30 2016, 1:48 PM
Yurik moved this task from All map-related tasks to Tracking on the Maps board.Feb 2 2016, 6:02 PM
Yurik moved this task from General to Tracking on the Maps (Kartographer) board.Mar 2 2016, 9:15 PM
Restricted Application added a subscriber: Poyekhali. · View Herald TranscriptJun 16 2016, 10:30 PM

Maps have been enabled on Wikidata. Enjoy :)

Yurik removed a project: Maps.Dec 15 2016, 4:40 AM
Kozuch awarded a token.Jan 9 2017, 6:06 PM