Page MenuHomePhabricator

List of openstreetmap items with Wikidata id
Open, Needs TriagePublic


There should be some way to get information which Wikidata id:s exists in Kartographer (see also T155925). For my purpose (which is using the information to detect which Wikidata pages have OSM wikidata tags) daily CSV export of existing ids to public web server with following information would be enough:

  • Wikidata id (value of the OSM wikidata-tag)
  • OSM relation id
  • OSM name if there is any
  • lat, lon

optionally all other OSM tags would be useful too.

Pulling this kind of information directly from OSM seems to be hard because loading all items with Wikidata ids with Overpass is not possible (least not without multiple queries) because generated dataset is too big.

Event Timeline

Zache created this task.Feb 28 2017, 7:28 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 28 2017, 7:28 AM
JGirault added a subscriber: JGirault.EditedFeb 28 2017, 7:40 PM

I think we're mixing a few concepts here. For clarification purposes:

  • Kartographer is a Mediawiki extension, it does not contain Wikimaps IDs.
  • I've never heard of Wikimaps IDs before. I guess you mean Wikidata IDs.
  • Wikidata pages don't have OSM boundaries (they may have coordinate location though), but OSM objects may contain a wikidata tag containing the corresponding Wikidata ID.
  • As Kartotherian (the server) downloads OSM database, it can easily retrieve OSM objects' boundaries from a Wikidata ID, using OSM objects' wikidata tag.

That said, what are you looking for?

It seems to me that you request an Export API to retrieve all OSM objects (and their OSM tags) that have a Wikidata ID. While Kartotherian could possibly provide something like that, I initially think it goes way beyond its natural purpose and scope.

Generally speaking I'm unclear why you'd make such request to our servers, rather than to OSM directly? Is it because the OSM Overpass API isn't powerful enough for you? If that's the case, I think you should make a feature request to the Overpass developers, or to the broader OSM community, because it looks like what you're really trying to do is querying their database, rather than ours.
If that's not the case and I didn't understand your request, I apologize, and please rephrase it !

Zache updated the task description. (Show Details)Feb 28 2017, 8:34 PM
Zache added a comment.Feb 28 2017, 9:25 PM

Sorry, yes i meant Wikidata id:s. and OSM wikidata tags. I tried to clarify my ticket.

I am trying to do couple of things. One is to do maps/infoboxes in fiwiki which would use wikidata ids (=which means boundaries, lines on the mapframe) if there is matching wikidata-tag at OSM and if there is none then use just coordinates. Second thing is that i would like to get some predictability when people are editing the maps and the infoboxes so that user doesn't need to test if there is OSM wikidata tag or not. Also if there is valid list of the OSM objects which have the wikidata tag then it can be used for the categorization in Wikipedia (eg. these lakes doesn't have yet Wikidata tags). Data from the regularly updated CSV could be pushed to Wikipedia with bot or something like that.

I added ticket to here is that even if there would be some good way to get wikidata tags from OSM there is still couple days delay between OSM edits and Kartotherian updates (am i correct?) So just knowing that there is OSM tag doesn't tell me if it is working in mapframe yet. Second thing is that as far i know there is no good way to track OSM wikidata-tag changes. One solution could be downloading the whole OSM planet data read stuff there but it is huge overkill for this and Kartotherian is doing it already.

Abbe98 added a subscriber: Abbe98.Mar 1 2017, 7:36 PM

I'm interpreting this as two issues:

  1. Automatically check if there is a OSM object connected to the Wikidata item from an specific article and if so is the case use it in a map-frame otherwise just use the coordinate.

I don't think it's possible right now but it would be nice to have Kartotherian exposing such information(lookup capacity) to Lua modules.

  1. Remove the need to check if there is a Wikidata tag in OSM.

This could maybe be solved with a Nominatim instance against the WMF OSM DB, exposed in visual editor as a search field. Allowing users to make text searches for objects with Wikidata IDs. Heavy task anyway.

Is there an Overpass endpoint for the WMF OSM instance?

On the database hosted in our WMF project (osmit) we collect wikidata ids with a VIEW.

The DB which serves Kartographer could host a similar view with the information requested.

I suggest you use a python script to get the data from overpass-turbo like this one (I wil upload the new version today that includes nodes and ways). I use that script to validate that OSM's wikidata IDs match Wikidata instance-of and possibly other properties.

Is there an Overpass endpoint for the WMF OSM instance?

I'm confused what you're asking here. The WMF doesn't have an OSM instance, just a rendering server. Any overpass server loaded with OSM data will work the same, but the WMF doesn't host one. The reasons for running your own overpass server are privacy, load, or SLAs, but that shouldn't stop the development of something which uses overpass.

Just for clarity about what will be in the new rendering database in relation to wikidata

  • There's going to be three tables with items with a wikidata ID
  • The tables will have columns geometry, wikidata ID, and OSM ID
  • Indexes will be present on all three columns, so it will be possible to query on any of them.
  • Having only data with wikidata IDs in it, lookups against the table will be much faster. The tables aren't very big.

I am not sure how much of that Kartotherian will expose.

Yurik added a comment.Mar 15 2017, 4:11 AM

@Pnorman are you adding an index to OSM ID for updates? Also, why an index on geometry? I don't see a usecase for that yet (can always be added later if needed)

The index on OSM id and geometry are standard osm2pgsql indexes

Lydia_Pintscher moved this task from incoming to monitoring on the Wikidata board.May 5 2017, 2:33 PM
mxn added a subscriber: mxn.Aug 10 2017, 8:16 PM
Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptAug 10 2017, 8:16 PM

You could also try querying overpass turbo, or more specifically - the CSV output of that query as described here. Alternatively, you could try querying an experimental service that combines all of Wikidata with all of OSM tags (no geometries).