Page MenuHomePhabricator

Wikimedia Hackathon 2026: Geographic Wikidata dumps
Closed, ResolvedPublic

Assigned To
Authored By
Danysan1
Apr 29 2026, 3:52 PM
Referenced Files
F79448756: image.png
May 3 2026, 1:00 PM
F79448165: image.png
May 3 2026, 1:00 PM
F79448144: image.png
May 3 2026, 1:00 PM

Description

Currently extracting geographic features from Wikidata requires an inefficient filtering through the already overburdened WDQS or downloading and handling the enormous dumps.
During the 2026 hackathon I want to create a Toolforge tool to regularly extract geographical features from the shared storage dumps and allow to download them from the web in common geographic formats (I was thinking FlatGeobuf and GeoParquet but other formats could be good candidates as well).

The critera for inclusion of items in these dumps will be

  • absence of any P585 or P376 statements
  • presence of a P625 statement
  • absence in the P625 statement of any P582 qualifier or presence with a value later than the current time
  • absence of any P580, P571 or P1619 statement or presence of them with a value earlier than the current time
  • absence of any P582, P576 or P3999 statement or presence of them with a value later than the current time

If I will find enough time I will update the initialization data pipeline of OSM-Wikidata Map Framework to use and test these geographic dumps.

Event Timeline

As source for the dumps I will use /public/dumps/public/wikidatawiki/entities/ which contains the same files listed in https://dumps.wikimedia.org/wikidatawiki/entities/ .
I will get the last update date with date -r /public/dumps/public/wikidatawiki/entities/latest-all.json.gz "+%Y-%m-%dT%H:%M:%S.000Z".

I generated a dump sample for quicker development with these commands:

cat /public/dumps/public/wikidatawiki/entities/latest-all.json.gz | gzip -d | head -10000 > example.json
echo ']' >> example.json

As a future evolution, we could use the geoshape (P3896) as the priority geometry, using P625 only as a fallback

The first implementation of the pipeline and portal is complete , for now only with a test filtered from only the first 100'000 lines of the Wikidata dump:

https://wikidata-geo-dumps.toolforge.org/

image.png (1,242×690 px, 85 KB)

https://wikidata-geo-dumps.toolforge.org/dumps/2026-04-29-test/

image.png (1,175×272 px, 11 KB)

https://wikidata-geo-dumps.toolforge.org/dumps/2026-04-29-test/places.fgb visualized in QGIS:

image.png (1,918×1,020 px, 605 KB)

Next steps:

  • Try a filtering on the full dump and face the obvious scaling problems that will pop up
  • Include all localized labels in the output
  • Allow P580, P571, P1619 (start dates) with values in the past
  • Allow P582, P576, P3999 (end dates) with values in the future
  • Check that no P582 qualifier is present in the P625 claim
  • Allow P3896 (geoshape) alternatively to P625
  • Handle file rotation to prevent wasting storage

The site is now up and running and serving geo dumps for the full DB. There are still some bugs and improvements needed but I'm opening separate tasks for them in Tool-wikidata-geo-dumps