Support Data:*.map in Graphs
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Yurik
	Dec 19 2016, 4:57 AM

Description

Graphs are able to access .tab pages, but there is no easy/good way to get geometries. Additionally, Vega2 only allows TopoJSON, which it internally converts to GeoJSON for processing (TopoJSON has ~4x space saving for transmission). The problem is, we already use the less efficient GeoJSON, which means that graphs are not able to consume them directly.

I see several solutions:

Encode GeoJSON to TopoJSON in PHP as part of mwapi.
- PROs: direct db access, allows to eventually store data as TopoJSON
- CONs: more like a blocker - there is no PHP TopoJSON encoder, only decoder
Do all TopoJSON processing in Kartotherian
- PROs: established JS libraries
- CONs: Kartotherian becomes a proxy, converting mwapi response into a TopoJSON
Patch Vega to allow direct GeoJSON consumption
- PROs: Client-side only code, consist backend for all features
- CONs: We continue wasting 4x bandwidth for the map data transmission, possible Vega fork

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Bumped shared lib - map: protocol support	mediawiki/extensions/Graph	wmf/1.29.0-wmf.7	+33 -8
	Bumped shared lib - map: protocol support	mediawiki/extensions/Graph	master	+33 -8

Customize query in gerrit

Event Timeline

Yurik created this task.Dec 19 2016, 4:57 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 19 2016, 4:57 AM

Yurik updated the task description. (Show Details)Dec 19 2016, 5:00 AM

I spoke with Vega ppl, it seems that GeoJSON can be consumed directly by specifying json instead of topojson. Support added at https://github.com/nyurik/graph-shared/commit/57974ef1054aae332721b3eb5bda10f5ff67b923 - will merge/deploy to graphoid in January.

Change 328328 had a related patch set uploaded (by Yurik):
Bumped shared lib - map: protocol support

https://gerrit.wikimedia.org/r/328328

gerritbot added a project: Patch-For-Review.Dec 20 2016, 2:59 AM

This now works in Graph Sandbox on data.wmflabs.org:

{
  "version": 2,
  "width": 740,
  "height": 500,
  "padding": 0,
  "data": [{
      "name": "world",
      "url": "map:///Naturalearthdata.com/US states with PR.map",
      "format": {"type": "json", "property": "data.features"},
      "transform": [{
        "type": "geopath",
        "projection": "albersUsa",
        "scale": 500,
        "translate": [350, 250]
      }]
    }
  ],
  "marks": [
    {
      "type": "path",
      "from": {"data": "world"},
      "properties": {
        "enter": {
          "stroke": {"value": "#fff"},
          "path": {"field": "layout_path"},
          "fill": {"value": "#ccc"}
        }
      }
    }
  ]
}

Technically wikiraw: already works - the above example would work with this url, but it would be a cleaner workaround and with localization support.
"url": "wikiraw://commons.wikimedia.org/Data:Naturalearthdata.com/US states with PR.map",

Or a different workaround - this time with localization (still hacky, but alas) - via jsondata api

"url": "wikiapi:///?format=json&formatversion=2&action=jsondata&title=Naturalearthdata.com/US_states_with_PR.map&uselang=en",
"format": {"type": "json", "property": "jsondata.data.features"},

Change 330277 had a related patch set uploaded (by Yurik):
Bumped shared lib - map: protocol support

https://gerrit.wikimedia.org/r/330277

Change 328328 merged by jenkins-bot:
Bumped shared lib - map: protocol support

https://gerrit.wikimedia.org/r/328328

Change 330277 merged by jenkins-bot:
Bumped shared lib - map: protocol support

https://gerrit.wikimedia.org/r/330277

Yurik moved this task from Needs review to Done on the Maps-Sprint board.Jan 3 2017, 11:10 PM

• debt closed this task as Resolved.May 30 2017, 7:24 PM

• debt claimed this task.