Page MenuHomePhabricator

[Bug] Some OSM relations didn't become polygons and are not been served through geoshapes service
Open, HighPublic

Description

Background

Recently the community have spotted some weird behavior with geoshapes service. Some Wikidata items are not available through the service and some are, see example below:

It seemed that something was wrong with the OSM replication script or the initial import script failed at some point.

Initial import script and Steps to reproduce

The log doesn't show any errors or warnings and they are available at /home/mbsantos/osm-initial-import.log

OSM replication script

The logs doesn't show any errors or warnings and they are available at /var/log/osmosis

Further investigation and steps to reproduce the actual results

With no related track from the logs I started to investigate the data stored in the PostgreSQL DB. I started with the relation Lake Garda (8569) from the example above. The problem is not with the relation, because it is stored in the table planet_osm_rels:

gis=> SELECT id FROM planet_osm_rels WHERE id = 8569;
  id  
------
 8569
(1 row)

But performing a https://github.com/kartotherian/geoshapes/blob/master/geoshapes.js#L64 geoshapes similar query, it returns nothing because the OSM relation didn't become a polygon at osm_planet_polygon:

gis=> SELECT tags->'wikidata' AS id, osm_id
gis-> FROM planet_osm_polygon
gis-> WHERE tags ? 'wikidata' AND tags->'wikidata' IN ('Q6414');                                                             
 id | osm_id
----+--------
(0 rows)

Expected Results

Request existent Wikidata-OSM link and got the GeoJSON through geoshapes service. See the same SQL queries for the Hôtel de Blossac (311766):

gis=> SELECT id FROM planet_osm_rels WHERE id = 3117766;                                                                     
   id
---------
 3117766
(1 row)
gis=> SELECT tags->'wikidata' AS id, osm_id                                                                                  
FROM planet_osm_polygon
WHERE tags ? 'wikidata' AND tags->'wikidata' IN ('Q3145754');                                                                
    id    | osm_id  
----------+---------
 Q3145754 | -311766
(1 row)

Environments Observed

  • All master machines:
    • map1004.eqiad.wmnet
    • map2004.codfw.wmnet

Testing Environment for QA

  • Beta Cluster environment will be set to explore the issue

Additional notes

Some references report a similar issue but the outcome doesn't apply to our infrastructure
https://help.openstreetmap.org/questions/28563/missing-polygons-or-rendering-issue
https://help.openstreetmap.org/questions/68009/osm2pgsql-how-import-relations-into-polygon-table

Details

Related Gerrit Patches:
operations/puppet : productionRe-enable eqiad crons
operations/puppet : productionDisable replicate and admin cron in eqiad

Event Timeline

MSantos created this task.Mar 12 2019, 1:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 12 2019, 1:40 PM
Jhernandez triaged this task as Normal priority.Mar 13 2019, 3:40 PM
Jhernandez raised the priority of this task from Normal to High.
Pikne added a subscriber: Pikne.EditedMar 24 2019, 9:09 AM

An observation related to route relations referred in T214350: none of those added in changeset 63573867 seem to be available as a geoline in Wikimedia system, unless individual member ways of a relation also have Wikidata tag. While relations added a few days before (changeset 63385246) and after (changeset 63605916) seem to be all available. I can't spot any differences in OSM tagging, e.g. geometry for r8821308 is not available via geoline, while r8797335 from another changeset is available.

One particular object (r8821310) from this "bad" changeset was highlighted as external data in Wikipedia article around December when map was added, but stopped displaying sometime in January. Now this route displays again in Wikipedia article and is available via geoline service, but apparently only because a couple of days ago another user added Wikidata tags to individual member ways of given relation.

Edit: I made a dummy edit (didn't change tags/member relations) to r8821320 which wasn't available in Wikimedia system, and now after its latest version is associated to new changeset, it is available via geoline.

It seems that the same problem occurs with geoline service.

Change 522072 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/puppet@production] Disable replicate and admin cron in eqiad

https://gerrit.wikimedia.org/r/522072

Mentioned in SAL (#wikimedia-operations) [2019-07-15T12:35:25Z] <gehel> reimporting OSM data for maps eqiad cluster - T218097

Change 522072 merged by Gehel:
[operations/puppet@production] Disable replicate and admin cron in eqiad

https://gerrit.wikimedia.org/r/522072

Mentioned in SAL (#wikimedia-operations) [2019-07-15T13:16:36Z] <gehel> repooling maps eqiad - T218097

Mentioned in SAL (#wikimedia-operations) [2019-07-17T14:30:59Z] <gehel> repool maps1004 - T218097

Change 524232 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/puppet@production] Re-enable eqiad crons

https://gerrit.wikimedia.org/r/524232

Change 524232 abandoned by Gehel:
Re-enable eqiad crons

Reason:
similar change already merged

https://gerrit.wikimedia.org/r/524232

MSantos closed this task as Resolved.Jul 26 2019, 4:14 PM

Services are stable and the problem is fixed in both clusters.

Haros added a subscriber: Haros.Fri, Nov 1, 1:31 PM

Lake Huron in Canada has the same problem. Has anything been identified on this? It makes all the maps of southern Canada/Great Lakes very disorienting. Probably not relevant but Maggiore and Huron both have intersecting political boundaries running along them. Could that be a causal issue?

@RobinLeicester thanks for the suggestion, I'll consider it while I continue the investigation I started about this OSM replication problem.