Page MenuHomePhabricator

Investigate missing highways
Closed, ResolvedPublic

Description

http://www.openstreetmap.org/way/332692000 is missing from the roads table in the database

Related Objects

StatusAssignedTask
OpenNone
OpenNone
OpenNone
InvalidNone
StalledPnorman
OpenPnorman
DeclinedNone
DeclinedNone
DeclinedNone
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
DuplicateNone
ResolvedPnorman
ResolvedNone
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
Resolveddebt
ResolvedPnorman
ResolvedPnorman
ResolvedNone
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman
DeclinedPnorman
ResolvedPnorman
DeclinedPnorman
ResolvedPnorman
ResolvedPnorman
ResolvedPnorman

Event Timeline

Pnorman created this task.Mar 17 2017, 6:53 PM

Works locally, so I'm going to wait until my test server catches up with diffs and try again

Diffs didn't help, and I found some more cases in Merrit.

Some ways missing are 429682967, 41939409. The last of these looks the best for debugging, being the simplest.

Nothing is obvious on the way that would cause this problem in isolation, but it's been updated recently, with only one change and nothing for a long time before that. Changeset 46886351 removed some tags and changed the position of node 519568901. The changeset made it out in the planet.osm.org minutely replication diff 002359569 (diff, state)

Manually filtering for the involved objects gives the diff

<?xml version='1.0' encoding='UTF-8'?>
<osmChange version="0.6" generator="Osmosis 0.43.1">
  <modify>
    <node id="519568901" version="2" timestamp="2017-03-16T03:04:42Z" uid="355617" user="pnorman" changeset="46886351" lat="50.1054581" lon="-120.7934351"/>
  </modify>
  <modify>
    <way id="41939409" version="4" timestamp="2017-03-16T03:04:42Z" uid="355617" user="pnorman" changeset="46886351">
      <nd ref="519566009"/>
      <nd ref="519567886"/>
      <nd ref="519567978"/>
      <nd ref="600677627"/>
      <nd ref="519568901"/>
      <nd ref="519568767"/>
      <tag k="attribution" v="GeoBase®"/>
      <tag k="geobase:acquisitionTechnique" v="GPS"/>
      <tag k="highway" v="tertiary"/>
      <tag k="is_in" v="Merritt,British Columbia"/>
      <tag k="lanes" v="2"/>
      <tag k="name" v="Voght Street"/>
      <tag k="source" v="Geobase_Import_2009"/>
    </way>
  </modify>
</osmChange>

The next step is to see if I can reproduce it. It's easy enough to find a road somewhere that needs editing for both geometry and tags.

Turns out to be from the update script which saved diffs with a .osm.gz suffix instead of .osc.gz. This caused it to be detected as the wrong file type.

Fixed in https://gist.github.com/pnorman/17b17e2d76505c7365278cbc1c7af80f/revisions#diff-01ef5096d4c7c0fe62a8e38dc274053b on my server, https://github.com/openstreetmap/chef/pull/118 for changes to the OpenStreetMap Chef script mine is based on

Pnorman closed this task as Resolved.Apr 2 2017, 10:33 PM
Pnorman reopened this task as Open.Apr 5 2017, 8:56 PM

I'm seeing this again, so need to do more debugging.

I tested with a pure tag change in diff 2391111 with way 23480609 and it persisted. With a geometry change done by deleting some nodes, moving others, and adding some new ones the object vanished from the rendering tables. This was with way 127018391 which appeared in diff 2391120. Checking the planet.osm.org replication diff against the changes-2391119.osc.gz file found no differences, so it looks like the osmosis --read-replication-interval --write-xml-change file="-" | osmium merge-changes --simplify -F osc --fsync -o "${file}" command line worked.

Still debugging.

So the cause is different, and I've isolated the issue and reported it upstream with https://github.com/openstreetmap/osm2pgsql/issues/735

I'm not sure how to fix it, but will continue tomorrow. My next step is to try osm2pgsql master, osm2pgsql 0.90.0, and if either of those changes the results, 0.92.0 compiled locally instead of packaged.

Pnorman moved this task from Backlog to Done on the Maps-Sprint board.

Fixed upstream, and backported. I believe this will get picked up for debian stretch. If not we can look at if we need to package it ourselves.

Pnorman moved this task from Done to Stalled/Waiting on the Maps-Sprint board.Apr 13 2017, 9:51 PM

Picked up for stretch, built, and updated on my server. I'm asking about jessie-backports.

Pnorman changed the task status from Open to Stalled.Apr 19 2017, 10:31 PM
debt moved this task from Stalled/Waiting to In progress on the Maps-Sprint board.May 30 2017, 7:32 PM
Pnorman closed this task as Resolved.Jun 6 2017, 7:23 PM

Confirming that 0.92.0+ds-2 is in jessie-backports, so this is resolved.