From a quick look at the data this is a key=>value on hstore column "tags"

Change 868415 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[operations/puppet@production] Exclude OSM tag that causes a failing import

https://gerrit.wikimedia.org/r/868415

Change 874806 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[operations/puppet@production] maps: Disable osm sync temporarily due to wrong data

https://gerrit.wikimedia.org/r/874806

I think at this point these are the ways forward I can think of:

• Full planet import
  • Codfw full planet import is pending so this is not going to be a problem for the one cluster
• Manually editing https://planet.openstreetmap.org/replication/hour/000/089/845.osc.gz
  • The problematic feature is deleted after so this is theoretically a relatively easy solution

For the manual intervention this involves:

• Downloading the file
• Edit the problematic lines: https://phabricator.wikimedia.org/P42743
• Apply using imposm diff --config <path> edited-diff.osc.gz

Change 874806 merged by Alexandros Kosiaris:

[operations/puppet@production] maps: Disable osm sync temporarily due to wrong data

https://gerrit.wikimedia.org/r/874806

Change 868415 abandoned by Jgiannelos:

[operations/puppet@production] Exclude OSM tag that causes a failing import

Reason:

Fix requires full planet import

https://gerrit.wikimedia.org/r/868415

Was this problem ever relayed to OSM btw ??? I mean this cannot have just been us that ran into this problem right ?

What would be the right channel to communicate this issue ?

Imposm should be already tested against the invalid layer case
https://github.com/omniscale/imposm3/blob/master/test/completedb_test.go#L171

I don't think excluding the tag is the solution...

Imposm should be already tested against the invalid layer case

Maybe we are on an older imposm version ? Related code for that test case is https://github.com/omniscale/imposm3/commit/53bb80726ca9456e4a0857b38803f9ccfe8e33fd which is about 2 years old.

OSM operational issues should go to:
https://operations.osmfoundation.org/
https://github.com/openstreetmap/operations/issues
Which we probably should document somewhere for any future problem reporting.

This is our apt package for imposm:

imposm3:
  Installed: 0.11.0+git20201104.4758cf4-1
  Candidate: 0.11.0+git20201104.4758cf4-1
  Version table:
 *** 0.11.0+git20201104.4758cf4-1 1001
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status

The fix is introduced later:

git --no-pager log 4758cf4..v0.11.1
commit 53bb80726ca9456e4a0857b38803f9ccfe8e33fd (tag: v0.11.1)
Author: Oliver Tonnhofer <olt@bogosoft.com>
Date:   Mon Jan 18 12:03:19 2021 +0100

    mapping: fix (way)zorder for non-int32 layer values

Upgrading the package will also do the trick.

I will defer to serviceops now that we have a better understanding of the problem and some potential solutions

I will explore our option to upgrade to 0.11.1 and get back to you

Mentioned in SAL (#wikimedia-operations) [2023-01-04T09:53:12Z] <effie> Upload imposm3_0.11.1-1 to buster-wikimedia - T325293

Change 875273 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] maps: enable OSM sync after imposm3 upgrade

https://gerrit.wikimedia.org/r/875273

Change 875273 merged by Effie Mouzeli:

[operations/puppet@production] maps: enable OSM sync on eqiad after imposm3 upgrade

https://gerrit.wikimedia.org/r/875273

Maybe we should add imposm to release monitoring ?
https://phabricator.wikimedia.org/diffusion/LLIC/browse/master/monitoring.json

T326230

It looks like OSM syncing is catching up with all the old diffs on eqiad after bumping imposm version.

eqiad is in sync now.

This is from the last log entries:

Jan 06 10:03:04 maps1009 imposm[14255]: [2023-01-06T10:03:04Z] 47:40:18 [info] Importing #90435 including changes till 2023-01-06 10:00:00 +0000 UTC (3m4s behind)

Things look OK

Will codfw follow, too? It seems that it's still 3 weeks +x behind...

There is a full planet import that is pending: https://phabricator.wikimedia.org/T314472