Page MenuHomePhabricator

Monitor age of OSM data
Closed, ResolvedPublic

Description

We should have a way of monitoring the age of OSM data. This will show problems with consuming updates. An example graph of this is on OpenStreetMap munin

pasted_file (305×547 px, 29 KB)

Although strictly speaking the lag is the difference between the local data timestamp and the planet.osm.org timestamp, it's common to just assume the planet.osm.org timestamp is the current time, avoiding the need to request it while monitoring. This has a maximum error of one minute, unless there are upstream problems.

The common way to do this is with https://github.com/openstreetmap/mod_tile/blob/master/munin/replication_delay. It wouldn't port directly to our monitoring, but there's nothing complicated there.

Event Timeline

Change 346106 had a related patch set uploaded (by Alexandros Kosiaris; owner: Gehel):
[operations/puppet@production] maps - collect OSM sync lag to prometheus

https://gerrit.wikimedia.org/r/346106

Change 346106 merged by Gehel:
[operations/puppet@production] maps - collect OSM sync lag to prometheus

https://gerrit.wikimedia.org/r/346106

debt triaged this task as Medium priority.Jun 9 2017, 7:44 PM
debt subscribed.

@Gehel - can you confirm that this has been done and merged or not done, please?

We have monitoring in place for OSM replication: eqiad, codfw. We should also have alerting at some point, but that's another ticket.

debt claimed this task.
debt moved this task from Backlog to Done on the Maps-Sprint board.

Perfect, thanks for the info. Closing this ticket as resolved.