Alert in need of triage: OsmSynchronisationLag (instance maps-test2001:9100)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	LSobanski
	Jul 10 2025, 8:09 AM

Description

The alert OsmSynchronisationLag has started firing 1 month ago.

Labels

alertname=OsmSynchronisationLag
cluster=maps
instance=maps-test2001:9100
job=node
prometheus=ops
severity=critical
site=codfw
source=prometheus
team=sre

Annotations

Name	Content
dashboard	https://grafana.wikimedia.org/d/000000305/maps-performances
description	OSM Synchronisation is behind 66d 8h 8m 41s
runbook	https://wikitech.wikimedia.org/wiki/Maps/Runbook
summary	Maps - OSM synchronization lag - codfw

Links

Triage metadata. Do not delete.

fingerprint=501947a2e1bcbb86

Event Timeline

LSobanski created this task.Jul 10 2025, 8:09 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 10 2025, 8:09 AM

This is on a maps-test server, maybe alert severity should be brought down. Anyhow, tagging Maps project for followup.

Removing serviceops since we won't be working on this.

LSobanski added a project: Infrastructure-Foundations.Thu, Jan 22, 10:44 AM

@MoritzMuehlenhoff is it possible that imposm got into the deadlock bug when we initialized it the last time? From the logs I don't see anything horrible, but it keeps stalling.

In T399158#11544541, @elukey wrote:

@MoritzMuehlenhoff is it possible that imposm got into the deadlock bug when we initialized it the last time? From the logs I don't see anything horrible, but it keeps stalling.

It's doesn't seem deadlocked? Lookig at the logs imposm is up-to-date in what it fetches:

Jan 22 11:02:45 maps-test2001 imposm[175531]: [2026-01-22T11:02:45Z] 1368:25:14 [info] Importing #117124 including changes till 2026-01-22 11:00:00 +0000 UTC (2m45s behind)
Jan 22 11:02:45 maps-test2001 imposm[175531]: [2026-01-22T11:02:45Z] 1368:25:14 [step] Starting: Importing #117124
Jan 22 11:02:45 maps-test2001 imposm[175531]: [2026-01-22T11:02:45Z] 1368:25:14 [step] Starting: Parsing changes, updating cache and removing elements
Jan 22 11:03:45 maps-test2001 imposm[175531]: [2026-01-22T11:03:45Z] 1368:26:14 [progress]   1m0s C:       0/s (4099) N:       0/s (246) W:       0/s (0) R:      0/s (0)
Jan 22 11:04:45 maps-test2001 imposm[175531]: [2026-01-22T11:04:45Z] 1368:27:14 [progress]   2m0s C:       0/s (20264) N:       0/s (669) W:       0/s (0) R:      0/s (0)
Jan 22 11:05:45 maps-test2001 imposm[175531]: [2026-01-22T11:05:45Z] 1368:28:14 [progress]   3m0s C:       0/s (144576) N:       0/s (4254) W:       0/s (303) R:      0/s (0)
Jan 22 11:06:45 maps-test2001 imposm[175531]: [2026-01-22T11:06:45Z] 1368:29:14 [progress]   4m0s C:       0/s (144576) N:       0/s (4254) W:       0/s (2083) R:      0/s (0)
Jan 22 11:07:45 maps-test2001 imposm[175531]: [2026-01-22T11:07:45Z] 1368:30:14 [progress]   5m0s C:       0/s (144576) N:       0/s (4254) W:       0/s (6188) R:      0/s (0)
Jan 22 11:08:45 maps-test2001 imposm[175531]: [2026-01-22T11:08:45Z] 1368:31:14 [progress]   6m0s C:       0/s (144576) N:       0/s (4254) W:       0/s (10765) R:      0/s (0)
Jan 22 11:09:38 maps-test2001 imposm[175531]: [2026-01-22T11:09:38Z] 1368:32:07 [step] Finished: Parsing changes, updating cache and removing elements in 6m53.385773415s

@MoritzMuehlenhoff you are totally right, I got sidetracked by all the zeros, but of course the logs are clearly stating that it is ok. I didn't notice that the task was created months ago, this is now not a problem anymore, sorry for the noise!

Alert in need of triage: OsmSynchronisationLag (instance maps-test2001:9100)Closed, ResolvedPublicActions