Observed issue
flink_taskmanager_job_task_operator_lastBatchMedianLag on the psi and omega sinks (occasionally chi as well) regularly spikes to impossible values, from months to ~20 years. The spikes last for a single scrape
Several appear in eqiad and codfw in the same minute, which points to shared input events rather than a cluster-specific problem
Some examples from the last 7 days:
| UTC | sink | reported lag |
| 2026-08-27 02:19 | psi | 0.8 y |
| 2026-08-29 00:34 | chi | 3.6 y |
| 2026-08-29 08:11 | omega | 5 y |
| 2026-08-31 06:57 | psi | 10-11 y |
| 2026-09-01 06:38-06:57 | psi | 4-20 y (6 spikes) |
Over 30 days: psi ~35 spike-hours per site, omega ~7, chi 1.
Thanos: max by (site, operator_name) (flink_taskmanager_job_task_operator_lastBatchMedianLag{job_name=~"cirrus_streaming_updater_consumer_search_.*",prometheus="k8s"} / 60000) > 10000
Why this happens
The lag gauge reports now - median(event_time) over the last bulk batch. For most revision-based updates, the producer sets event_time from revision.rev_dt (the MediaWiki rev_timestamp)
This works for an ordinary edit, where the revision and event are both new. When a current event refers to an old revision, however, the gauge reports the age of the revision as update lag
The producer already avoids this for delete and undelete events, which use the event's dt because "these events operate on pre-existing revisions, the rev_dt is in the past". Other non-delete events still use rev_dt. Possible sources include a visibility_change on an old revision or a revision imported with its original timestamp, but we have not confirmed which event types produced the observed spikes.
Same class of bug as T387038, where weighted-tag backfills carried old dt values and the fix was to report lag from meta.dt for TAGS_UPDATE.
Impact
The search update-lag SLO is avg by (site, job_name) of this metric < 600s, so every scrape holding one of these values is an SLI failure regardless of aggregation.
Currently this accounts for ~1-1.5 h/month per site out of a 36 h budget. It doesn't fire on its own, but it adds some background noise to the SLO
Note this issue almost certainly manifests more readily in the smaller clusters because they process smaller batches and have more time between batches. Small batches would pretty directly make it easier for a few old timestamps to move the median considerably
Proposal
- Extend the T387038 approach to REV_BASED_UPDATE: report lag from meta.dt (when the event platform accepted the event) instead of the event time. Leave the event time itself alone because it is also the ordering key for merging colliding updates
- The SLO should measure delay in the current update path, not the age of the affected revision. Using meta.dt measures from event platform receipt through the producer, Kafka handoff, consumer, and OpenSearch write, while removing revision age from the measurement. It also matches the existing handling for TAGS_UPDATE and fixes the whole class of old-revision events rather than only the possible sources identified so far