Page MenuHomePhabricator

cirrus-streaming-updater reports multi-year update lag for events that touch old revisions
Open, Needs TriagePublic

Description

Observed issue

flink_taskmanager_job_task_operator_lastBatchMedianLag on the psi and omega sinks (occasionally chi as well) regularly spikes to impossible values, from months to ~20 years. The spikes last for a single scrape

Several appear in eqiad and codfw in the same minute, which points to shared input events rather than a cluster-specific problem
Some examples from the last 7 days:

UTCsinkreported lag
2026-08-27 02:19psi0.8 y
2026-08-29 00:34chi3.6 y
2026-08-29 08:11omega5 y
2026-08-31 06:57psi10-11 y
2026-09-01 06:38-06:57psi4-20 y (6 spikes)

Over 30 days: psi ~35 spike-hours per site, omega ~7, chi 1.

Thanos: max by (site, operator_name) (flink_taskmanager_job_task_operator_lastBatchMedianLag{job_name=~"cirrus_streaming_updater_consumer_search_.*",prometheus="k8s"} / 60000) > 10000

Why this happens

The lag gauge reports now - median(event_time) over the last bulk batch. For most revision-based updates, the producer sets event_time from revision.rev_dt (the MediaWiki rev_timestamp)

This works for an ordinary edit, where the revision and event are both new. When a current event refers to an old revision, however, the gauge reports the age of the revision as update lag

The producer already avoids this for delete and undelete events, which use the event's dt because "these events operate on pre-existing revisions, the rev_dt is in the past". Other non-delete events still use rev_dt. Possible sources include a visibility_change on an old revision or a revision imported with its original timestamp, but we have not confirmed which event types produced the observed spikes.

Same class of bug as T387038, where weighted-tag backfills carried old dt values and the fix was to report lag from meta.dt for TAGS_UPDATE.

Impact

The search update-lag SLO is avg by (site, job_name) of this metric < 600s, so every scrape holding one of these values is an SLI failure regardless of aggregation.
Currently this accounts for ~1-1.5 h/month per site out of a 36 h budget. It doesn't fire on its own, but it adds some background noise to the SLO

Note this issue almost certainly manifests more readily in the smaller clusters because they process smaller batches and have more time between batches. Small batches would pretty directly make it easier for a few old timestamps to move the median considerably

Proposal

  • Extend the T387038 approach to REV_BASED_UPDATE: report lag from meta.dt (when the event platform accepted the event) instead of the event time. Leave the event time itself alone because it is also the ordering key for merging colliding updates
  • The SLO should measure delay in the current update path, not the age of the affected revision. Using meta.dt measures from event platform receipt through the producer, Kafka handoff, consumer, and OpenSearch write, while removing revision age from the measurement. It also matches the existing handling for TAGS_UPDATE and fixes the whole class of old-revision events rather than only the possible sources identified so far

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Measure page-change lag from receipt timerepos/search-platform/cirrus-streaming-updater!212ryankemperT436736-lag-after-mergemain
Measure sink lag from meta.dt for all update typesrepos/search-platform/cirrus-streaming-updater!210ryankemperT436736-lag-from-meta-dtmain
Customize query in GitLab

Event Timeline

RKemper updated the task description. (Show Details)

@dcausse I uploaded a reference patch, lmk if it looks sane

@RKemper thanks for digging into this!
The update lag has been designed to ideally capture the time between a user making an edit and the time the update is received by the backend. Unfortunately this does not seem to work the way we wanted, it seems like we can't trust the page-change event-time.
Checking an example I found that page imports are indeed to blame (example: https://bol.wikipedia.org/w/index.php?title=Inmusamman:Log&page=MediaWiki+mossa%3AProtectedpagetext) in this case the system is setting the revision time to the event time, I could argue that it's not really correct. In case of imports the event-time should be the time of the import and not the time of the imported revision.
@Ottomata what do you think, should page-change events in case of imports record their event-times as the time of the import instead of the revision time? Or possibly flag that the event is coming from an import so that client can be aware that the timestamps seen there might not reflect the time a user had interacted with mediawiki?

In the meantime I think we could indeed try to workaround this issue by using meta.dt, I'll review the patch you uploaded.

Change #1334719 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] cirrus-streaming-updater: bump to latest image

https://gerrit.wikimedia.org/r/1334719

@dcausse I reverted the first fix before deployment, turns out using receipt time for merge ordering could let a delayed edit displace a delete of the same revision

I've changed the approach to set the outgoing timestamp after deduplication in SUP !212. Since the schema docs will now be out of date, I uploaded Schema !82 to document the new behavior

@dcausse I reverted the first fix before deployment, turns out using receipt time for merge ordering could let a delayed edit displace a delete of the same revision

I agree that there'll be a slight behavior change in the merge ordering but I'm not sure it'll be as bad as it looks.
Events gets aggregated with a 10 min window, this window is based on org.wikimedia.eventutilities.flink.stream.EventTimeAssigner (which is no longer in line with the event-time set in the patch you reverted by assigning meta.dt there, I'm still wrapping my head around how bad that is).
Then the window will get dedupped using the UpdateEvent event-time (which is now meta.dt for page changes) using org.wikimedia.discovery.cirrus.updater.producer.graph.UpdateEventMerger.

Trying to simulate what this actually changes I tried to simulate the grouping in hive I find (for the 8 days of September and excluding undelete for now) only two instances where the two approaches actually disagrees:

+-------+--------+----------+--------------------+----------------------+--------------------------------------------+----------------------------------------------------+
|wiki_id|page_id |rev_id    |prior_logic_op_order|meta_dt_logic_op_order|prior_logic_ts                              |meta_dt_logic_ts                                    |
+-------+--------+----------+--------------------+----------------------+--------------------------------------------+----------------------------------------------------+
|enwiki |84164610|1373665566|[edit, delete]      |[delete, edit]        |[2026-09-07T05:01:39Z, 2026-09-07T05:01:40Z]|[2026-09-07T05:01:41.012Z, 2026-09-07T05:01:41.042Z]|
|lawiki |328261  |3985893   |[delete, edit]      |[edit, delete]        |[2026-09-05T07:45:56Z, 2026-09-05T07:45:56Z]|[2026-09-05T07:45:56.753Z, 2026-09-05T07:45:57.102Z]|
+-------+--------+----------+--------------------+----------------------+--------------------------------------------+----------------------------------------------------+

Both pages are actually deleted so neither approach is actually good, on one time the top-level dt was right (enwiki) for the other meta.dt was correct. What's important is that most of the time they agree.
I suspect that the root cause is that neither timestamps is actually a strong guarantees of the ordering of operation in the database.

If we take a look at undeletes (which is known to be problematic because a chain of delete/undelete/delete stays on the same revision) we jump to 550 revisions (which remains low in the grand scheme of things).
Also note that because it's running via spark it does not necessarily mean that those would have been part of the same 10mins window:

+------------+---------+----------+------------------------------------+------------------------------------+----------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
|wiki_id     |page_id  |rev_id    |prior_logic_op_order                |meta_dt_logic_op_order              |prior_logic_ts                                                                          |meta_dt_logic_ts                                                                                        |
+------------+---------+----------+------------------------------------+------------------------------------+----------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
|enwiki      |84131686 |1372978465|[edit, undelete, delete]            |[edit, delete, undelete]            |[2026-09-03T08:30:24Z, 2026-09-03T08:30:24Z, 2026-09-03T09:26:17Z]                      |[2026-09-03T08:30:25.928Z, 2026-09-03T09:26:19.188Z, 2026-09-03T09:26:38.121Z]                          |
|enwiktionary|12030069 |92391802  |[undelete, delete]                  |[delete, undelete]                  |[2026-08-30T06:56:54Z, 2026-09-04T09:06:40Z]                                            |[2026-09-04T09:06:40.826Z, 2026-09-04T11:18:09.121Z]                                                    |
|enwiktionary|12030180 |92392171  |[undelete, delete]                  |[delete, undelete]                  |[2026-08-30T08:00:46Z, 2026-09-04T09:06:40Z]                                            |[2026-09-04T09:06:40.793Z, 2026-09-04T11:19:12.058Z]                                                    |
|zhwiki      |9600318  |94216299  |[edit, undelete, delete]            |[edit, delete, undelete]            |[2026-09-06T09:56:30Z, 2026-09-06T09:56:30Z, 2026-09-06T10:30:53Z]                      |[2026-09-06T09:56:31.254Z, 2026-09-06T10:30:54.422Z, 2026-09-06T10:31:21.551Z]                          |
|commonswiki |196788280|1256604822|[undelete, undelete, delete, delete]|[delete, undelete, delete, undelete]|[2026-08-04T19:00:50Z, 2026-08-04T19:00:50Z, 2026-09-04T03:42:18Z, 2026-09-05T03:59:33Z]|[2026-09-04T03:42:19.976Z, 2026-09-04T09:00:15.481Z, 2026-09-05T03:59:34.852Z, 2026-09-05T08:17:45.759Z]|
|commonswiki |163681245|1272447270|[move, undelete, delete, delete]    |[move, delete, undelete, delete]    |[2026-09-05T06:00:27Z, 2026-09-05T06:00:27Z, 2026-09-05T06:00:30Z, 2026-09-05T06:00:47Z]|[2026-09-05T06:00:30.977Z, 2026-09-05T06:00:31.747Z, 2026-09-05T06:00:35.961Z, 2026-09-05T06:00:49.824Z]|
|enwiktionary|12030004 |92391617  |[undelete, delete]                  |[delete, undelete]                  |[2026-08-30T06:30:39Z, 2026-09-04T09:06:40Z]                                            |[2026-09-04T09:06:40.998Z, 2026-09-04T11:15:55.177Z]                                                    |
|enwiki      |63408715 |1373725348|[edit, undelete, delete]            |[edit, delete, undelete]            |[2026-09-07T16:05:21Z, 2026-09-07T16:05:21Z, 2026-09-07T16:10:58Z]                      |[2026-09-07T16:05:25.990Z, 2026-09-07T16:11:03.098Z, 2026-09-07T16:11:39.259Z]                          |
|zhwiki      |9599224  |94202906  |[edit, undelete, delete, delete]    |[edit, delete, undelete, delete]    |[2026-09-05T10:27:08Z, 2026-09-05T10:27:08Z, 2026-09-05T11:15:39Z, 2026-09-05T11:19:54Z]|[2026-09-05T10:27:10.216Z, 2026-09-05T11:15:40.641Z, 2026-09-05T11:19:13.949Z, 2026-09-05T11:19:54.844Z]|
|commonswiki |198060043|1267794403|[undelete, delete]                  |[delete, undelete]                  |[2026-08-26T17:23:39Z, 2026-09-01T21:18:06Z]                                            |[2026-09-01T21:18:07.732Z, 2026-09-01T21:19:54.482Z]                                                    |
+------------+---------+----------+------------------------------------+------------------------------------+----------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+

The first thing that stands out is that using the prior logic the operation order cannot be right, an undelete cannot happen after an edit, the meta.dt ordering looks more sane.
Reason is probably because undeletes were still assigning rev_dt instead of dt the same way it was fixed in https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/commit/afc4c1caee35a2b51ce17017afc1d86c700a4783

If we compare pure event time and meta.dt we're down to zero disagreement between meta.dt and dt orderings.

Overall I think the patch you reverted had actually a nice side-effect is that it fixed the handling of undeletes.

Given that neither dt nor meta.dt provide a strong guarantee of the actual order of operation in the db I'd be leaning toward keeping your first approach.

Regarding the discrepancy that was introduced in EventTimeAssigner I'd be tempted to ignore it for now. To the best of my knowledge this might cause imports to be out of the window and probably bypass the window because of the late-event filter (LateEventProcessFunction) and thus be processed first. Given that imports are generally done prior any other changes to the imported revisions I think this is more or less correct.

I think all this is clearly not ideal and page-change should provide better ways to infer the order of operations but given all these limitations the first version you uploaded is a reasonable workaround.

If you want to reproduce the tables please see the notebook at stat1008.eqiad.wmnet:~dcausse/notebooks/page-change-ordering.ipynb.

I might have missed something so please let me know if you disagree.

@dcausse Sorry for taking so long on this. I took a look; the notebook uses revision time for undeletes, but Erik's fix already made those use the event's dt (the same existing behavior I overlooked in my earlier patch). Does that change your leaning?

@dcausse Sorry for taking so long on this. I took a look; the notebook uses revision time for undeletes, but Erik's fix already made those use the event's dt (the same existing behavior I overlooked in my earlier patch). Does that change your leaning?

Oops my bad, I can re-run the extraction with this logic in mind, this should remove the noise regarding undeletes.
I doubt it'll change the conclusion of this, in that either dt or meta.dt for the dedup/merge window is fine (both can be wrong).

@dcausse Sorry for taking so long on this. I took a look; the notebook uses revision time for undeletes, but Erik's fix already made those use the event's dt (the same existing behavior I overlooked in my earlier patch). Does that change your leaning?

Oops my bad, I can re-run the extraction with this logic in mind, this should remove the noise regarding undeletes.
I doubt it'll change the conclusion of this, in that either dt or meta.dt for the dedup/merge window is fine (both can be wrong).

Re-ran the job on all events in September. over the 46M events, only 10 cases see a disagreement between event_time and meta.dt

+------------+---------+----------+------------------------------+----------------------+--------------------------------------------+----------------------------------------------------+
|wiki_id     |page_id  |rev_id    |pure_event_time_logic_op_order|meta_dt_logic_op_order|pure_event_time_logic_ts                    |meta_dt_logic_ts                                    |
+------------+---------+----------+------------------------------+----------------------+--------------------------------------------+----------------------------------------------------+
|enwiki      |84164610 |1373665566|[edit, delete]                |[delete, edit]        |[2026-09-07T05:01:39Z, 2026-09-07T05:01:40Z]|[2026-09-07T05:01:41.012Z, 2026-09-07T05:01:41.042Z]|
|enwiktionary|12212443 |92929888  |[create, delete]              |[delete, create]      |[2026-09-25T14:53:25Z, 2026-09-25T14:53:26Z]|[2026-09-25T14:53:36.485Z, 2026-09-25T14:53:40.931Z]|
|sqwiki      |406822   |3048629   |[create, delete]              |[delete, create]      |[2026-09-24T00:21:39Z, 2026-09-24T00:21:54Z]|[2026-09-24T00:21:54.931Z, 2026-09-24T00:21:59.013Z]|
|eswiki      |11599521 |175382824 |[delete, edit]                |[edit, delete]        |[2026-09-16T09:19:31Z, 2026-09-16T09:19:31Z]|[2026-09-16T09:19:31.726Z, 2026-09-16T09:19:31.918Z]|
|lawiki      |328261   |3985893   |[delete, edit]                |[edit, delete]        |[2026-09-05T07:45:56Z, 2026-09-05T07:45:56Z]|[2026-09-05T07:45:56.753Z, 2026-09-05T07:45:57.102Z]|
|ruwiki      |11616011 |155290432 |[create, delete]              |[delete, create]      |[2026-09-26T01:07:56Z, 2026-09-26T01:08:04Z]|[2026-09-26T01:08:04.653Z, 2026-09-26T01:08:05.208Z]|
|trwiki      |4747504  |37958388  |[create, delete]              |[delete, create]      |[2026-09-14T14:50:44Z, 2026-09-14T14:52:38Z]|[2026-09-14T14:52:39.837Z, 2026-09-14T14:53:58.191Z]|
|sqwiki      |406823   |3048631   |[create, delete]              |[delete, create]      |[2026-09-24T00:21:40Z, 2026-09-24T00:21:54Z]|[2026-09-24T00:21:54.986Z, 2026-09-24T00:21:59.409Z]|
|commonswiki |200199838|1280304605|[delete, edit]                |[edit, delete]        |[2026-09-25T08:19:25Z, 2026-09-25T08:19:25Z]|[2026-09-25T08:19:25.850Z, 2026-09-25T08:19:27.834Z]|
|ptwiki      |7966512  |73049518  |[create, delete]              |[delete, create]      |[2026-09-25T06:28:17Z, 2026-09-25T06:28:25Z]|[2026-09-25T06:28:25.995Z, 2026-09-25T06:28:26.341Z]|
+------------+---------+----------+------------------------------+----------------------+--------------------------------------------+----------------------------------------------------+
  • 7 cases where the event_time holds the true ordering
  • 3 cases where it's meta.dt

10 cases out of 46M events is probably not enough to draw a conclusion but indeed event time seems to be slightly better. I don't know if this justifies adding another field just to track the pipeline lag... I think I still prefer your first approach of using meta.dt as the pipeline event-time even though it might not be accurate in some cases. @RKemper would you have objections restoring your first implementation? Anything obvious I might be missing?

Practically speaking what does it mean:

  • we may ship an create event while the page should be deleted
  • it might not be not too bad because the window will wait and the time the content will be fetched from the MW endpoint the page will be marked as deleted and thus the event will be discarded
  • worst-case we fetch the content out of a lagged mariadb replica, we will index a page that supposed to be deleted, at search time we hide missing revisions (the deleted page can't be shown)
  • the consistency check process with eventually cleanup this page