Change #1259966 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] revertrisk-wikidata: add predictions to events stream

https://gerrit.wikimedia.org/r/1259966

@Ottomata Is there any straightforward/existing solution that would allow us to do backfilling once we enable the stream so that teams have past data to analyze?

Hm, straightforward...probably not, but there are surely ways!

We should have mediawiki.page_change.v1 events in Hive in the event_sanitized.mediawiki_page_change_v1 table going back to December 2024. Looks like we've been keeping revert_risk_prediction events since October 2025.

Probably easiest to backfill would be manually using Spark.

• Find all event_sanitized.mediawiki_page_change_v1 records where wiki_id = wikidatawiki
• call the LiftWing endpoint and construct a new DataFrame with data matching event_santized.mediawiki_page_revert_risk_prediction_change_v1.
• Insert into event_santized.mediawiki_page_revert_risk_prediction_change_v1 with proper partition keys.

@AKhatun_WMF recently did something like this for backfilling edit types datasets too, so she might have some advice.

You would have to do manually yes. I am not sure of the cadence, assuming it is hourly or daily, just raw python code with loops are inefficient and prone to failures. I created a devenv airflow instance to run the processing from a backfilling date (Oct 2025 in this case?).

Also mine was just running some SQL with joins. This seems to require some API calling. The devenv airflow instance should be fine, but I'd think of any API limitations you might hit when backfilling. Having a devenv also means you can pause and restart dags later if required (if and when you face API or other limitations).

Change #1259966 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] revertrisk-wikidata: add predictions to events stream

https://gerrit.wikimedia.org/r/1259966

Change #1305890 had a related patch set uploaded (by AikoChou; author: AikoChou):

[operations/deployment-charts@master] ml-services: add revertrisk-wikidata to ml-staging-codfw

https://gerrit.wikimedia.org/r/1305890

Change #1305890 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: add revertrisk-wikidata to ml-staging-codfw

https://gerrit.wikimedia.org/r/1305890

Change #1306630 had a related patch set uploaded (by AikoChou; author: AikoChou):

[operations/deployment-charts@master] changeprop: add liftwing revertrisk-wikidata stream to staging

https://gerrit.wikimedia.org/r/1306630

Change #1306630 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: add liftwing revertrisk-wikidata stream to staging

https://gerrit.wikimedia.org/r/1306630

Change #1307434 had a related patch set uploaded (by AikoChou; author: AikoChou):

[operations/deployment-charts@master] changeprop: add liftwing revertrisk-wikidata to production

https://gerrit.wikimedia.org/r/1307434

Change #1307438 had a related patch set uploaded (by AikoChou; author: AikoChou):

[operations/mediawiki-config@master] EventStreamConfig: add page_revert_risk_wikidata_prediction_change.v1

https://gerrit.wikimedia.org/r/1307438

Change #1307441 had a related patch set uploaded (by AikoChou; author: AikoChou):

[operations/deployment-charts@master] ml-services: enable revertrisk-wikidata event stream predictions

https://gerrit.wikimedia.org/r/1307441

Update:

Tested the event-stream prediction path on ml-staging-codfw with no issues. Production patches are prepared: ml-services / changeprop / EventStreamConfig

Before moving forward, flagging a capacity concern for discussion next week.

Added load from the event stream

Sampled mediawiki.page_change.v1 for wikidata over 5 min: mean ~5 events/s, median 5, peak ~12 events/s.

Current serving state (revertrisk-wikidata, prod)

• Pinned at 2 replicas, 1 MI210 each (minReplicas = maxReplicas = 2), workers: 2.
• Enterprise traffic: ~13 req/s avg, ~19.3 req/s peak (istio-sidecar).
• Latency: p50 400ms / p95 1.5s / p99 2.5s.

Capacity analysis

• With p50 ≈ 400ms as service time, per-worker ceiling ≈ ~2.5 req/s → eqiad's 2 MI210 give a ~10 req/s comfortable aggregate.
• Enterprise peak (~19.3 req/s) is already above that, and p99 = ~6× p50 shows the service is queueing under peak load today.
• Adding the event stream pushes eqiad to ~18 req/s avg / ~31 req/s peak on the same 2 MI210 — a large overcommit. Expect degraded Enterprise tail latency and/or event-consumer lag.
• eqiad MI210 are constrained (4/6 per DC in use), and codfw's idle GPUs don't help since it takes no traffic.

Options to discuss

• Separate isvc for the event path so it doesn't compete with Enterprise serving and can be sized/scaled independently
• Add eqiad capacity via MI300 partitions

Staging is done and healthy. Goal is to agree on capacity before merging the prod patches.

Forgive me if I have forgotten a discussion about this, but!

EventStreamConfig: add page_revert_risk_wikidata_prediction_change.v1

Is there a reason this needs its own special stream? other page change streams have all wikis in the same stream. The task description says:

Produce Wikidata RevertRisk predictions to the mediawiki.page_revert_risk_prediction_change stream,

So, I guess these could just go straight to the same stream?

Is there a reason this needs its own special stream? other page change streams have all wikis in the same stream.

The reason I went with a separate stream is that it keeps the current "one model -> one stream" approach consistent. (We should update the task title of T415892, since it's a bit misleading.)

The current setup is roughly:

• page_revert_risk_prediction_change = Language-agnostic RevertRisk model predictions
• page_revert_risk_multilingual_prediction_change = Multilingual RevertRisk model predictions
• page_revert_risk_wikidata_prediction_change = Wikidata RevertRisk model predictions

The main reason for separating the multilingual model is that both the language-agnostic and multilingual models make predictions for Wikipedia, with some overlapping wikis. If they shared the same stream, consumers could get multiple prediction events for the same revision. We also found that updating the schema to support multiple model predictions per event would be a bit complicated (per T415892#11627398)

Wikidata RevertRisk doesn't have that overlap since it only serves Wikidata, so technically it could share page_revert_risk_prediction_change. However, doing so would blur the current assumption that each model has its own stream (and eventually its own Data Lake table).

Ah! Okay, I did not realize that this was a Wikidata specific model. Interesting! That makes total sense then. Thank you!

Update: proceeding to production.

Following up on the capacity concern in T420883#12086054: I've confirmed Wikimedia Enterprise also consumes mediawiki.page_change events, so the event-stream path and Enterprise requests hit the same Wikidata revisions. With the local cache in place, the added event traffic should largely dedupe against Enterprise (and help warm the cache) rather than doubling GPU load, so I'm moving ahead. I'll watch the cache hit rate and latency after enabling.

Change #1307438 merged by jenkins-bot:

[operations/mediawiki-config@master] EventStreamConfig: add page_revert_risk_wikidata_prediction_change.v1

https://gerrit.wikimedia.org/r/1307438

Mentioned in SAL (#wikimedia-operations) [2026-07-13T13:17:33Z] <aude@deploy2003> Started scap sync-world: Backport for [[gerrit:1310059|Enable ChartWizard on the beta cluster (T431990)]], [[gerrit:1308656|streams: webrequest - pageview - trending (T430675)]], [[gerrit:1307438|EventStreamConfig: add page_revert_risk_wikidata_prediction_change.v1 (T420883)]], [[gerrit:1309894|Remove nonexistent autopatrolled group from Outreach Wiki (T431959)]]

Mentioned in SAL (#wikimedia-operations) [2026-07-13T13:19:14Z] <aude@deploy2003> aikochou, javiermonton, aude, gkm563: Backport for [[gerrit:1310059|Enable ChartWizard on the beta cluster (T431990)]], [[gerrit:1308656|streams: webrequest - pageview - trending (T430675)]], [[gerrit:1307438|EventStreamConfig: add page_revert_risk_wikidata_prediction_change.v1 (T420883)]], [[gerrit:1309894|Remove nonexistent autopatrolled group from Outreach Wiki (T431959)]] synced to the testservers

Mentioned in SAL (#wikimedia-operations) [2026-07-13T13:28:45Z] <aude@deploy2003> Finished scap sync-world: Backport for [[gerrit:1310059|Enable ChartWizard on the beta cluster (T431990)]], [[gerrit:1308656|streams: webrequest - pageview - trending (T430675)]], [[gerrit:1307438|EventStreamConfig: add page_revert_risk_wikidata_prediction_change.v1 (T420883)]], [[gerrit:1309894|Remove nonexistent autopatrolled group from Outreach Wiki (T431959)]] (duration: 11m 12s)

Change #1307441 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: enable revertrisk-wikidata event stream predictions

https://gerrit.wikimedia.org/r/1307441

Change #1307434 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: add liftwing revertrisk-wikidata to production

https://gerrit.wikimedia.org/r/1307434

Change #1310561 had a related patch set uploaded (by AikoChou; author: AikoChou):

[analytics/refinery@master] Add revert_risk_wikidata prediction to event sanitization allowlist

https://gerrit.wikimedia.org/r/1310561

Change #1310570 had a related patch set uploaded (by AikoChou; author: AikoChou):

[operations/deployment-charts@master] EventStreams - Expose mediawiki.page_revert_risk_wikidata_prediction_change.v1 stream

https://gerrit.wikimedia.org/r/1310570

Change #1310570 merged by jenkins-bot:

[operations/deployment-charts@master] EventStreams - Expose mediawiki.page_revert_risk_wikidata_prediction_change.v1 stream

https://gerrit.wikimedia.org/r/1310570

Update

The revertrisk-wikidata prediction stream is now live in production and available via EventStreams:
https://stream.wikimedia.org/v2/ui/#/?streams=mediawiki.page_revert_risk_wikidata_prediction_change.v1

Following up on the capacity concern raised in T420883#12086054: capacity is holding up. We do see a modest increase in latency (p95: ~1.5s → ~2s, p99: ~2.5s → ~3s), but nothing concerning at this point. We'll continue to monitor it. (dashboard)

Separately, we're seeing occasional 500 responses, but these predate this deployment and are not affecting either the prediction stream or Enterprise serving. Based on the logs (P94827), they appear to originate from fetch_labels_from_api() when querying certain Wikidata properties (e.g. P5271 and P4413). I'll file a separate task to investigate this.

The remaining item is the event_sanitized allowlist patch for data retention. @Ottomata, could you take a look when you have a chance? :)

Change #1310561 merged by Ottomata:

[analytics/refinery@master] Add revert_risk_wikidata prediction to event sanitization allowlist

https://gerrit.wikimedia.org/r/1310561

The remaining item is the event_sanitized allowlist patch for data retention

Merged!

Should go out with the next refinery train, probably tomorrow.

Change #1319847 had a related patch set uploaded (by AikoChou; author: AikoChou):

[operations/deployment-charts@master] ml-services: scale revertrisk-wikidata to 3 replicas

https://gerrit.wikimedia.org/r/1319847

Change #1319847 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: scale revertrisk-wikidata to 3 replicas

https://gerrit.wikimedia.org/r/1319847

Update

Enterprise reported a latency regression after the event stream went live on 2026-07-15. Since the event-stream workload and Enterprise traffic share the same deployment, I started investigating a short-term mitigation by increasing revertrisk-wikidata from 2 -> 3 replicas.

I tested the replica increase (2 -> 3) in ml-serve-codfw (has no production traffic) first before rolling it out to ml-serve-eqiad.

The rollout required manual intervention because rolling updates temporarily require both the old and new revisions to coexist. In this case, scaling from 2 -> 3 replicas briefly requires capacity for 5 GPU-backed pods, while only 4 MI210 GPUs are currently available in each cluster. The new revision could therefore not be fully scheduled, and I had to recreate the InferenceService to recover. codfw is now healthy again and running with 3 replicas.

eqiad has the same GPU layout, so I decided not to deploy the replica increase there on Friday, since recovering would require recreating the InferenceService, causing a brief interruption to production traffic.

Current plan for Monday:

1. Reduce the ChangeProp concurrency to lower the incoming load.
2. Scale the service from 2 -> 1 replicas.
3. Scale from 1 -> 3 replicas (avoids the temporary GPU shortage during the rolling update).
4. Monitor latency and service health.

Change #1320762 had a related patch set uploaded (by AikoChou; author: AikoChou):

[operations/deployment-charts@master] changeprop: Reduce revertrisk-wikidata concurrency 5 -> 2

https://gerrit.wikimedia.org/r/1320762

Change #1320762 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: Reduce revertrisk-wikidata concurrency 5 -> 2

https://gerrit.wikimedia.org/r/1320762

Change #1321896 had a related patch set uploaded (by AikoChou; author: AikoChou):

[operations/deployment-charts@master] ml-services: scale revertrisk-wikidata to 1 replica in eqiad

https://gerrit.wikimedia.org/r/1321896

Change #1321899 had a related patch set uploaded (by AikoChou; author: AikoChou):

[operations/deployment-charts@master] ml-services: scale revertrisk-wikidata to 3 replicas

https://gerrit.wikimedia.org/r/1321899

Change #1321896 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: scale revertrisk-wikidata to 1 replica

https://gerrit.wikimedia.org/r/1321896

Change #1321899 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: scale revertrisk-wikidata to 3 replicas

https://gerrit.wikimedia.org/r/1321899