@Ottomata iirc you mentioned that you've already have an ETL setup that could do this. Is this the case? If yes, could you provide more info or maybe a link? Thanks!

Hmm, I think this?

There is prior art in airflow-dags where people use the Spark Cassandra connector along with stored (and templated) SQL queries to insert data into Cassandra.

• Dags: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/tree/main/main/dags/cassandra_load

• Helper: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/main/config/dag_config.py#L105-126

• Examples of templated job Spark SQL queries: https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/+/refs/heads/master/hql/cassandra/daily/

cc also @JAllemandou

Hm, should we decline or redescribe this task?

Now that the lambda is the source of the topic predictions, perhaps a backfill would simply be a slow iteration over all pages, warming the cache? Similar to Search's 'saneitizer' approach.

We can re-use this task to decide if we want to do backfilling and how to perform it if yes :)

I definitely like the idea of slow iteration over all pages to warm the cache. Another option I see here would be plugging Hoarde to page_change events, which would trigger lambda and backfill naturally over time. If we'd like to do more aggressive backfilling, we would need to decide on who/how/where would run the backfilling process.

I did some load tests against our endpoint and against Hoarde to check what maximum RPS we could expect for backfilling. Even though the latency for a single request is quite high (~400ms p50, 1s p90), our model can handle a lot of concurrent requests allowing us to reach ~100RPS. We'd need to make sure we take advantage of high concurrency in our backfilling process.

plugging Hoarde to page_change events, which would trigger lambda and backfill naturally over time

FWIW, for Edit Suggestions we are proposing to warm LAC (hoarde) via MW JobQueue. Very PoC ideas here, but I'm thinking a MW config that maps from MW DomainEvents (like edits, etc.) to LAC endpoints to call. We could also potentially use the LAC response to produce e.g. mediawiki.page_outlink_topic_prediction_change.v1 events here. This may give us a way to eventually decouple LiftWing from change-prop.

Anyway! Just ideas right now!

The idea of warming up LAC this way via MW JobQueue sounds great to me!

We could also potentially use the LAC response to produce e.g. mediawiki.page_outlink_topic_prediction_change.v1 events here.

Something like this would definitely be a great advantage for us and first step of change-prop decoupling. If we go for the LAC warming approach as you suggested, we'd be processing each event twice: one full inference for LAC warming and another one for producing the prediction events.
You mention it shouldn't live in the EventBus, where do you envision those POC ideas to land eventually?

You mention it shouldn't live in the EventBus, where do you envision those POC ideas to land eventually?

I'm not sure! The LinkedArtifactsCache warming code needs a different home. We could put it in EventBus for now, but it doesn't seem right!

Until we do T432050: EventBus - move event producers to a different extension, producing events from an LAC response is fine in EventBus. In the PoC, there is a LinkedArtifactsWarmedEvent DomainEvent (naming very likely to change!). This can be subscribed to by any MW DomainEvent listener, and filtered by artifact (event) type.

Just created T432733: Server Side Edit Suggestions - Linked Artifacts Cache precompute job