ORES articletopic scores pushed into Kafka via T240549: Configure ORES to publish new drafttopic scores to Kafka need to get to HDFS somehow, and new scores for the same page need to override old scores. Presumably EventGate can handle this.
Open questions:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T240517 [EPIC] Growth: Newcomer tasks 1.1.1 (ORES topics) | |||
| Resolved | EBernhardson | T240556 Load ORES articletopic data into ElasticSearch via the weekly bulk update | |||
| Invalid | None | T240553 Consume ORES articletopic data from Kafka and store it in HDFS | |||
| Resolved | Ottomata | T240549 Configure ORES to publish new drafttopic scores to Kafka | |||
| Resolved | Halfak | T240609 Produce drafttopic score events on every edit to English Wikipedia articles |
The relevant conversation happened in T240549: Configure ORES to publish new drafttopic scores to Kafka; I read it as saying that this task is actually a no-op, since the mechanism (which uses EventGate with the mediawiki/revision/score schema and stores data in the event.mediawiki_revision_score hive table) is already score type agnostic and will handle whatever score types ORES outputs when its precache endpoint gets called.
We just deployed a change that will make it so that a mediawiki/revision/score event is produced every time an article is edited. So that should show up in the stream and eventually HDFS.
Do we need a mechanism to get rid of data for deleted pages?
@Tgr and @Halfak , did this question get resolved? If not, I can open a separate ticket for it.
Additionally, @Halfak are you and your team able to do a QA check on the data flow into HDFS, or would you like us to see if we can do it from our side before closing out the ticket?