As per conversation on Slack, @JAllemandou will send some sample data to unblock work on this task. We still need T299059 in order to complete this task, but the sample data will get us started.

Here is some data:

hdfs dfs -du -s -h hdfs:///wmf/data/wmf/structured_data/commons/entity/snapshot=2022-01-24
32.8G

hive>select count(1) from structured_data.commons_entity where snapshot='2022-01-24';
72659834

Let me know if it doesn't work as expected :)

gather all wikidata-ids from all commons depicts and is digital representation of statements

Should that include P921 "main subject"? I think it's somewhat common to use it as a stronger version of "depicts".

Ok this is pretty much done. We're writing to Hive instead of hdfs, to make it easier to export to Cassandra

Some observations/comparisons

Data for ptwiki running the old IMA on the 2022-02-21 snapshot:

• 21092 suggested images for 106699 articles (387282 unillustrated articles in total)

Data for ptwiki using the new data pipeline on the 2022-02-21 snapshot:

• 1518613 suggestions for 126259 articles
• including suggestions for 81177 of the same articles as the old IMA

We're expecting more suggestions for the new pipeline as we're not limiting the number of suggestions for a single article, and we're expecting suggestions for more articles because we're also using depicts data ... so the above is within the boundaries of what we'd expect

Note that it's difficult to compare the output of the 2 directly, as we've discovered a flaw in some of the underlying library code the original algorithm uses which causes a bunch of data to be dropped. Working around that atm, will do some more data comparisons before closing

technical note: storing to hive is actually storing to HDFS :) Hive is a SQL engine on top of HDFS structured data.

Ok ran the old IMA again but with the data-loss part fixed (I think!), and now from it we're getting suggestions for 82399 articles, 81177 of which are also suggested by the new pipeline

We'd expect all suggestions from the old IMA to also be suggested by the new pipeline.

All the suggestions (or at least the ones I've checked) that are not in the new pipeline that are suggested by the IMA can be explained as follows:

• an image that is linked from a great many articles on any wiki is considered an icon (with the threshold depending on the size of the wiki)
• an article that has only icons as images is considered unillustrated
• the IMA allowed icons to be used as image suggestions, which meant that someone could add a suggestion to an article, but the article would still be considered unillustrated (a bug)
• the new implementation removes icons from the suggestion list

... so to me it seems the the new pipeline is working as expected. Waiting confirmation from @mfossati and then we can close this

👍 See https://gitlab.wikimedia.org/repos/generated-data-platform/datapipelines/-/merge_requests/44