@EBernhardson re-imported the data, and now for ptwiki at least we have ~130k articles again. Still trying to see what went wrong

Here's what's involved in doing a Cassandra-based solution

One other option:

LGTM too, thanks everyone

I thought @hnowlan 's patch meant that this was deployed but it's still not working so I guess not?

This can probably be closed now?

Ok looks like the data has imported correctly, hooray!

Done, processed 58495 files

Ideal except for we'd have to re-rewrite a bunch of code ...

I have no idea why that is ... we're just using df.write.saveAsTable(). Is there any config we can do to improve this?

Waiting for the patch to be merged before closing this

yeah, basically it's one dataset - we didn't think of it that way at the start, but it turns out the data is the same shape for both so it's all in the same table

Ok so now the data is being written to the tables image_suggestions_search_index_full and image_suggestions_search_index_delta in the hive db analytics_platform_eng. Partitioned by a snapshot column in the format yyyy-mm-dd

Data is being written to the tables image_suggestions_search_index_full and image_suggestions_search_index_delta in the hive db analytics_platform_eng. Partitioned by a snapshot column in the format yyyy-mm-dd

Still, my personal feeling is that we should target the overall Commons search system effectiveness for users, rather than focusing on eventual recall changes due to the activation of a feature.

Marco's sampled the search terms from the logs based on a mixture of popularity and random, but just looking at the sampled search terms for French, for example, very few of them match up with wikidata labels and therefore won't have any synonyms ... and seeing that the point of this exercise is to capture the effect of the synonyms patch, we've probably been barking up the wrong tree.

Update on this ticket - looking at the data I'm not sure that what we've gathered is capturing the effect of the synonyms patch, and I think we might need to curate it more carefully.

One limitation of the current import scripts is they expect everything to be sourced from partitioned hive tables. Typically we partition by a date col of the airflow execution date. Would it take much to arrange these into a partitioned table? Saving to hive partitions might also resolve the permissions issues by way of different defaults, although I'm not entirely sure.

@Cparle which confidence level are we using in the current iteration of the data pipeline?

Ok to resolve this @CBogen ?

Confidence >= 90%

Having spoken to @Eevans about this I'm going to close this ticket. Because the script runs only once a week there's no way to completely prevent out of date data from being served to users, and keeping the has-suggestion flags up to date in the wiki search indices should prevent the particular problem we were trying to fix with this ticket from ever reaching users

Merged, closing

Patch written https://gitlab.wikimedia.org/repos/generated-data-platform/datapipelines/-/tree/T299890-exclude-rejections but can't fully test it until the schema is deployed (see https://gitlab.wikimedia.org/repos/generated-data-platform/topics/image-suggestions-feedback/-/merge_requests/1)

As an alternative to a public API, we could provide a flat file containing all image suggestions for any wiki quite easily (in .csv format or similar)

https://gitlab.wikimedia.org/repos/generated-data-platform/datapipelines/-/merge_requests/46

... aaand merged. Closing the ticket

Ok ran the old IMA again but with the data-loss part fixed (I think!), and now from it we're getting suggestions for 82399 articles, 81177 of which are also suggested by the new pipeline

I think all remaining refactoring work has been done, so closing

Ok this is pretty much done. We're writing to Hive instead of hdfs, to make it easier to export to Cassandra

We've changed our approach to calculating confidence scores, and are now estimating them before storing image suggestions. This ticket is therefore no longer necessary for image suggestions, as we don't have another use case for getting images for a particular Q-id

In T302925#7773106, @kostajh wrote:

A potential issue with the 2-3 hour delay will be that image suggestions that have been rejected could be potentially resurfaced to users.

We have example data files on hdfs

Also ... I don't think the way the data is being stored allows for that anyway. We store the user who has rejected an image, not the tool they were using at the time, see P21420 Perhaps this is what the comment field is intended for? Not sure.

It would mean that, yes

In T294468#7770740, @kostajh wrote:

AIUI this plan sounds OK. To recap Growth team's existing use cases:

...

3. Currently, when a user accepts or rejects an image suggestion, we enqueue a job with CirrusSearch to reset the weighted tag for hasrecommendation:image for that article. With the new API, we would also send an HTTP request to the feedback endpoint proposed in T294468#7748996. I assume that the search index updating code in T299884 (cc @Cparle) would take into account where an article has feedback before updating the weighted tag hasrecommendation:image for an article, or perhaps a new field like hasfeedback:image.rejected would be useful to someone.

Moved this into blocked - it should be quite easy to do once we have T299789 done, so there's no point in wasting effort doing it before then

In the source-of-truth for image suggestions (Cassandra, see T293808) we'll be storing the value of instance of for each article. This means we can exclude articles with instance of==Q5

We'll need at least a preliminary dataset from to do this work

This ticket was a change to an interface that never made it to production and is no longer in development, so closing

ok grand

Will there be another API with some business logic to complement the generic API?

After running queries on the labeled data, it turns out the most reliable confidence score is simply based on the source of the match

Note that we currently have a notebook for gathering the data but we don't have agreement about how to get the data into cassandra yet

Another source of ground truth might be images that were added then reverted within e.g. a day?

Hmmm ok so you have no dump-and-reload mechanism? If not we'll have to keep the data from the previous run in order to work out the __DELETE_GROUPING__ part

@Multichill is the bot just using wbsetclaim then a null edit? Are you getting this with any of your other bots?

@Zbyszko this isn't api documentation as such, but explains how MediaSearch works https://www.mediawiki.org/wiki/Extension:WikibaseMediaInfo/MediaSearch MediaSearch just uses the standard search api with a media-specific profile loaded if you're searching in the File namespace, though there are a couple of features that were developed specifically for media, namely haswbstatement and wbstatementquantity (in WikibaseCirrusSearch) and custommatch (in WikibaseMediaInfo)