Page MenuHomePhabricator
Create Task
Maniphest T299781

[EPIC] Image suggestions backend
Closed, ResolvedPublic
Actions

Description

This epic covers all the work that needs to be carried out by Structured Data to

  • productionize the experimental work we did incorporating wikidata data into mediasearch (T283869)
  • store image suggestions data in such a way that it can be consumed by other teams/projects

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedCBogenT299781 [EPIC] Image suggestions backend
ResolvedmfossatiT296814 [EPIC] Article-level image suggestions data pipeline
 ResolvedmfossatiT300045 Refactor commons_wikidata_links/gather_data.ipynb notebook as a python script
 ResolvedmfossatiT302095 [M] Compute the Wikidata tags update slice for the Commons search index
 DeclinedNoneT299787 Push wikidata data to the commonswiki_file search index
 ResolvedCparleT299789 [XL] Store a list of unillustrated articles with suggested images in hdfs
 ResolvedCparleT301687 Calculate image suggestions confidence score without using elasticsearch
 ResolvedCparleT299884 Prepare has-recommendation data for import to wiki search indices
 ResolvedmfossatiT299885 [L] Push unillustrated articles with their suggestions, suggestion reasons and confidence scores to Cassandra
 DeclinedCparleT307092 Prune suggestions from previous iterations from Cassandra
 InvalidmfossatiT299408 Add lead image data to the data file generated for populating weighted_tags in the commons search index
 ResolvedmfossatiT302434 [L] Orchestrate image suggestions tasks with Apache Airflow
 ResolvedmfossatiT303816 [L] Refactor data pipeline queries
 ResolvedCparleT307983 Write search index data for image suggestions into a hive table rather than local hdfs files
 ResolvedmfossatiT306387 [M] Create image suggestions pipeline documentation
 ResolvedCparleT299890 [M] Exclude previously rejected image suggestions when generating new suggestions
 ResolvedmfossatiT307356 Image suggestions data pipeline onboarding request
 ResolvedmfossatiT307362 Memory errors break the expected output of some Airflow tasks
 ResolvedCparleT307371 Data pipeline weekly schedule is on hold
 ResolvedmfossatiT307899 Data pipeline grooming
 ResolvedCparleT298684 [M] Add 'custommatch' params to commons config for searching media files using wikidata ids
 ResolvedCBogenT281582 [EPIC] Develop a confidence score for MediaSearch results
 ResolvedCparleT272710 Investigate whether the probability-of-an-image-being-good score is useful as a confidence score
 DeclinedNoneT283863 Incorporate image-recommendation-test results into the image recs API confidence score
 ResolvedCBogenT283865 [XL] Estimate coverage of image suggestions at different confidence levels
 ResolvedCparleT286563 [M] Create new search profile for commons that uses weighted_tags

Event Timeline

Cparle created this task.Jan 21 2022, 5:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 21 2022, 5:15 PM
Cparle moved this task from Incoming to Epics on the Structured-Data-Backlog (Current Work) board.Jan 21 2022, 5:15 PM
Cparle added projects: Epic, Image-Suggestions, Image-Suggestion-API.
Cparle added a subtask: T299408: Add lead image data to the data file generated for populating weighted_tags in the commons search index.Jan 21 2022, 5:24 PM
Cparle added a subtask: T296814: [EPIC] Article-level image suggestions data pipeline.Jan 21 2022, 6:12 PM
Cparle added a subtask: T298684: [M] Add 'custommatch' params to commons config for searching media files using wikidata ids.
Cparle mentioned this in T283869: [EPIC] Experiment with incorporating Image Matching Algorithm data into commons search index.Jan 21 2022, 6:15 PM
Cparle mentioned this in T299895: Incorporate acceptance/rejection of suggested images for articles back into MediaSearch model.Jan 24 2022, 10:58 AM
Cparle added subscribers: Marco, CBogen, matthiasmullie and 3 others.Jan 24 2022, 11:01 AM
Cparle edited subscribers, added: mfossati; removed: Marco.Jan 24 2022, 11:04 AM
Cparle added a subscriber: AUgolnikova-WMF.Jan 24 2022, 11:06 AM
lbowmaker subscribed.Jan 24 2022, 1:46 PM
CBogen added a subtask: T299895: Incorporate acceptance/rejection of suggested images for articles back into MediaSearch model.Jan 24 2022, 1:55 PM
CBogen removed a subtask: T299895: Incorporate acceptance/rejection of suggested images for articles back into MediaSearch model.
Cparle mentioned this in T292142: [EPIC] Image Suggestions Notifications for More Experienced Contributors.Jan 25 2022, 1:51 PM
Cparle updated the task description. (Show Details)Jan 25 2022, 2:19 PM
Miriam subscribed.Jan 28 2022, 3:23 PM
Cparle closed subtask T299408: Add lead image data to the data file generated for populating weighted_tags in the commons search index as Invalid.Feb 1 2022, 11:40 AM
mfossati changed the status of subtask T300045: Refactor commons_wikidata_links/gather_data.ipynb notebook as a python script from Open to In Progress.Feb 2 2022, 10:36 AM
CBogen added a subtask: T281582: [EPIC] Develop a confidence score for MediaSearch results.Feb 7 2022, 3:38 PM
mfossati closed subtask T300045: Refactor commons_wikidata_links/gather_data.ipynb notebook as a python script as Resolved.Feb 18 2022, 6:06 PM
CBogen closed subtask T299787: Push wikidata data to the commonswiki_file search index as Declined.Feb 23 2022, 5:31 PM
mfossati changed the status of subtask T299885: [L] Push unillustrated articles with their suggestions, suggestion reasons and confidence scores to Cassandra from Open to In Progress.Apr 11 2022, 9:10 AM
Cparle closed subtask T299789: [XL] Store a list of unillustrated articles with suggested images in hdfs as Resolved.Apr 21 2022, 2:43 PM
mfossati closed subtask T299884: Prepare has-recommendation data for import to wiki search indices as Resolved.Apr 26 2022, 2:07 PM
Cparle closed subtask T299885: [L] Push unillustrated articles with their suggestions, suggestion reasons and confidence scores to Cassandra as Resolved.Apr 27 2022, 2:43 PM
CBogen closed subtask T281582: [EPIC] Develop a confidence score for MediaSearch results as Resolved.May 9 2022, 4:41 PM
mfossati closed subtask T299890: [M] Exclude previously rejected image suggestions when generating new suggestions as Resolved.Jun 8 2022, 1:48 PM
mfossati mentioned this in T311814: [EPIC] Section-level image suggestions data pipeline.Jul 1 2022, 9:30 AM
mfossati mentioned this in T311831: [SPIKE] Consider Structured Data on Commons to gather additional image QIDs.Jul 1 2022, 11:37 AM
mfossati mentioned this in T311832: [SPIKE] Consider other Wikidata properties to gather additional image QIDs.Jul 1 2022, 11:49 AM
CBogen moved this task from To Do to Epics on the Image-Suggestions board.Jul 7 2022, 6:32 PM
CBogen moved this task from Epics to Data Pipeline on the Image-Suggestions board.Jul 7 2022, 6:35 PM
mfossati closed subtask T296814: [EPIC] Article-level image suggestions data pipeline as Resolved.Jul 18 2022, 9:23 AM
Cparle closed subtask T298684: [M] Add 'custommatch' params to commons config for searching media files using wikidata ids as Resolved.Jul 18 2022, 2:10 PM
CBogen added a comment.Jul 18 2022, 7:25 PM

@mfossati and @Cparle , can this epic be closed?

Cparle added a comment.Jul 19 2022, 11:27 AM

I think so yes. The pipeline code hasn't been merged into the main repo yet, but that doesn't need to be part of an epic (and there's a ticket for it)

CBogen closed this task as Resolved.Aug 4 2022, 3:52 PM
CBogen claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits