Page MenuHomePhabricator
Create Task
Maniphest T299885

[L] Push unillustrated articles with their suggestions, suggestion reasons and confidence scores to Cassandra
Closed, ResolvedPublic
Actions

Description

NOTE: blocked by T299408 and T299789

User story

As a product manager, I want a single Source Of Truth for image suggestions
As a developer, I need a place to store whether an image suggestion is accepted or rejected, so I can use that data in future
So we need to store image suggestions with confidence scores and acceptance/rejection data

Cassandra will the the Source Of Truth for images suggestions, and so once we have all the data we want to store it there

We want to store all the data in the suggestions, title_cache and instanceof_cache tables in T293808

To do this we'll need at least a preliminary dataset from T299789, and will probably use T297934

Related Objects
Search...

Event Timeline

Cparle created this task.Jan 24 2022, 10:14 AM
Cparle updated the task description. (Show Details)
Cparle updated the task description. (Show Details)
Cparle mentioned this in T296814: [EPIC] Article-level image suggestions data pipeline.Jan 24 2022, 10:33 AM
Cparle mentioned this in T299890: [M] Exclude previously rejected image suggestions when generating new suggestions.Jan 24 2022, 10:36 AM
Cparle updated the task description. (Show Details)Jan 24 2022, 10:38 AM
Cparle mentioned this in T299895: Incorporate acceptance/rejection of suggested images for articles back into MediaSearch model.Jan 24 2022, 10:58 AM
Cparle removed a project: Epic.Jan 24 2022, 12:46 PM
CBogen edited projects, added Structured-Data-Backlog; removed Structured-Data-Backlog (Current Work).Jan 24 2022, 2:52 PM
Cparle mentioned this in T292147: [L] Send Image Suggestions notifications to experienced users.Jan 25 2022, 12:10 PM
Cparle updated the task description. (Show Details)Jan 25 2022, 6:21 PM
Cparle updated the task description. (Show Details)Jan 31 2022, 11:57 AM
Tgr subscribed.Feb 7 2022, 2:26 AM
mfossati added a parent task: T296814: [EPIC] Article-level image suggestions data pipeline.Feb 18 2022, 5:52 PM
Cparle claimed this task.Feb 23 2022, 5:29 PM
Cparle removed Cparle as the assignee of this task.
Cparle added a comment.EditedFeb 23 2022, 5:36 PM

Note that we currently have a notebook for gathering the data but we don't have agreement about how to get the data into cassandra yet

CBogen edited projects, added Structured-Data-Backlog (Current Work); removed Structured-Data-Backlog.Mar 14 2022, 4:34 PM
CBogen moved this task from Incoming to Ready for Estimation on the Structured-Data-Backlog (Current Work) board.
Cparle updated the task description. (Show Details)Mar 14 2022, 5:31 PM
CBogen renamed this task from Push unillustrated articles with their suggestions, suggestion reasons and confidence scores to Cassandra to [L] Push unillustrated articles with their suggestions, suggestion reasons and confidence scores to Cassandra.Mar 23 2022, 4:22 PM
CBogen moved this task from Ready for Estimation to Ready for Development on the Structured-Data-Backlog (Current Work) board.Mar 24 2022, 1:23 PM
mfossati mentioned this in T302434: [L] Orchestrate image suggestions tasks with Apache Airflow.Apr 6 2022, 9:13 AM
mfossati claimed this task.Apr 6 2022, 9:21 AM
mfossati moved this task from Ready for Development to Blocked on the Structured-Data-Backlog (Current Work) board.
mfossati subscribed.

Waiting for some initial input data from T299789: [XL] Store a list of unillustrated articles with suggested images in hdfs, then we can start.

mfossati moved this task from Blocked to Ready for Development on the Structured-Data-Backlog (Current Work) board.Apr 8 2022, 3:47 PM

Moving back to ready: preliminary data available now, thanks @Cparle !

mfossati changed the task status from Open to In Progress.Apr 11 2022, 9:10 AM
mfossati moved this task from Ready for Development to Doing on the Structured-Data-Backlog (Current Work) board.
mfossati moved this task from Doing to Code Review on the Structured-Data-Backlog (Current Work) board.Apr 26 2022, 3:15 PM
  • Manual executions from stat1008's command line populated the target Cassandra tables fine
  • two manual executions of relevant tasks in the Airflow DAG were successful
NOTE: if you have access to the relevant Airflow machine, see http://localhost:8600/taskinstance/list/?_flt_3_dag_id=image-suggestions&_flt_3_state=success&_oc_TaskInstanceModelView=execution_date&_od_TaskInstanceModelView=desc.

Merge request at https://gitlab.wikimedia.org/repos/generated-data-platform/datapipelines/-/merge_requests/47

Cparle closed this task as Resolved.Apr 27 2022, 2:43 PM

Merged, closing

Cparle closed subtask T307092: Prune suggestions from previous iterations from Cassandra as Declined.May 3 2022, 8:43 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL