Change Details

**Open Questions** [] How can we grade all 3000 options by the end of the experiment? How many people will we need in order to accomplish this over what length of time? [x] Do we want people to tell us why they rejected/skipped? Which options? [x] How many recommendations should we show each user? [x] Do we want to show it to everyone, language requirements? [x] How will we know people like the task? [] Can we tell if things are easy vs hard? [] Can we see how people like it and if so, by user type? [x] What will be our check points to determine if we tweak [x] Do we want to test how well users understand and their ability to write good captions for the accepted recommendations? [] How do we learn what supplementary information users need to make good decisions? [] Which info (image categories, source of suggestion, image resolution, etc) do people refer to the most in making their decisions? [] Which information leads to more accurate ratings? [x] Do we also want to enable the ability for users to filter suggestions by interest topics ORES in this test tool? [] Practical usage by (esp. experienced) editors - do we want to monitor if people start using the tool in unintended ways? Such as using it to find unillustrated articles to add their own images? **Product Decisions** - We will have one suggested image per article instead of multiple images - This iteration of the MVP will not include Image Captions - There are no language constraints for this task. As long as there is an article available in the language we will surface it. We want to be deliberate in ensuring this task is completed by a variety of languages. For this MVP to be considered a success, we want the task completed in at least five different languages including English, an indic language and Latin language. - We will have a check point two weeks after the launch of the feature to check if the feature is working properly and if modifications need to be made in order to ensure we are getting the answers to our core questions. The check point is not intended to introduce scope creep. - We aren't able to filter by categories in this iteration of the MVP, but it could be a possibility in the future through the CPT API - We will Surface a survey each time a user says no and sparingly surface a survey when a user clicks Not Sure or Skip - We need three annotations from 3000 different users on 3000 different matches - We will know people like the task if they return to complete it on three distinct dates

**Open Questions** [] How can we grade all 3000 options by the end of the experiment? How many people will we need in order to accomplish this over what length of time? [x] Do we want people to tell us why they rejected/skipped? Which options? [x] How many recommendations should we show each user? [x] Do we want to show it to everyone, language requirements? [x] How will we know people like the task? [] Can we tell if things are easy vs hard? [x] Can we see how people like it and if so, by user type? [x] What will be our check points to determine if we tweak [x] Do we want to test how well users understand and their ability to write good captions for the accepted recommendations? [] How do we learn what supplementary information users need to make good decisions? [] Which info (image categories, source of suggestion, image resolution, etc) do people refer to the most in making their decisions? [] Which information leads to more accurate ratings? [x] Do we also want to enable the ability for users to filter suggestions by interest topics ORES in this test tool? [] Practical usage by (esp. experienced) editors - do we want to monitor if people start using the tool in unintended ways? Such as using it to find unillustrated articles to add their own images? **Product Decisions** - We will have one suggested image per article instead of multiple images - This iteration of the MVP will not include Image Captions - There are no language constraints for this task. As long as there is an article available in the language we will surface it. We want to be deliberate in ensuring this task is completed by a variety of languages. For this MVP to be considered a success, we want the task completed in at least five different languages including English, an indic language and Latin language. - We will have a check point two weeks after the launch of the feature to check if the feature is working properly and if modifications need to be made in order to ensure we are getting the answers to our core questions. The check point is not intended to introduce scope creep. - We aren't able to filter by categories in this iteration of the MVP, but it could be a possibility in the future through the CPT API - We will Surface a survey each time a user says no and sparingly surface a survey when a user clicks Not Sure or Skip - We need three annotations from 3000 different users on 3000 different matches - We will know people like the task if they return to complete it on three distinct dates we will compare frequency of return by date across user type to understand if there was more stickiness for this task by how experienced a user is

**Open Questions** [] How can we grade all 3000 options by the end of the experiment? How many people will we need in order to accomplish this over what length of time? [x] Do we want people to tell us why they rejected/skipped? Which options? [x] How many recommendations should we show each user? [x] Do we want to show it to everyone, language requirements? [x] How will we know people like the task? [] Can we tell if things are easy vs hard? [x] Can we see how people like it and if so, by user type? [x] What will be our check points to determine if we tweak [x] Do we want to test how well users understand and their ability to write good captions for the accepted recommendations? [] How do we learn what supplementary information users need to make good decisions? [] Which info (image categories, source of suggestion, image resolution, etc) do people refer to the most in making their decisions? [] Which information leads to more accurate ratings? [x] Do we also want to enable the ability for users to filter suggestions by interest topics ORES in this test tool? [] Practical usage by (esp. experienced) editors - do we want to monitor if people start using the tool in unintended ways? Such as using it to find unillustrated articles to add their own images? **Product Decisions** - We will have one suggested image per article instead of multiple images - This iteration of the MVP will not include Image Captions - There are no language constraints for this task. As long as there is an article available in the language we will surface it. We want to be deliberate in ensuring this task is completed by a variety of languages. For this MVP to be considered a success, we want the task completed in at least five different languages including English, an indic language and Latin language. - We will have a check point two weeks after the launch of the feature to check if the feature is working properly and if modifications need to be made in order to ensure we are getting the answers to our core questions. The check point is not intended to introduce scope creep. - We aren't able to filter by categories in this iteration of the MVP, but it could be a possibility in the future through the CPT API - We will Surface a survey each time a user says no and sparingly surface a survey when a user clicks Not Sure or Skip - We need three annotations from 3000 different users on 3000 different matches - We will know people like the task if they return to complete it on three distinct dates we will compare frequency of return by date across user type to understand if there was more stickiness for this task by how experienced a user is