Image Suggestion API (Proof of Concept)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Charlotte
	Aug 19 2020, 5:35 PM

Description

Context

The image recommendations proof of concept will test putting an API layer over the image matching algorithm and MediaSearch. The API layer will serve up results from the image matching algorithm and additionally append non-duplicate results from MediaSearch to the image matching algorithm results.

Proof of Concept Success Criteria

We know whether results from the algorithms combined are superior to the results of image matching or MediaSearch alone
We know whether the combined forces of image matching and MediaSearch have varying results depending on wiki
Combining results from both sources and serving them up is performant (quick response time)
We’re able to test results across multiple wikis and compare effectiveness
We know whether the API design is the right fit for the features planned by Android, Growth, and Structured Data.

"Happy Path" Scope

List Image Suggestions for a Given Article

As a developer, when I make a request to the Image Suggestion API with a page title
- I expect to see a list of recommended images for the page I requested
  - The list should be static and paginated
  - The list should be at most 10 images per 1 page request
  - Of the 10 images
    - A maximum of 3 images should come from the ImageMatchingAlgorithm
    - 7-10 images should be from MediaSearch

List Image Suggestions for all Wikipedia languages

As a developer, when I make a request to the Image Suggestion API with a page title,
- I expect to be able to make requests for all Wikipedia projects in any language
  - Android will need across all Wikis. The following Wikis will be used in Structured Data’s test plan:
    - Arabic Wikipedia
    - Cebuano Wikipedia
    - English Wikipedia
    - Vietnamese Wikipedia

Provide the Source of an Image

As a developer, when I receive a list of images
- I expect to know the source of how the suggestion was provided
  - e.g. I see the image suggestion for the Frog page is from "Commons"

Provide the Confidence Rating of an Image

As a developer, when I receive a list of images
- I expect to know the confidence rating for each image recommended per page requested
  - e.g. I see that the image for "Amazonian Tree Frog.jpg" has a confidence rating of "high"

Error Scenarios

As a developer, when I make a request to the Image Suggestion API with page-ID
- I expect to get an error stating the full page title is required to make a query
As a developer, when I make a request to the Image Suggestion API
- And there are no image suggestions for the page
- I expect to receive an error stating there were no results found
Unauthorized
Unauthenticated
Service Unavailable

Out of Scope

Filter # of Image Suggestions Per Article Request

As a developer, when I provide a parameter to limit the number of image suggestions per page
- I expect to get somewhere between 1 and 10 images recommended per page requested

Record Users’ Judgement on Suggestion

As a contributor, when I accept/skip/reject an image suggestion, I expect my feedback will be taken in and update the state/quality/confidence of the suggestion.

Apply and record revision when image suggestion is accepted

As a contributor, when I've accepted an image suggestion to add to a page, I expect my edit to have been made and I can review the revision in the page history

As a developer, when I make a request to the Image Suggestion API, I expect to get metadata on the image as follows:
- Notes
- Descriptions
- Categories
- Captions
- Depicts
- Size
- Copyright
- Date
- User
- Image type

As a developer, when I make a request to the Image Suggestion API, I expect to be able to filter the results by image source

Non-Functional Requirements

Authentication
Authorization
Performance
API Product Metrics
ML Metrics
Documentation

Details

Due Date: Mar 3 2021, 5:00 AM

Related Objects
Search...

Status	Assigned	Task
Resolved	CBogen	T254768 [EPIC] Image recommendations proof-of-concept phase
Resolved	• sdkim	T260832 Image Suggestion API (Proof of Concept)
Resolved	• nnikkhoui	T272721 List unillustrated pages and their image suggestions (ImageMatchingAlgorithm only)
Declined	None	T272739 Provide the "image source" for each image recommended
Declined	None	T272740 Provide the "confidence rating" for each image recommended
Resolved	BPirkle	T272734 List Image Suggestion for all unillustrated articles (Algo + MediaSearch)
Declined	None	T272737 List Image Recommendations Per Wikipedia Project
Resolved	• nnikkhoui	T273225 Image Suggestion API "Hello World"
Declined	• sdkim	T273227 Replace ImageMatchingAlgorithm Mocked Data with Persisted Data
Resolved	• sdkim	T273779 📊 [SPIKE] Automate the generation of a .tsv flat file
Resolved	None	T274194 Remove boilerplate service-template-node code

Event Timeline

• Charlotte created this task.Aug 19 2020, 5:35 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 19 2020, 5:35 PM

• Charlotte moved this task from Needs Triage to Tracking on the Wikipedia-Android-App-Backlog board.Aug 19 2020, 5:44 PM

kaldari added a project: Growth-Structured-Tasks.Aug 19 2020, 7:33 PM

@Charlotte, thank you for your response. I have some follow-up questions:

Could you please add a link to the OKR in BetterWorks, please?
Is there already work being done in this area on our shared Platform-Product roadmap?
What would happen if the Platform team wasn't able to implement this API at the moment?
Are there any significant dependencies?
How will we measure the impact of this API?

Thank you!

@Charlotte below are some more follow-up questions from our team:

Is the API to add an image to an article? Is it to hook to the suggestions from search? Something else?

In T260832#6405466, @Naike wrote:

@Charlotte, thank you for your response. I have some follow-up questions:

Could you please add a link to the OKR in BetterWorks, please?

https://app.betterworks.com/app/#/goal/9734137

Releasing this specific feature is not an OKR because of the way we were told to write our OKRs. The feature has been put on the roadmap to contribute to this specific OKR.

Is there already work being done in this area on our shared Platform-Product roadmap?

Not sure what you're asking. If you mean, is a similar API presently being built, then no.

What would happen if the Platform team wasn't able to implement this API at the moment?

The Android feature won't get built. If you're not able to do it at all, then the Growth feature likely won't get built either.

Are there any significant dependencies?

Not sure what you're asking. Our dependency is on you. By the time you are ready to start, the Research team (specifically @Miriam) will have defined what is needed to be built. You may rely on her expertise, or on that of the Commons team, but so far as I know there is nothing else that you will require from other teams.

How will we measure the impact of this API?

Usage of the Android feature (and eventually the Growth feature) built from it.

Is the API to add an image to an article? Is it to hook to the suggestions from search? Something else?

Our needs are extensively documented in the parent ticket to this task. Briefly, we need it to do 3 things:

Suggest lead images from an article in language X to be added to the same article in language Y (for Android)
Suggest relevant images to add to articles (for Growth - correct me @MMiller_WMF if I am incorrect)
Add the image to the article in a place to be determined, most likely the infobox

What we mean about "lead" and "relevant" images is extensively discussed in T256081.

Hi @Charlotte, nice to meet you. Can you connect with @eprodromou on the Product side, to take this through our intake process?

In T260832#6424321, @AMooney wrote:

Hi @Charlotte, nice to meet you. Can you connect with @eprodromou on the Product side, to take this through our intake process?

We already spoke about it, and I was told by @Naike to fill this out. So is this conversation with Evan an additional thing, or did you not know we had already spoken?

Arrbee edited projects, added Tech-Product API Roadmap; removed Platform Team Workboards (Platform-Product Roadmap).Sep 2 2020, 2:15 PM

@Charlotte PET will be reviewing this for feasibility. I'll check with you beforehand.

Naike moved this task from Untriaged to In Review on the Tech-Product API Roadmap board.Sep 8 2020, 11:04 AM

Naike moved this task from In Review to Untriaged on the Tech-Product API Roadmap board.Sep 8 2020, 11:10 AM

Naike moved this task from Untriaged to In Review on the Tech-Product API Roadmap board.

I'm trying to understand the scope of this ticket. As an engineer, building an API means creating an interface that answers a web request by accessing some backend system, such as a database or a suggestion engine. What backend system would this API use? Is there something in place that can be used to generate image suggestions?

I just saw the CSV files in a comment to the parent task T256081#6385411. Do I understand correctly that this ticket asks for an experimental API that is based on fixed set of manually collected and deployed data?

In T260832#6450158, @daniel wrote:

I just saw the CSV files in a comment to the parent task T256081#6385411. Do I understand correctly that this ticket asks for an experimental API that is based on fixed set of manually collected and deployed data?

This has been discussed with @eprodromou and @sdkim, so I'm not sure why the scope is unknown. We discussed two things: first, a backend system where data generated by the algorithm @Miriam is working on will be stored (and the mechanism to actually populate this database), and second, an API to handle requests from the app to that system. The scope and details of the backend system were a task for them to discuss.

The API is not meant to be experimental, but is going to power a production feature.

Naike set the point value for this task to 1.Sep 11 2020, 10:54 AM

Linked here. https://phabricator.wikimedia.org/T262341

• sdkim assigned this task to • JKatzWMF.Sep 16 2020, 8:29 PM

• eprodromou moved this task from In Review to Later on the Tech-Product API Roadmap board.Sep 23 2020, 8:02 PM

• sdkim moved this task from Later to Next on the Tech-Product API Roadmap board.Sep 30 2020, 7:52 PM

• sdkim mentioned this in T259487: [EPIC] Structured Task API [DD - discovery].Sep 30 2020, 8:54 PM

• sdkim added a parent task: T259487: [EPIC] Structured Task API [DD - discovery].Sep 30 2020, 8:57 PM

• sdkim removed a project: Tech-Product API Roadmap.

leila moved this task from FY2020-21-Research-July-September to In Progress on the Research board.Oct 16 2020, 8:34 PM

leila edited projects, added Research; removed Research (FY2020-21-Research-July-September).

CBogen added a project: Image-Suggestions.Dec 9 2020, 10:06 PM