Page MenuHomePhabricator

Image Matching Structured Task: Research Q3-Q4
Closed, ResolvedPublic

Description

Q3 and Q4 weekly progresses on the Research front for the Image Matching structured task

Event Timeline

Weekly updates:

  • Worked on image placeholder identification [T277828]: identified a category-based approach to label images as placeholders, and incorporated it into the algorithm. Trained a simple neural network to distinguish between normal images and placeholder images, with accuracy 88%+
  • Worked with Platform Engineering to fix bug T277875 and identified the source of the problem
  • Helped Structured Data refining the design of the test for image recommendations POC results T273092

Weekly updates:

  • Worked with Platform Engineering to integrate placeholder checks T277828
  • Worked with Marhsall to define the Annual Plan directions for structured tasks

Weekly updates:

  • Image Recs on Android is now available in Beta. It will be in production next week if everything goes well.

Weekly updates:

  • No major updates.

Weekly updates:

  • Started analyzing the data in T273057 to check for early sign of algorithm accuracy by source of recommendation

Weekly updates:

  • Ran accuracy diagnositcs on the data from T273057. For the 2 largest sources, it looks like the algorithm has a 64-71% accuracy, meaning the majority of users say@Ai that the algorithm recommends a good match:

Screenshot from 2021-05-21 17-59-27.png (384×1 px, 93 KB)

It looks like the time it takes a user to give a response is also inversely proportional with goodness of the match:
Screenshot from 2021-05-21 18-00-15.png (312×685 px, 34 KB)

  • Next @AikoChou and I will breakdown these metrics by user expertise and topic.
  • Discussed with ML team + Platform + Product about when and how to convert this algo into a model that can be served through LiftWing.

Analyzed with Aiko the results of the Android POC, to understand
(1) The extent to which newcomers behave and annotate data differently
(2) The extent to which non-english users struggle with the POC
(3) The reliability of newcomers annotations (via agreement)

Results are here: https://docs.google.com/spreadsheets/d/1BOPYt_R2c1XZp2HgeTi346Kkge9w_9oNLFyONMHEPTw/edit#gid=0

Weekly updates:
Finalized the analysis of Android data, and helped the Growth team with the decision-making process around whether to deploy the "add an image" task as part of the newcomers structured tasks. They decided to go for it next fiscal year.