Page MenuHomePhabricator

AI/ML Model Request: Image auto-crop / focus point detection
Open, Needs TriagePublic

Description

Please respond to the following questions, and provide as much detail as possible for each.

Scoping details

  • Use case: Describe the user-facing experience(s) that this model will serve. Who is the intended audience? Where and how will the model outputs be surfaced to users? Please feel free to link to any demos, prototypes, design files, etc.

Reader Growth has multiple reader-facing features that would benefit from smart cropping such that the most important elements of an original image are automatically included in its cropped version (e.g., Image Browsing will crop images for display in a carousel, Share will crop images to fit in a card).
Apps Team: image cropping for article view, and full-screen images within the redesigned explore feed: Figma mockup
Note: REx may also have use cases cc @HFan-WMF

  • Model purpose: What should the model do? What does it need to predict or generate?

Given an original image and target aspect ratio, the model should (1) predict which portions of the image are of interest and (2) auto-crop the image to that ratio while capturing those most important elements.

  • Goal: What's the goal of this user experience? What patterns in user behavior do we want to impact? What metrics will let us know we're successful?

As we see digital life shift to be more dependent on rich multimedia, we want to improve the polish of our image-based experiences and engage readers. Metrics may vary by feature where we use the model, but examples would be clickthrough and task completion rate.

  • Prior art: How much of the UI for this experience has already been developed and/or tested? Are there any previous models or manually-created rules that we can learn from?

none, to my knowledge!

Prioritization details

  • Timing: When are you hoping to launch an experiment or feature using this model? How flexible is your timeline? Is there any other planned work that's blocked by this experiment or feature?

flexible, happy to determine timeline together

  • KR impact: Which KRs are enabled by this project, and how critical is this project for moving the needle on those KRs?

OW3.1, potentially OW3.2 cc @HFan-WMF

Other comments

  • [Optional] Model requirements: If you have any specific concerns around model performance (latency, cost, etc.) or model output quality (likelihood of false positives, ability to detect all possible instances, etc.), please note them here.
  • [Optional] Is there anything else you'd like to share?