Page MenuHomePhabricator

Translate image captions in “Suggested edits”
Closed, ResolvedPublic5 Story Points

Description

Conceptual descriptions

  • Unlock criteria:
    • Users have set multiple languages in the app
    • 15 edited image captions
  • Feed: Show unstructured (aka “moderately structured“) file captions in “From” language that don’t have any structured captions yet. Why should we use unstructured data as a source? Because this is how we increase the pool of structured data captions.
    • To discuss: Preferably, we are displaying captions that are associated with Wikipedia articles. After all, it’s the *Wikipedia* Android app (according to @Mholloway, indexing will take several weeks, so this is more something for version 2)
  • Card information architecture (output as plain text), from Commons file page
    • File name
    • Author, if not available, display Artist
    • Date
    • Source
    • [Show more structured data based on available space)
  • Images are displayed in 16:9 ratio
  • Tapping the card leads editors to the Translate image caption screen
  • UI enhancements for “Suggested edits“
    • Removing info icon from input field since it’s available at various other places, we don’t want to interrupt the flow.
    • Add a microphone button/icon to the input field to promote voice input
    • Truncate bottom of card in an improved way, as showcased here.
    • Instead of only being a tap target, the bottom sheet should be draggable as well (Details in T223130)

Visuals

👉Zeplin: T223131 | Add image caption translation to Suggested Edits feed

Event Timeline

Charlotte triaged this task as Normal priority.May 21 2019, 4:49 PM
Charlotte set the point value for this task to 4.
Charlotte changed the point value for this task from 4 to 5.May 21 2019, 4:55 PM
schoenbaechler renamed this task from Add image caption translation to Suggested Edits feed to Translate image captions within “Suggested edits”.May 22 2019, 1:28 PM
schoenbaechler updated the task description. (Show Details)
schoenbaechler renamed this task from Translate image captions within “Suggested edits” to Translate image captions to “Suggested edits”.May 22 2019, 5:04 PM
schoenbaechler updated the task description. (Show Details)
schoenbaechler renamed this task from Translate image captions to “Suggested edits” to Translate image captions in “Suggested edits”.May 22 2019, 5:08 PM

The other question:

Does the Translate image captions or Add image captions have the "unlocked" dialogs or "unlocked" notifications?

Like this: T217369

The other question:
Does the Translate image captions or Add image captions have the "unlocked" dialogs or "unlocked" notifications?
Like this: T217369

Yes - I think I remember @schoenbaechler saying the designs for these are in progress. Robin?

Yes @cooltey, as @Charlotte mentioned, it’s work in progress. Todo list can be found in the EPIC: T207338, it currently consists of:

  • Designs for notifications and unlock dialogs
  • Designs for Explore feed
  • Should we promote caption editing on article pages? (Discussed in T209997)
cooltey added a comment.EditedMay 24 2019, 10:17 PM

Just noticed that the current icons in Suggested edits feed seem to be darker than the new one, please see the following screenshot, should we update the icons color in other ticket?

Mholloway added a comment.EditedMay 29 2019, 5:21 PM

Show unstructured (aka “moderately structured“) file captions in “From” language that don’t have any structured captions yet. Why should we use unstructured data as a source? Because this is how we increase the pool of structured data captions.

I'd like to raise a technical issue here: the MediaWiki API actually gives us unstructured captions by parsing them from the page HTML, which makes for a very slow query, to the point where we have to limit the number of candidates requested to avoid the request timing out. I'll try to implement this with a preference for (unstructured caption AND no structured caption in source lang) in the first instance, but if we run into very poor performance or limited result numbers, I might ask if we can relax this.

cooltey added a comment.EditedMay 30 2019, 12:53 AM

Hi @schoenbaechler

[Show more structured data based on available space)

Not sure how we can show the structured data dynamically since the endpoint does not return the data. (e.g. Place of publication, Language, Publisher...etc)

@Mholloway, is it possible to show the data like: "Publisher", "Language"? And if it is possible, would it be also return name of the field in the response?
https://commons.wikimedia.org/wiki/File:Grimms_Märchen_Anmerkungen_(Bolte_Polivka)_I_308.jpg
https://en.wikipedia.org/w/api.php?format=json&formatversion=2&errorformat=plaintext&action=query&prop=imageinfo&iiprop=extmetadata&titles=File:Grimms%20Märchen%20Anmerkungen%20(Bolte%20Polivka)%20I%20308.jpg

Mholloway added a comment.EditedMay 30 2019, 1:21 AM

Actually, I hope to persuade you that it's not worth including any extmetadata fields (which appear structured on the page and in the API, but actually aren't), even the unstructured captions, for the same reason: parsing them from the page HTML slows down the queries dramatically and limits the number of results we can reasonably request at a time (which of course matters a lot since we have to do some filtering). I hope to have a testing API up by tomorrow which illustrates this, using optional query parameters for different unstructured caption inclusion strategies.

On the bright side, as more SDC fields come online, we should be able to add as many as we need without any impact on performance.

Oh, since you mentioned language, I should say that all captions will come with language codes; and also, the new API will fully handle language variants.

Thanks for bringing this up @cooltey.


@Mholloway

I'll try to implement this with a preference for (unstructured caption AND no structured caption in source lang) in the first instance, but if we run into very poor performance or limited result numbers, I might ask if we can relax this.

Ok, thanks for the info!

Actually, I hope to persuade you that it's not worth including any extmetadata fields (which appear structured on the page and in the API, but actually aren't), even the unstructured captions, for the same reason

My latest info on this comes from @JoeWalsh in T223130#5185241, where he suggests to start with these fields: “(...) structured captions, unstructured captions, description, artist, credit, license, and depicts (...).” Is that still the case? The ideal scenario (mentioned in T223132) would be:

  • Image caption (Source: unstructured, else structured)
  • Depicts
  • Author
  • Artist
  • Date
  • Source/Photographer
  • Licensing
  • File usage on Wikipedia in the current app language(s) [not relevant for Suggested edit feed cards]
  • More infos (link to file page on Commons) [not relevant for Suggested edit feed cards]

Oh, since you mentioned language, I should say that all captions will come with language codes; and also, the new API will fully handle language variants.

Sounds great! 👌

Actually, I hope to persuade you that it's not worth including any extmetadata fields (which appear structured on the page and in the API, but actually aren't), even the unstructured captions, for the same reason

My latest info on this comes from @JoeWalsh in T223130#5185241, where he suggests to start with these fields: “(...) structured captions, unstructured captions, description, artist, credit, license, and depicts (...).” Is that still the case? The ideal scenario (mentioned in T223132) would be:

  • Image caption (Source: unstructured, else structured)
  • Depicts
  • Author
  • Artist
  • Date
  • Source/Photographer
  • Licensing
  • File usage on Wikipedia in the current app language(s) [not relevant for Suggested edit feed cards]
  • More infos (link to file page on Commons) [not relevant for Suggested edit feed cards]

Maybe the engineers can chime in here, but I'd suggest doing a follow-up imageinfo query for this info on a per-image basis for selected candidates, similar to how a page summary request is performed on demand for page previews. Getting unstructured imageinfo for a single image is reasonably fast, but doing so for large numbers in a single request (as I'm doing when generating suggestions) gets to be a problem.

Mholloway added a comment.EditedMay 31 2019, 5:43 PM

I'd be remiss if I didn't mention that there's a pretty substantial number of NSFW images on Commons. Has the team considered that randomized image suggestions will likely end up surfacing these to users who aren't necessarily expecting them?

Unfortunately there's no way I know of filtering these at the present moment (there is a "bad image list" construct that some wikis use, but Commons doesn't use it). On the bright side, there's work I'm helping out with (T214201) to get an AI-based NSFW image scoring system set up. Once that's productionized, we could update the API code to avoid recommending images with NSFW likelihood scores above a certain threshold.

I'd be remiss if I didn't mention that there's a pretty substantial number of NSFW images on Commons. Has the team considered that randomized image suggestions will likely end up surfacing these to users who aren't necessarily expecting them?
Unfortunately there's no way I know of filtering these at the present moment (there is a "bad image list" construct that some wikis use, but Commons doesn't use it). On the bright side, there's work I'm helping out with (T214201) to get an AI-based NSFW image scoring system set up. Once that's productionized, we could update the API code to avoid recommending images with NSFW likelihood scores above a certain threshold.

Yes, it had crossed my mind. And I'm not overjoyed at the idea of (worst case scenario) showing nudes to a kid or something. So let's please update the API as soon as the NSFW scoring system is online, but meanwhile I think it's an acceptable risk - especially since we are going to prioritise images that are associated with articles, which are not likely to be NSFW. (I mean, there's a nonzero chance, but it is lower than if we just showed a random image from a pool of literally everything.)

@Johan - We should probably put something in the help docs about this, eh?

Actually, I hope to persuade you that it's not worth including any extmetadata fields (which appear structured on the page and in the API, but actually aren't), even the unstructured captions, for the same reason

My latest info on this comes from @JoeWalsh in T223130#5185241, where he suggests to start with these fields: “(...) structured captions, unstructured captions, description, artist, credit, license, and depicts (...).” Is that still the case? The ideal scenario (mentioned in T223132) would be:

  • Image caption (Source: unstructured, else structured)
  • Depicts
  • Author
  • Artist
  • Date
  • Source/Photographer
  • Licensing
  • File usage on Wikipedia in the current app language(s) [not relevant for Suggested edit feed cards]
  • More infos (link to file page on Commons) [not relevant for Suggested edit feed cards]

Maybe the engineers can chime in here, but I'd suggest doing a follow-up imageinfo query for this info on a per-image basis for selected candidates, similar to how a page summary request is performed on demand for page previews. Getting unstructured imageinfo for a single image is reasonably fast, but doing so for large numbers in a single request (as I'm doing when generating suggestions) gets to be a problem.

@cooltey and @Dbrant - Thoughts on @Mholloway's comment above?

Johan added a comment.Jun 3 2019, 2:59 PM

Yes, it had crossed my mind. And I'm not overjoyed at the idea of (worst case scenario) showing nudes to a kid or something. So let's please update the API as soon as the NSFW scoring system is online, but meanwhile I think it's an acceptable risk - especially since we are going to prioritise images that are associated with articles, which are not likely to be NSFW. (I mean, there's a nonzero chance, but it is lower than if we just showed a random image from a pool of literally everything.)

@Johan - We should probably put something in the help docs about this, eh?

Yes.

We could also exclude certain Commons categories (including subcategories), if that's simple enough?

Johan added a comment.Jun 3 2019, 3:04 PM

(This is probably cultural, but personally I'd be more concerned about without warning people asking them to caption violence/mutilations/executions than e.g. nudity, as a worst case scenario.)

We could also exclude certain Commons categories (including subcategories), if that's simple enough?

We could get categories for a file from the API, but (there's always a but): the categories returned are only the immediate categories themselves and don't include parent categories. So, for example, File:Flag_of_Panama.svg is a member of Category:Blue,_red,_white_flags and this fact is returned by the API, but it doesn't tell us about the parent categories Category:Tricolor_flags > Category:Color_combinations_of_flags > Category:Flags_by_color > Category:Flags.

In other words, trying to blacklist by category terms (without being overly broad) would be pretty tricky in practice; unfortunately, we can't just blacklist "Category:Sex" and "Category:Violence" and call it a day...

Johan added a comment.Jun 3 2019, 3:56 PM

OK. We could list a number of categories if we wanted, I suppose, to diminish risks; they'd not cover everything but it'd be something to help avoid serving an execution to someone who hadn't been specifically looking for that kind of material as they're helping out on their commute to work.

Johan added a comment.Jun 3 2019, 3:57 PM

If we think it's worth the time.

OK. We could list a number of categories if we wanted, I suppose, to diminish risks; they'd not cover everything but it'd be something to help avoid serving an execution to someone who hadn't been specifically looking for that kind of material as they're helping out on their commute to work.

It depends what categories are available. But remember that we'll only be serving images that are already used in articles - and even articles on e.g., executions don't always have graphic images - for example: https://en.wikipedia.org/wiki/Execution_of_Saddam_Hussein just has a picture of Saddam.

So I'm not sure how many of these we'd be filtering out if we did a category blacklist. I do recommend that we do an experiment though - if we can get a random sample of (say) 100 images using @Mholloway's randomiser and see what percentage of them is likely to cause offence, we might get a better idea of the scale of a potential issue?

It's probably worth mentioning here that the suggestions API currently attempts to get "popular" files (as measured by incoming links) — this was intended to bias the search in favor of images that are likely to be used in articles, but probably also has the happy side-effect of reducing the likelihood of turning up images that would be considered objectionable in most contexts.

It's probably worth mentioning here that the suggestions API currently attempts to get "popular" files (as measured by incoming links) — this was intended to bias the search in favor of images that are likely to be used in articles, but probably also has the happy side-effect of reducing the likelihood of turning up images that would be considered objectionable in most contexts.

I think we should be mostly covered with this. We can also put similar information in the help file, so that people realise we do actually give a damn.

cooltey added a comment.EditedJun 3 2019, 5:08 PM

Actually, I hope to persuade you that it's not worth including any extmetadata fields (which appear structured on the page and in the API, but actually aren't), even the unstructured captions, for the same reason

My latest info on this comes from @JoeWalsh in T223130#5185241, where he suggests to start with these fields: “(...) structured captions, unstructured captions, description, artist, credit, license, and depicts (...).” Is that still the case? The ideal scenario (mentioned in T223132) would be:

  • Image caption (Source: unstructured, else structured)
  • Depicts
  • Author
  • Artist
  • Date
  • Source/Photographer
  • Licensing
  • File usage on Wikipedia in the current app language(s) [not relevant for Suggested edit feed cards]
  • More infos (link to file page on Commons) [not relevant for Suggested edit feed cards]

Maybe the engineers can chime in here, but I'd suggest doing a follow-up imageinfo query for this info on a per-image basis for selected candidates, similar to how a page summary request is performed on demand for page previews. Getting unstructured imageinfo for a single image is reasonably fast, but doing so for large numbers in a single request (as I'm doing when generating suggestions) gets to be a problem.

@cooltey and @Dbrant - Thoughts on @Mholloway's comment above?

@Charlotte
Yes, that's what I've been doing on my working in progress PR, and a follow-up imageinfo query is really doing well.

Thanks for working on this @cooltey, all issues related to this task are listed in T225635. Moving this to QA signoff.

ABorbaWMF added a subscriber: ABorbaWMF.

Looking good so far on 2.7.50282-alpha-2019-06-18

Dbrant closed this task as Resolved.Jun 26 2019, 11:36 AM