Page MenuHomePhabricator

Structured tasks: temporary free text for "other" rejection reason
Closed, ResolvedPublic

Description

In T301884: Add a link: rejection reasons, we found that "Other" was chosen as the rejection reason for link rejections about 14% of the time, more commonly than another of the enumerated options, "text should include more or fewer words". We have also learned from consulting with newcomers heavily using the feature that they wish they could type in their own response when they select "Other".

We want to allow a free text field for "Other" in a limited capacity for our learning purposes. It might help us identify if there is a rejection reason we should add or an improvement to make to the algorithm. This would be the limited capacity:

  • On both the "images" and "links" tasks.
  • Only for our four pilot wikis and French, where we have staff who can read the responses (ES, AR, CZ, BN, FR).
  • Only for a limited amount of time (Disable week of July 18, 2022)

The reason we want this to be limited are:

  • We don't have plans to be able to use this free-text field in an ongoing way, and so we don't want to cause newcomers to spend time needlessly.
  • There is a risk that users type text in the field that would need to be acted upon by Trust & Safety, and there is not a sustainable way to patrol that text.
Design

image.png (2×3 px, 345 KB)

https://www.figma.com/file/2SONd8P1tsexIB5coMOp8h/Add-links-structured-task-v1.0?node-id=2883%3A54766

Implementation notes
Add a link

Since the user can return to the suggestion and change the rejection reason, we also need to store the text the user entered for "other" in order to show it again.

Add an image

We don't need the same mechanism for storing the response as add a link since the rejection automatically submits the response.

Data storage

Reason will be sent in action_data field for impression and close events with rejection_dialog active interface.

"action_data": "link_target=Integrita dat;link_text=integrity dat;probability_score=0.7947532534599304;series_number=0;rejection_reason=other;acceptance_state=rejected"
Deployed wikis

TBD

Acceptance criteria
  • When rejecting a suggestion, selecting "Other" in the rejection reason dialog should enable the text input field. De-selecting the field should disable the text input field.
  • "impression" and "close" events for "rejection_dialog" active_interface should include the entered reason in action_data if "rejection_reason" includes "other".
  • When returning to a previously rejected link suggestion, clicking on the ellipsis button in the link inspector should bring up the rejection dialog with the previously entered text (if text was entered).

Event Timeline

@RHo -- could you please add quick designs here for desktop and mobile?

Also, everyone should note that this is not part of T300851: [EPIC] Growth: "add a link" structured task 2.0 -- it can be prioritized separately or in the future.

@RHo , should we remove this task from the current sprint board? Are you planning to work on this any time soon?

RHo added a subscriber: KStoller-WMF.

@RHo , should we remove this task from the current sprint board? Are you planning to work on this any time soon?

Hi @MShilova_WMF - sorry I should have updated this, the designs are done (adding in the task description) and it is ready for dev. However, it can be moved off if not a priority. So a question for @KStoller-WMF perhaps?

This seems like a good chance to learn more about each structured task and see if we should add more "rejection reasons" or even adjust the algorithm. However, it's also temporary code that will need to be reverted, so it doesn't seem like it's worth a lengthy engineering effort if this will be very complex to implement.
@kostajh do you have an effort estimate for this so we can better prioritize?

This seems like a good chance to learn more about each structured task and see if we should add more "rejection reasons" or even adjust the algorithm. However, it's also temporary code that will need to be reverted, so it doesn't seem like it's worth a lengthy engineering effort if this will be very complex to implement.
@kostajh do you have an effort estimate for this so we can better prioritize?

It seems like something that would occupy an engineer's time for 2-3 days. Things we'd have to do:

  • config variable (or hardcode) the list of wikis we want this feature to deploy to
  • swap out the existing "Other" field with the new one
    • would we want new instrumentation for this field or should we just pretend that it is the existing "other" field for events logged by interacting with it?
  • store the text of the field somewhere -- do you want this stored in a user_property or should it go into data logged via EventLogging?

After writing out the above, I think putting the answers into the EventLogging action_data is probably the simplest thing, although I would need to double check the maximum length for data that we could add there.

  • store the text of the field somewhere -- do you want this stored in a user_property or should it go into data logged via EventLogging?

We can't store it in user properties, it is per-task. I agree EventLogging seems like the way to go.

Hi @nettrom_WMF, I wanted to confirm with you the additional field we will be sending for this. I was thinking of adding other_reason field to action_data so that the possible responses for rejection_reason field are uniform regardless of whether this free text field is enabled or not. Does that sound good to you?

For wikis without the text field

"action_data": "link_target=Integrita dat;link_text=integrity dat;probability_score=0.7947532534599304;series_number=0;rejection_reason=other;acceptance_state=rejected"

For wikis with the text field

"action_data": "link_target=Integrita dat;link_text=integrity dat;probability_score=0.7947532534599304;series_number=0;rejection_reason=other;acceptance_state=rejected;other_reason=customreason"

Hi @KStoller-WMF — could you confirm the wikis where this change should be deployed to? I know the description mentions "our four pilot wikis and French" but let's spell this out since the list does change over time so I wanted to be sure.

Change 805226 had a related patch set uploaded (by MewOphaswongse; author: MewOphaswongse):

[mediawiki/extensions/GrowthExperiments@master] Create SelectWithTextInputWidget

https://gerrit.wikimedia.org/r/805226

Hi @KStoller-WMF — could you confirm the wikis where this change should be deployed to? I know the description mentions "our four pilot wikis and French" but let's spell this out since the list does change over time so I wanted to be sure.

@mewoph Sure, good point.
I updated the description and added the wiki language codes: ES, AR, CZ, BN, FR

Change 805418 had a related patch set uploaded (by MewOphaswongse; author: MewOphaswongse):

[schemas/event/secondary@master] Add other_reason action_data to image_suggestion_interaction and link_suggestion_interaction schemas

https://gerrit.wikimedia.org/r/805418

Change 805456 had a related patch set uploaded (by MewOphaswongse; author: MewOphaswongse):

[mediawiki/extensions/GrowthExperiments@master] Structured task: Support free text for "other" rejection reason

https://gerrit.wikimedia.org/r/805456

Task typeDesktopMobile
Add image
Screen Shot 2022-06-14 at 9.47.28 AM.png (864×1 px, 114 KB)
Screen Shot 2022-06-14 at 9.31.55 AM.png (1×752 px, 168 KB)
Add link
Screen Shot 2022-06-14 at 9.45.56 AM.png (764×1 px, 126 KB)
Screen Shot 2022-06-14 at 9.33.12 AM.png (1×752 px, 174 KB)

Sample events:

Add image

event: {
  "action": "close",
  "action_data": "filename=Building_33_with_Hangar_1_in_the_background._View_toward_northwest_-_Naval_Air_Station_Moffett_Field,_Hanger_No._1,_Cummins_Avenue,_Moffett_Field,_Sunnyvale,_Santa_Clara_County,_CA_HAER_CA-335-A-4.tif;recommendation_source=commons;recommendation_source_projects=;series_number=1;total_suggestions=2;rejection_reasons=other;acceptance_state=rejected;other_reason=other reason",
  "is_mobile": true,
  "active_interface": "rejection_dialog",
  "newcomer_task_token": "fp866dankmtjm9lmleptbk79qc4brh9t",
  "page_id": 1799,
  "page_title": "Moffett Field airship hangars",
  "homepage_pageview_token": "bll7c558nmi3cust7iv4jrhsrq84a15p",
  "$schema": "/analytics/mediawiki/structured_task/article/image_suggestion_interaction/1.4.0",
  "meta": {
    "stream": "mediawiki.structured_task.article.image_suggestion_interaction",
    "domain": "localhost"
  },
  "dt": "2022-06-14T16:27:53.862Z"
}

Add link

event: {
  "action": "close",
  "action_data": "link_target=Metafora;link_text=metafor;probability_score=0.5916731953620911;series_number=0;rejection_reason=other;acceptance_state=rejected;other_reason=other reason",
  "is_mobile": true,
  "active_interface": "rejection_dialog",
  "newcomer_task_token": "9rc4veeilj8mc2ra7ro33fgh66rs9si4",
  "page_id": 181,
  "page_title": "Sobecký gen",
  "homepage_pageview_token": "t9o30gjjs3tigh8eq6493irh7mt2aafr",
  "$schema": "/analytics/mediawiki/structured_task/article/link_suggestion_interaction/1.7.0",
  "meta": {
    "stream": "mediawiki.structured_task.article.link_suggestion_interaction",
    "domain": "localhost"
  },
  "dt": "2022-06-14T16:44:09.053Z"
}

Change 805480 had a related patch set uploaded (by MewOphaswongse; author: MewOphaswongse):

[operations/mediawiki-config@master] Structured task: enable free text for "other" rejection reason

https://gerrit.wikimedia.org/r/805480

Change 805418 merged by jenkins-bot:

[schemas/event/secondary@master] Add other_reason action_data to image_suggestion_interaction and link_suggestion_interaction schemas

https://gerrit.wikimedia.org/r/805418

Change 805226 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Create SelectWithTextInputWidget

https://gerrit.wikimedia.org/r/805226

Moving to QA, though we still need to enable in beta labs for QA (I left a comment in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/805480) and then to enable in group2 next Thursday after the train arrives.

Change 805456 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Structured task: Support free text for "other" rejection reason

https://gerrit.wikimedia.org/r/805456

Change 807576 had a related patch set uploaded (by MewOphaswongse; author: MewOphaswongse):

[operations/mediawiki-config@master] Structured task: enable free text for "other" rejection reason

https://gerrit.wikimedia.org/r/807576

Change 805480 merged by jenkins-bot:

[operations/mediawiki-config@master] Structured task: enable free text for "other" rejection reason in betalabs

https://gerrit.wikimedia.org/r/805480

Change 805480 merged by jenkins-bot:

[operations/mediawiki-config@master] Structured task: enable free text for "other" rejection reason in betalabs

https://gerrit.wikimedia.org/r/805480

This is now deployed, so we can QA the "other" rejection reason on any of the beta labs wikis, cc @Etonkovidova

Change 805480 merged by jenkins-bot:

[operations/mediawiki-config@master] Structured task: enable free text for "other" rejection reason in betalabs

https://gerrit.wikimedia.org/r/805480

This is now deployed, so we can QA the "other" rejection reason on any of the beta labs wikis, cc @Etonkovidova

Checked in beta - all looks as expected; moving to Test in Production.

Only for a limited amount of time (likely one week).

@KStoller-WMF @nettrom_WMF we are enabling this today and the functionality will be available when wmf.18 is in group2, in a couple of hours. Should we plan to disable it 1 week from now, or perhaps on July 11 so there is at least a full week of data?

Change 807576 merged by jenkins-bot:

[operations/mediawiki-config@master] Structured task: enable free text for "other" rejection reason

https://gerrit.wikimedia.org/r/807576

Mentioned in SAL (#wikimedia-operations) [2022-06-30T13:13:06Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: fb399065b123db536ae244a0c0fada61eb906a6e: Structured task: enable free text for "other" rejection reason (T304099) (duration: 03m 46s)

@nettrom_WMF can override me here, but I would suggest disabling ~July 18th, so we have plenty of data and we aren't asking that engineers work on Phab tasks during Inspiration Week.

Checked in production (wmf.18) - all look as expected; keeping the task in Test in Production to check after it'll be disabled.

Note: active_interface: 'rejection_dialog' is present in action: 'confirm_reject_suggestion' and action: 'close'

action: 'confirm_reject_suggestion', action_data: 'filename=French_connection_statue.JPG;recommendati…other;acceptance_state=rejected;other_reason=test', is_mobile: false, active_interface: 'rejection_dialog'

action: 'close', action_data: 'filename=French_connection_statue.JPG;recommendati…other;acceptance_state=rejected;other_reason=test', is_mobile: false, active_interface: 'rejection_dialog'

Change 814708 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] Structured task: Disable free text for "other" rejection reason

https://gerrit.wikimedia.org/r/814708

Change 814708 merged by jenkins-bot:

[operations/mediawiki-config@master] Structured task: Disable free text for "other" rejection reason

https://gerrit.wikimedia.org/r/814708

Mentioned in SAL (#wikimedia-operations) [2022-07-18T07:54:23Z] <kharlan@deploy1002> Synchronized wmf-config: Config: [[gerrit:814708|Structured task: Disable free text for "other" rejection reason (T304099)]] (duration: 02m 41s)

Etonkovidova claimed this task.