Page MenuHomePhabricator

[SPIKE] What could be contributing to the drop-off in the Reply Tool comment funnels?
Closed, ResolvedPublic

Description

This task is about investigating what – if any – technical issues could be contributing to the drop-off between the saveintent and savesuccess steps in the Reply Tool comment funnels. E.g. might people be encountering errors when attempting to save the comment they posted with the Reply Tool? Might the tool/server timeout when people are attempting to save the comment they posted with the Reply Tool? etc.

Background

In T247139, we are looking at how people are engaging with the Reply Tool at the Arabic, Czech and Hungarian Wikipedias...wikis where the tool is available to everyone by default.

As part of the analysis we are doing in T247139 [i], we noticed that a non-trivial percentage of people who reached the saveintent step within the Reply Tool comment funnel (read: they clicked the Reply button) did not reach the savesuccess step (read: the comment they drafted in earlier steps was successfully published to the talk page).

You can observe this for yourself by reviewing the following:

Reply Tool modesaveintent --> savesuccess drop-off rangeData
sourceDrop-off ranges from 0% (0 edits group) to 7% (1-4 edits group)Comments made with wikitext
visualDrop-off ranges from 0% (0 edits group) to 6% (1-4 edits group)Comments made with VisualEditor

Done

  • Document whether there are any observable technical issues that could be contributing to the drop-off rates mentioned in the ===Background section above

i. You can view the full report here: https://nbviewer.jupyter.org/github/wikimedia-research/Discussion-tools-analysis-2020/blob/master/Engagement-Metrics/Reply-Tool-Workflow-Engagement-Metrics.ipynb#By-Wiki

Event Timeline

ppelberg moved this task from Incoming to Upcoming on the Editing-team (FY2021-22 Kanban Board) board.
ppelberg updated the task description. (Show Details)

Wherever the saveSuccess event is missing, there should instead be a saveFailure event, which should contain the error details. We should analyze those. I don't remember right now where to access the data.

OK, I figured out where I can access it. For future reference: https://superset.wikimedia.org/superset/sqllab → presto_analytics_hive → event.

Then I ran this query:

SELECT event.save_failure_type, event.save_failure_message, count(*) as count
from editattemptstep
where event.integration='discussiontools' and event.action='saveFailure'
group by event.save_failure_type, event.save_failure_message
order by count desc

Here's the result, with notes from me:

save_failure_typesave_failure_messagecountNotes
responseUnknownblocked20User is blocked. Looks like we show the reply tool interface to blocked users, and only fail when they try posting (and the message lacks details). We should improve this.
image.png (757×2 px, 100 KB)
responseUnknownabusefilter-warning15Warning from an abuse filter. As this is only a warning, the user should be able to click "Reply" again to post their comment.
extensionCaptchacaptcha15CAPTCHA (heh, I expected this to be the top one). Usually triggered by trying to post an external link in the comment. The user should be able to post their comment after filling out the CAPTCHA.
responseUnknowndiscussiontools-commentid-notfound14"Could not find the comment you're replying to on the page." The comment could have been deleted while the user was writing their reply (or it could be a bug on our side).
extensionAbuseFilterabusefilter-disallowed11Disallowed by abuse filter.
responseUnknownapierror-visualeditor-docserver-http8Internal error where we couldn't contact Parsoid.
responseUnknownhttp7Usually means that the user's network connection got disconnected (and, if this entry arrived in our database, that means it re-connected later and they were hopefully able to retry and post their comment).
responseUnknownwikimedia-globalblocking-ipblocked-range2User's IP address is globally blocked.
responseUnknownassertanonfailed2The user started replying while logged out, then logged in in another browser tab, and we show them a message to confirm they know which username they're posting under. The user should be able to click "Reply" again to post their comment.
extensionSpamBlacklistspamblacklist2Disallowed by spam blacklist.
responseUnknownlatestnotfound1Internal error where the latest revision of the page they tried posting on doesn't exist? Pretty weird, maybe that page got deleted. I'd expect that to give a different error message that says so directly, though.

(That's the total number of times each type of error happened, not the number of sessions or users who experienced it. It might be just one person who was blocked and clicked the button 20 times before they gave up. I don't want to spend too much time on this, but it should be possible to do this better or merge it with the report somehow.)

(These are all the details we have. We don't log the actual error message the user saw, or further details like which abuse filter was triggered or which comment they were replying to. The schema EditAttemptStep doesn't have fields for anything else, they could be added but it would take a bit of work.)

OK, I figured out where I can access it. For future reference: https://superset.wikimedia.org/superset/sqllab → presto_analytics_hive → event.

This is great, @matmarex.

Then I ran this query:

SELECT event.save_failure_type, event.save_failure_message, count(*) as count
from editattemptstep
where event.integration='discussiontools' and event.action='saveFailure'
group by event.save_failure_type, event.save_failure_message
order by count desc
  • @matmarex: To confirm: the query above searches all edits that have ever been made with discussiontools, correct?

Note: here is a permalink to this query for us to reference in the future: https://superset.wikimedia.org/superset/sqllab?savedQueryId=178.

Here's the result, with notes from me:

save_failure_typesave_failure_messagecountNotes
responseUnknownblocked20User is blocked. Looks like we show the reply tool interface to blocked users, and only fail when they try posting (and the message lacks details). We should improve this.
image.png (757×2 px, 100 KB)

See T270803.

responseUnknownabusefilter-warning15Warning from an abuse filter. As this is only a warning, the user should be able to click "Reply" again to post their comment.

This doesn't seem too concerning to me considering I expect people who are new to trigger abuse filters more often than people who have more editing experience.

extensionCaptchacaptcha15CAPTCHA (heh, I expected this to be the top one). Usually triggered by trying to post an external link in the comment. The user should be able to post their comment after filling out the CAPTCHA.

Same as above RE newer people.

responseUnknowndiscussiontools-commentid-notfound14"Could not find the comment you're replying to on the page." The comment could have been deleted while the user was writing their reply (or it could be a bug on our side).
  • @matmarex: does this need a ticket? If so, I can file. Note: I'm not too concerned about this considering how infrequently this seems to be happening.
extensionAbuseFilterabusefilter-disallowed11Disallowed by abuse filter.

Same as above RE newer people.

responseUnknownlatestnotfound1Internal error where the latest revision of the page they tried posting on doesn't exist? Pretty weird, maybe that page got deleted. I'd expect that to give a different error message that says so directly, though.
  • @matmarex: What is the error message that is currently shown?

(That's the total number of times each type of error happened, not the number of sessions or users who experienced it. It might be just one person who was blocked and clicked the button 20 times before they gave up. I don't want to spend too much time on this, but it should be possible to do this better or merge it with the report somehow.)

Noted. This is a great starting place. I've created a "saveFailure incidences" sheet within the existing DiscussionTools usage report workbook; I'll re-run the query very two weeks for the next month or so and see if we see much movement in these numbers.

(These are all the details we have. We don't log the actual error message the user saw, or further details like which abuse filter was triggered or which comment they were replying to. The schema EditAttemptStep doesn't have fields for anything else, they could be added but it would take a bit of work.)

Understood. I don't think this is necessary just yet.

  • @matmarex: To confirm: the query above searches all edits that have ever been made with discussiontools, correct?

No, it looks like it searches the last 90 days (running select min(dt) from editattemptstep returns '2020-10-13' right now). Older data is probably deleted permanently (per https://meta.wikimedia.org/wiki/Data_retention_guidelines#How_long_do_we_retain_non-public_data?).

responseUnknowndiscussiontools-commentid-notfound14"Could not find the comment you're replying to on the page." The comment could have been deleted while the user was writing their reply (or it could be a bug on our side).
  • @matmarex: does this need a ticket? If so, I can file. Note: I'm not too concerned about this considering how infrequently this seems to be happening.

No, comments being deleted is normal, although uncommon.

responseUnknownlatestnotfound1Internal error where the latest revision of the page they tried posting on doesn't exist? Pretty weird, maybe that page got deleted. I'd expect that to give a different error message that says so directly, though.
  • @matmarex: What is the error message that is currently shown?

"Could not find latest revision for title"

I tried now and I can consistently reproduce this by opening the reply widget, deleting the talk page (or moving it without creating a redirect), then trying to save a reply.

  • @matmarex: To confirm: the query above searches all edits that have ever been made with discussiontools, correct?

No, it looks like it searches the last 90 days (running select min(dt) from editattemptstep returns '2020-10-13' right now). Older data is probably deleted permanently (per https://meta.wikimedia.org/wiki/Data_retention_guidelines#How_long_do_we_retain_non-public_data?).

Got it.

responseUnknowndiscussiontools-commentid-notfound14"Could not find the comment you're replying to on the page." The comment could have been deleted while the user was writing their reply (or it could be a bug on our side).
  • @matmarex: does this need a ticket? If so, I can file. Note: I'm not too concerned about this considering how infrequently this seems to be happening.

No, comments being deleted is normal, although uncommon.

Understood.

responseUnknownlatestnotfound1Internal error where the latest revision of the page they tried posting on doesn't exist? Pretty weird, maybe that page got deleted. I'd expect that to give a different error message that says so directly, though.
  • @matmarex: What is the error message that is currently shown?

"Could not find latest revision for title"

I tried now and I can consistently reproduce this by opening the reply widget, deleting the talk page (or moving it without creating a redirect), then trying to save a reply.

I see.

  • And to be sure: title in the message above would be the title of the page the person is attempting to save a comment to, not the word "title," right?
  • And to be sure: title in the message above would be the title of the page the person is attempting to save a comment to, not the word "title," right?

No, that is literally the error message.

  • And to be sure: title in the message above would be the title of the page the person is attempting to save a comment to, not the word "title," right?

No, that is literally the error message.

Understood.

Next steps

  • @ppelberg create new ticket for copy change and associate this to-be-created ticket with T241403.

create new ticket for copy change and associate this to-be-created ticket with T241403.

I filed T273068.

Thank you, Bartosz.