Page MenuHomePhabricator

Pre-deployment regression testing (topic subscriptions)
Closed, ResolvedPublic

Description

This task is about ensuring Manual Topic Subscriptions (T263820)

Testing timing

Testing instructions

Meta

1. Enabling topic subscriptions

2. Disabling topic subscriptions

3. Verify feature appears as expected
Exhaustively test Special:Preferences to ensure topic subscriptions appears/does not appear in expected ways. Where "exhaustively" means doing things like:

  • If the DiscussionTools beta feature is enabled in Special:Preferences#mw-prefsection-betafeatures the Enable topic subscription appears enabled within Special:Preferences#mw-prefsection-editing
  • If the Enable topic subscription is enabled within Special:Preferences#mw-prefsection-editing, all ==H2== headings on talk pages should have a Subscribe link/affordance within them
  • If the Enable topic subscription is disabled within Special:Preferences#mw-prefsection-editing, NO ==H2== headings on talk pages should have a Subscribe link/affordance within them

4. Verify new comment notifications are delivered through the proper channels
This grouping refers to verifying new comment notifications are delivered through the channels "enabled" for Talk page subscription in Special:Preferences#mw-prefsection-echo.

5. Verify new comment notifications are received for comments, regardless of the interface used to publish them

6. Verify new comment notifications are no longer sent when someone unsubscribes from a section

  • Verify Unsubscribing from the talk page works as expected
  • Verify Unsubscribing from within Echo on mobile and desktop works as expected (T279150)

7. Verify new comment notifications are bundled as expected
See: T275949#6980357.

8. Verify new comment notifications are only sent when new comments are posted
E.g. a notification SHOULD be sent when someone posts a new comment with full-page editing or the Reply Tool. A notification should NOT be sent when someone makes an edit to an existing comment with full-page editing.

9. Verify content within the new comment notifications in Echo is displaying as expected
E.g. Username of comment's author, a portion of what's written in the new comment person is being notified about, section title, etc.

10. Verify the calls to actions within the new comment notifications in Echo function as expected
E.g. links to author's user page, link to diff, link to precise location on talk page where comment was posted, etc.

11. Ensure unsubscribe affordance disappears from within Notifications after having unsubscribed from a topic

12. De-duplicating pings
When a new comment is posted in a topic you are subscribed and that "new comment" contains a ping with your username in it, verify you only receive one notification for this comment: a notification within Alerts and not a new comment notification in Notices.

Requirements

  • Execute the steps listed in the Testing instructions section above
  • File tickets for any issues that surface while executing the steps listed in the Testing instructions section above

Responses from @ppelberg and @matmarex to issues uncovered (June 23)

Note: the only blocker among the issues above is the New comment notifications are not being delivered via email when that channel is enabled for talk page; the remaining issues can be addressed after the feature is made available as a beta feature at the first set of wikis.

Related Objects

Event Timeline

Updated task description to include the two additional test cases [i][ii] that emerged during today's team meeting.


i. 11. Ensure unsubscribe affordance disappears from within Notifications after having unsubscribed from a topic
ii. 12. De-duplicating pings

@Ryasmeen
Hi Rummana
Can you please raise the questions that were uncovered in last week's testing? TY!

@Ryasmeen
Hi Rummana
Can you please raise the questions that were uncovered in last week's testing? TY!

Sure.

  1. New comment notifications are not being delivered via email when that channel is enabled for talk page.
  1. When you click on a notification item, it takes you to the correct position on the talk page and highlights the comment. However, the highlight remains despite moving focus to elsewhere on the page. That means, even when you click on a Reply link for another comment on that talk page, the highlight stays for the notified comment, thus making it look confusing. It only goes away when a new comment gets highlighted after posting or after refreshing the page. cc: @iamjessklein
  1. Also noticed the counter-intuitive and confusing Echo behavior, where:

    a. Notifications shift order based on read status instead of just inverting the colors?

    b. Notification bundles disappear/gets unbundled after reading all the comments under them which can also be confusing.

cc: @DLynch, @matmarex, @Esanders.

Couple more observations:

When you click on a notification item, it takes you to the correct position on the talk page and highlights the comment. However, the highlight remains despite moving focus to elsewhere on the page. That means, even when you click on a Reply link for another comment on that talk page, the highlight stays for the notified comment, thus making it look confusing. It only goes away when a new comment gets highlighted after posting or after refreshing the page.

  1. The highlight actually remains even after refreshing the page now, it only gets cleared when the page gets updated in some way.
  1. When multiple comments posted by the same user in quick succession has same timestamp, the subscribed user only gets notified for the first comment.
  1. When I clicked on a notification bundle, it took me to the latest reply under that subscribed topic correctly. However, there was one instance where it scrolled to the second comment of that bundle (it had three notifications under it). It only happened once, I will see if I can reproduce this issue again.
  1. After clicking on Unsubscribe for a single notification of a bundle on the notification panel, it removes the more options icon. I find it unusual to remove a UI element from notification panel after interacting with it, specially if it is categorized as one of the triage actions. But then, the Unsubscribe option stays for the other comments under that notification bundle for the same subscribed topic. Clicking on them just shows the same dialog "You have unsubscribed" on top right corner. This seems redundant. Can we improve this workflow by either not removing this option at all or by removing that option for other comment notifications under the same topic as well after unsubscribing.

Also, after clicking Unsubscribe from the notification panel it does not remove the "Unsubscribe" link on the talk page next to the section title. That is also inconsistent and confusing. Refreshing the page however changes the "Unsubscribe" link to the "Subscribe" and also removes the Unsubscribe option from other notifications for that topic.

  1. After clicking on Unsubscribe for a single notification of a bundle on the notification panel, it removes the more options icon. I find it unusual to remove a UI element from notification panel after interacting with it, specially if it is categorized as one of the triage actions.

I like that on mobile view we grouped "View Changes" and "Unsubscribe" together under the "More options", that way even if we remove the option to "Unsubscribe" after unsubscribing, the icon for More Options (. . .) stays, which minimizes the unexpected visual UI change.

Alternatively, have we considered changing the "Unsubscribe" option to "Subscribe" once we click on it? I can see a scenario where someone might accidentally click on it and might want to subscribe back on the same panel.

cc: @iamjessklein.

  1. When multiple comments posted by the same user in quick succession has same timestamp, the subscribed user only gets notified for the first comment.

And this happens even when the comments belong to two separate subscribed topics/threads.

  1. New comment notifications are not being delivered via email when that channel is enabled for talk page.

It works for me – I just got an email for this edit: https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Talk:Main_Page&oldid=prev&diff=497325&diffmode=source

image.png (508×1 px, 46 KB)

Can you check the following:

  1. When you click on a notification item, it takes you to the correct position on the talk page and highlights the comment. However, the highlight remains despite moving focus to elsewhere on the page. That means, even when you click on a Reply link for another comment on that talk page, the highlight stays for the notified comment, thus making it look confusing. It only goes away when a new comment gets highlighted after posting or after refreshing the page. cc: @iamjessklein

This is intended, but maybe we should reconsider it. @Tacsipacsi also complained about it on the implementation task (T281471):

Expected
[…]
The highlight […] never fades.

I don’t like when such highlights are “stuck” on the page until I navigate away; maybe I already do something entirely different. I like GitHub’s approach: when I open an anchor link (e.g. https://github.com/wikimedia/parsoid/pull/5#issuecomment-371981298), it highlights the comment, this highlight doesn’t fade out automatically, but it disappears immediately when I click anywhere on the page outside of that comment.

That's interesting, I never noticed that before. I think we're open to changing that, although it might also be weird in some cases (e.g. if the highlight would disappear when you click [reply]).

Also, to clarify, "the highlight never fades" only means that it's not on a timer (it doesn't fade by itself), but it is removed if you navigate to a different anchor on the page, e.g. by clicking a section in the table of contents.

  1. When multiple comments posted by the same user in quick succession has same timestamp, the subscribed user only gets notified for the first comment.

And this happens even when the comments belong to two separate subscribed topics/threads.

This seems to be a known issue, our code notes that:

includes/Notifications/EventDispatcher.php
				// Compare comments by name, as ID could be changed by a parent comment
				// being moved/deleted. The downside is that multiple replies within the
				// same minute will only fire one notification.
				count( $oldParser->findCommentsByName( $newComment->getName() ) ) === 0

But I think we can actually improve it, by checking if there are more comments now than there were before, rather than only checking if there are any now and zero before. There may still be edge cases where a second notification does not happen (e.g. if someone removes a comment and adds a new one in the same edit), but it would be better.

Also, after clicking Unsubscribe from the notification panel it does not remove the "Unsubscribe" link on the talk page next to the section title. That is also inconsistent and confusing. Refreshing the page however changes the "Unsubscribe" link to the "Subscribe" and also removes the Unsubscribe option from other notifications for that topic.

This is known as T284322.

Regarding all other notifications weirdnesses you point out, all of them are consistent with the current behavior of notifications; we did not implement most of those behaviors ourselves, they already existed in Echo, and we've reused them (we've also reused some things from Flow's notifications, like the unsubscribe action). We can change a lot of that, but we should probably change it for all notifications (or at least ours and Flow's), and it would be a more difficult project for us than the other changes proposed here.

(I guess next step is for Peter to read all of the above, make decisions on which changes we should prioritize and maybe file tickets.)

Just to agree, we should definitely distinguish between issues in Notifications and issues in our-notifications. 😁

LZaman updated the task description. (Show Details)
LZaman updated the task description. (Show Details)
LZaman updated the task description. (Show Details)

Meta

  1. New comment notifications are not being delivered via email when that channel is enabled for talk page.

It works for me – I just got an email for this edit: https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Talk:Main_Page&oldid=prev&diff=497325&diffmode=source

image.png (508×1 px, 46 KB)

Can you check the following:

The email address I have provided is not yet confirmed; however, I have yet to receive a confirmation email (I checked the spam folder) despite having done the following:

  1. Clicked the Confirm your email address button that appears in Special:Preferences#mw-prefsection-personal
  2. In response to not receiving an email after doing "1.", I changed the email address associated with my account and followed the confirm email steps provided in the flow.

...@matmarex can you think of other things I can try to cause an email confirmation email to be sent?


Note: I am checking this as Rummana as out today and tomorrow.

I have a confirmed e-mail address on the Beta Cluster, and – after I remembered to go to https://en.wikipedia.beta.wmflabs.org/wiki/Special:Preferences#mw-prefsection-echo and turn on e-mail notifications for "Talk page subscription" – I immediately received an e-mail notification triggered by an edit from my other account.

I have a confirmed e-mail address on the Beta Cluster, and – after I remembered to go to https://en.wikipedia.beta.wmflabs.org/wiki/Special:Preferences#mw-prefsection-echo and turn on e-mail notifications for "Talk page subscription" – I immediately received an e-mail notification triggered by an edit from my other account.

Noted. This leads me to think that the issue may in fact lie with the email confirmation workflow. As such, I've filed a task for the issue I experienced here: T285527.

I'm going to resolve this task considering:

  • A) Regression testing is complete
  • B) Tickets have been filed for the issues that surfaced during regression testing