Proposed copy:

• Main copy: Help us make Notifications work better for people like you.
• Secondary copy: How do you use this tool? How could it improve? Please take a survey.

In terms of targeting the survey, Jonathan and I were thinking as follows: a) aim at 2-3 wikis, including at least one were Flow is used a lot and b) aim the survey at users who received a certain number of notifications in the previous month (since the tools we're considering, if they are useful to anyone, will be relevant only to people who actually use Notifications with some frequency).

Nick suggested perhaps limiting the survey requests to people who use the "Automatically enable all new beta features" option because "we know they want to be distracted from normal editing tasks."

Timing: we'd want to do this soon, so that we can be finished before this tool is scheduled to be used to solicit opt-in for the Notifications beta feature (some time in mid to late March).

This plan will be discussed Monday in the Community - Edit Team meeting.

Some thoughts (and a quick mockup):

• The general approach proposed makes sense.
• We may want to use a specific icon to avoid associations with the beta feature, and blindly discarding the invite to the survey thinking it is the same.
• We definitely don't want to ask the same user to take the survey on every different wiki she visits. We need to figure out how to avoid this. In the case of T117669, we are asking to take a specific action on each wiki, but for this case we will be asking to take the exact same survey which may be perceived as spammy to do more than once.
• We may want to adjust the text to be more concise. I tried to reduce the text a bit, but we may even want to go further if we don't want the panel to take too much space (especially when translated text gets longer).
• We need to check some legal considerations about the survey. If I recall correctly, for surveys created with an external tool, a specific label was included to indicate that the user will be using such service (for which we may not be in a position to guarantee the clear non-tracking policies that apply to Wikimedia projects).

@Pginer-WMF I agree we want to avoid showing this across wikis (although if we only have the survey link up for a day or two, which may be enough time to get the data we need, this could be less of an issue).

I've never needed to use a special label when asking people to fill out a Qualtrics survey before (this is the tool we used for pretty much all surveys in Community Engagement), but I do recall there being some pushback about this on at least one previous survey delivered through CentralAuth (maybe?). Since this survey link will be displayed to a large number of users, it makes sense to cover our bases.

@egalvezwmf , @HaithamS , do either of you know any best practices for indicating in the survey invite that clicking the link redirects to an externally-hosted survey?

Hi @Capt_Swing, has legal already reviewed your survey? They typically will offer guidance when they review your survey (which is required for all WMF surveys).

Also, according to a discussion[1] back in November, the text preferred seemed to be: "Survey data handled by a third party (please see [link their site for their privacy policy])" But I would double check with legal.

[1] https://meta.wikimedia.org/wiki/Meta:Requests_for_help_from_a_sysop_or_bureaucrat/Archives/2015-11#Re:_Survey_powered_by_Qualtrics

@Pginer-WMF, the mockup shown above is what people looked at in the Community meeting htis morning. In light of today's discussion in the team meeting, do you propose changes to this design? I.e., to make it look less like a notification and more like...a feedback link (whatever that looks like)?

Update: I've reached out to @APalmer_WMF, and provided her with a copy of the survey questions for review.

In T128937#2096840, @jmatazzoni wrote:

@Pginer-WMF, the mockup shown above is what people looked at in the Community meeting htis morning. In light of today's discussion in the team meeting, do you propose changes to this design? I.e., to make it look less like a notification and more like...a feedback link (whatever that looks like)?

What I was proposing is to use the feedback icon we used in other parts of the UI, while still keeping the style similar to the one used in the rest of the panel (and keeping it attached to the bottom to make it part of the chrome, not the notification content). If we want to go further in that direction, we can make the action more compact and label it explicitly as "feedback":

I think this should make it more obvious that this is an action to provide feedback about this tool and not a general communication channel. Along the lines of the link we provided on Content Translation (T87354) or Media Viewer in the past.

In T128937#2096260, @Capt_Swing wrote:

@Pginer-WMF I agree we want to avoid showing this across wikis (although if we only have the survey link up for a day or two, which may be enough time to get the data we need, this could be less of an issue).

Limiting it in time can definitely help.

I've never needed to use a special label when asking people to fill out a Qualtrics survey before (this is the tool we used for pretty much all surveys in Community Engagement), but I do recall there being some pushback about this on at least one previous survey delivered through CentralAuth (maybe?). Since this survey link will be displayed to a large number of users, it makes sense to cover our bases.

As I mentioned, I just recalled some discussions. I'm not sure which was the final decision on that.
From the practical side, I'm mainly interested in knowing (a) whether I have to update the designs to accommodate such text and (b) which is the text we need if any.

What I was proposing is to use the feedback icon we used in other parts of the UI, while still keeping the style similar to the one used in the rest of the panel

That's a good call @Pginer-WMF. I think you're moving in the right direction here. I'd like to run a revised design past the Community meeting on Monday morning. But before I do, I'd like to see what we can do to make it even clearer that this isn't a notification.

I think one problem here is that the invitation is right next to the content queue so it can look like it's part of the content. I took the liberty of mocking up what it would look like if we put the invitation at the bottom instead.

It's just one idea, though I hope you'll consider it and/or improve upon it. Let's try to have something we can put forward Monday AM. Thanks .

I understand the goals for this as follows (in order):

1. Make users aware that they can provide feedback about how notifications work for them.
2. Don't be distracting or break the user workflows. We don't want users to be distracted form the urgent work they may be doing.
3. Don't give the impression that this is a general purpose communication channel that other extensions should be able to use.

Footers and headers provide a reference for orientation (by being at the top and the bottom of the panel in this case). If we place something else as the very bottom element where the footer used to be we may be forcing users to reorient. I think we can get more problems by getting in the usual user way (2) than by concerns about it being a new communication channel (3).

I think we need to consider 3, but not be driven only by that. I doubt there will be any confusion with the proposed state and, given the temporary nature of the feature, it will be solved by removing the feature after a while (which is the plan already). Also, anticipating those issues (or the lack of) would be good for a similar feature such as T117669.

Hey @Pginer-WMF, in today's team meeting we agreed this can move forward. We also heard a little bit of discomfort with the way the invitation looks a bit like a notification. Did you have any ideas for altering the design somewhat to mitigate that? Or should we move forward with this?

I think the current approach should work. We can emphasise the call to action in bold as we do for UI actions:

Another option is to make it even more compact which may make it closer to actions like those in the footer while still being aligned with the general design of the panel. The only drawback is that we may need to reduce the invitation text which was helpful to clarify the purpose and motivate . I tried to make it compact while still preserving the basics: it is a feedback mechanism, it's about notifications and leads to a survey; but there is no room for the motivation/encouragement ("help make it better").

I think we may get more participation with the former, but if there are some concerns with that I'm also ok with the latter approach.
In any case, the icons are the ones below:

• Feedback icon:
  feedback.svg1 KB

  Download
• "Info" and "close" icons are available as such in the icon repo.

@Pginer-WMF @jmatazzoni looking good! A few questions and requests:

1. per @egalvezwmf's comment above, we need to add "Survey data handled by a third party (please see their privacy policy)" somewhere in the solicitation.
2. please let me know before you make this live, so that I can make sure the survey is ready for action!
3. we need to be ready to turn this notification off on fairly short notice (less than 24 hours). This is in case we get a huge flood of responses and are in danger of going over the response-limit of our Qualtrics account. If we exceed this limit, we will be charged for any additional responses. Since this is the first time we've sent out a survey through this channel, it's hard to gauge what the level of response will be, so we need to remain vigilant.
4. which wikis do you plan to release this survey link on?

In T128937#2123174, @Capt_Swing wrote:

@Pginer-WMF @jmatazzoni looking good! A few questions and requests:

1. per @egalvezwmf's comment above, we need to add "Survey data handled by a third party (please see their privacy policy)" somewhere in the solicitation.

Sorry, I didn't understood clearly from the comment that the text should be on the invite (and not in the survey itself). I'll update the designs to reflect that.

Actually @Pginer-WMF, I've received new information from @APalmer_WMF that suggests perhaps we don't need to place this text in the body of the notification. I've posed the question to her directly and should hear back soon. You can hold off 'til then.

Thanks for your new designs Pau. I feel this one hits a very nice balance of looking harmonious in the Notifications panel but not looking like a notification. It does an excellent job, I believe, of clear and inviting and looking like what it is -- a feedback link.

https://phab.wmfusercontent.org/file/data/pwtyftqj4rglmjn25gvd/PHID-FILE-ld7g74rh4yciubx2re43/Panel-invite-feedback-one-liner.png

I am going to try to get together all the information we need to make this happen and put this out as a spec.

SPEC: FEEDBACK LINK IN NOTIFICATIONS PANEL
First Deployment, Invitation for Notifications Survey Participation

• Design mockup: https://phab.wmfusercontent.org/file/data/pwtyftqj4rglmjn25gvd/PHID-FILE-ld7g74rh4yciubx2re43/Panel-invite-feedback-one-liner.png
• Wording: Provide feedback: take this quick survey about notifications.
  • Note: this should be written in such a way that longer translations will simply wrap to a second line.
• Icon: feedback icon.
• Survey Link: https://wikimedia.qualtrics.com/SE/?SID=SV_eRMpgEPhVcUEkn3
• Dismiss [X]: clicking the X permanently dismisses the invitation for that user on that wiki.
• Information [?] tooltip/hover text: This feedback link is designed to help Wikimedia developers learn more about how people use Notifications so we continue to improve the tool. Clicking the survey link will send you to a third-party website.
• Proposed target audience:
  • Show the invitation to users on all English wikis (to start) WHO...
  • ...currently have more than 5 unread notifications.
  • Note: we must be able to disable the invitation quickly when we’ve reached our survey limit.
• Closing the survey:
  • Once we’ve reached the limit, a custom message will be displayed that says something like “This survey is now closed. If you would like to share your feedback about Notifications, you can do it here: https://www.mediawiki.org/wiki/Talk:Notifications”
  • Once we’ve reached the limit, Jonathan will email e2@lists.wikimedia.org to request the survey link be removed.
  • Once we evaluate the survey results, we will follow this up with another round aimed at French speakers.

• Timing: Once this is ready, we’ll schedule a time for it to go live (to give Jonathan time to turn on the survey).

I added the description at the top of the ticket (combined with some of the info I added before) so that it is easier to find.

@Quiddity asks about the following:

We'd also discussed showing the invitation to: "...currently opted-in to the 'Automatically enable new beta features" user preference.
Is it possible to do a mix of these?

Nick, Pau and Jonathan and I talked about this in the design meeting this morning. We understand the concern but are worried this would cut down the number of people who see this too severely. According to the figures Neil pulled for us recently, only about 3.3% of enwiki users have > 5 unread messages at any time -- about 8K folks, more or less. At typical pickup reates, Jonathan thinks that is probably about the number who need to see this to get the 1000 we need.

Change 278208 had a related patch set uploaded (by Mooeypoo):
Add footer notice widget for invitations for feedback

https://gerrit.wikimedia.org/r/278208

Change 278208 merged by jenkins-bot:
Add footer notice widget for invitations for feedback

https://gerrit.wikimedia.org/r/278208

Change 279256 had a related patch set uploaded (by Catrope):
Add footer notice widget for invitations for feedback

https://gerrit.wikimedia.org/r/279256

Change 279256 merged by jenkins-bot:
Add footer notice widget for invitations for feedback

https://gerrit.wikimedia.org/r/279256

I just saw this on test. Please make the words "Provide feedback." bold.

Also, @Pginer-WMF , the implementation on test uses an i-in-a-circle "information" icon instead of the ? Help icon. Information doesn't seem wrong. What do you want?

I prefer the "?". I think the icon code for it is "help", not "info" (as I indicated in the ticket description by mistake). I cannot check it in the icon repo since seems to be down right now.
It is the one used on the Notification Page currently, and I think it translates slightly better to other languages than "i"

Change 279439 had a related patch set uploaded (by Catrope):
Enable $wgEchoFooterNotice on enwiki

https://gerrit.wikimedia.org/r/279439

Looking at this on beta, I see one last thing: the "Dismiss" X is black in color instead of the gray that all the other Xs are. It should be the same color.

Change 279439 merged by jenkins-bot:
Enable $wgEchoFooterNotice on enwiki

https://gerrit.wikimedia.org/r/279439

@Pginer-WMF, @jmatazzoni take a look at the most recent in test.wikipedia.org,
close-up

overall look

in betalabs - the text is bolded

Note:

1. In the tooltip text, shouldn't be there a comma before so?

<a class="oo-ui-buttonElement-button" role="button" title="This feedback link is designed to help the developers learn more about how people use Notifications so we continue to improve the tool. Clicking the survey link will send you to a third-party website." tabindex="0" aria-disabled="false" rel="nofollow"><span class="oo-ui-iconElement-icon oo-ui-icon-help">

2. Does not keep track of users that already filled out the survey - the invitation will be back when a user starts a new session. Especially, since the invitation appears only when there are 5 un-read notifications, it may seem to be random to users.

3. Alerts show the invitations when there are 5 unread alerts just as Messages do - although the counter will not show 5 unread alerts due to the face that the counter gets to zero when Alerts open. Since it's hard to figure out the correlation between 5 unread notifications and the invitation display - it's ok.

Everything else - keeping track of unread 5 messages/alerts, re-directing to qualtrics, and the invitation being completely dismissed by clicking on the X button - is fine.

I tried it on beta. For the time restricted exposure we are planning I think it already works well.

So I'd just provide some feedback in case we really want to spend some time polishing rough corners:

• The speaker icon looks a bit bigger compared to the text. Currently is about 30px and making it 25px would keep it more balanced.
• It is a bit weird that clicking on the info icon does nothing. The original idea was for it to link to a page (e.g., maybe this one but with a more end-user focused introduction). I recall some discussion about this, but I'm not sure of the final conclusion. If we want to just provide a quick note, the tooltip mechanism alone could work but it would be good to have a tooltip that remains visible all the time you are over the icon and also shows on click (which the default one of the browser does not).
• The tooltip text seems a bit too long. I tried to make it more compact: "This survey will help us learn more about how people use Notifications, so we continue to improve the tool. Clicking the link will send you to a third-party website."

@jmatazzoni I agree that we should put in a test for edit count, and since we're explicitly targeting experienced editors here 100 sounds good.

I also agree that it would be useful to surface this on other English projects (the survey is already set up to capture what projects people are editing, so we should be able to segment responses from different projects).

I don't know whether 1 is "too low" or not... I'll leave the final call up to you!

Change 280078 had a related patch set uploaded (by Mooeypoo):
Change parameters for user survey footer message

https://gerrit.wikimedia.org/r/280078

Change 280078 merged by jenkins-bot:
Change parameters for user survey footer message

https://gerrit.wikimedia.org/r/280078

Change 280112 had a related patch set uploaded (by Catrope):
Change parameters for user survey footer message

https://gerrit.wikimedia.org/r/280112

Change 280115 had a related patch set uploaded (by Catrope):
Enable Echo footer notice on all English language wikis

https://gerrit.wikimedia.org/r/280115

Change 280112 merged by jenkins-bot:
Change parameters for user survey footer message

https://gerrit.wikimedia.org/r/280112

Change 280115 merged by jenkins-bot:
Enable Echo footer notice on all English language wikis

https://gerrit.wikimedia.org/r/280115

Changes deployed: reduced the notification threshold to 2, added an edit count requirement of 100, and enabled it on the other English wikis.

Still seeing very low response rate for the survey. Can we please reduce threshold to 1 soonest?

Change 280367 had a related patch set uploaded (by Catrope):
Change threshold for survey invitation from 2 unread notifs to 1

https://gerrit.wikimedia.org/r/280367

Change 280367 merged by jenkins-bot:
Change threshold for survey invitation from 2 unread notifs to 1

https://gerrit.wikimedia.org/r/280367

Change 280369 had a related patch set uploaded (by Mattflaschen):
Change threshold for survey invitation from 2 unread notifs to 1

https://gerrit.wikimedia.org/r/280369

Change 280370 had a related patch set uploaded (by Mattflaschen):
Change threshold for survey invitation from 2 unread notifs to 1

https://gerrit.wikimedia.org/r/280370

Change 280370 merged by jenkins-bot:
Change threshold for survey invitation from 2 unread notifs to 1

https://gerrit.wikimedia.org/r/280370

Change 280369 merged by jenkins-bot:
Change threshold for survey invitation from 2 unread notifs to 1

https://gerrit.wikimedia.org/r/280369

French translation will be provided before Friday.

French translation is ready. I'm waiting for the Privacy Statement to be put on line, and then I'll upload the French translation to Qualtrics.

Next message will be when it's done.

I just noticed a small problem and am moving this back to In Development: The invitation panel includes a ? (Information) icon. The icon has a highlight state, and holding the hover brings up an informational message about the invitation. But if a user CLICKS on the icon, the message never appears -- even if the user continues to hold the hover after clicking.

I'd submit that clicking on this icon is a pretty natural impulse. Since we're going to repurpose this panel for other message, it seems worthwhile to fix this small bug. Can we make it so that the info message will appear on hover and/or click?

French translation is up on Qualtrics, and privacy statement is available in French (and linked from survey). This survey is ready to be deployed on French-language Wikimedia wikis.

I'll make a new task for the French survey.

Change 282430 had a related patch set uploaded (by Mattflaschen):
Also show the survey tooltip on click

https://gerrit.wikimedia.org/r/282430

Change 282430 merged by jenkins-bot:
Also show the survey tooltip on click

https://gerrit.wikimedia.org/r/282430

Change 282715 had a related patch set uploaded (by Mattflaschen):
Also show the survey tooltip on click

https://gerrit.wikimedia.org/r/282715

Change 282715 merged by jenkins-bot:
Also show the survey tooltip on click

https://gerrit.wikimedia.org/r/282715