This ticket is covering toolabs, so maybe we don't need to include it for this one
https://phabricator.wikimedia.org/T95155

Meeting with Learning and Evaluation scheduled for Tuesday, Aug 4, 2pm in Diderot.

Possible questions:

To what extent are you satisfied with the WMF meeting the following technical needs of your (language) community? (Very dissatisfied to very satisfied)

• Bug fixes and maintenance of existing software
• Developing new features or software that the community needs
• Communicating about technical issues
• Documentation
• Language-specific issues (user interface localization, input support, etc.)
• Appropriate prioritization of technical work (?)

How long have you been an editor/contributor on X project?

What is your gender?

Updated question list:

To what extent are you satisfied with the WMF meeting the following technical needs of [your language] community? (very dissatisfied to very satisfied)

• Bug fixes and maintenance of existing software
• Developing new features or software that the community needs
• Communicating about technical issues
• Documentation
• Language-specific issues (user interface localization, input support, etc.)

How satisfied are you overall with the technical support provided by the WMF to [your language] community? (very dissatisfied to very satisfied)

About how many years have you been an editor or contributor on [your language] project?

Which gender do you identify with?

@Doc_James, @Harej, @Jalexander, @MZMcBride, @Ricordisamoa, @Bmueller: Any important broad areas of technical support that I'm missing from this question list? (I don't want to get too specific because people hate taking long surveys and most people won't understand things like 'API support').

How do you plan on administering this survey?

@Harej: That's not completely finalized yet. Related tasks are:

• T107829 - Generate list of core community members to target for technical support satisfaction survey
• T107982 - Figure out system for inviting core community members to take satisfaction survey

Any suggestions are welcome at those tasks.

I guess the question I have is how useful is a very dissatisfied or very satisfied response?

Phrasing question as "Do you have any issues with X: yes/no" If yes what potential solutions do you think would solve this? Anyway just a thought.

In T107217#1519993, @kaldari wrote:

Any important broad areas of technical support that I'm missing from this question list?

I perceive "Communicating about technical issues" and "Documentation" as being too vague.
Also, assistance in housekeeping with local scripts and gadgets is quite different from maintaining 'real' software and users should be questioned separately about it.

PS: an "undecided" option is needed.

@Ricordisamoa Do you have any suggestions for better wording for "Communicating about technical issues" or "Documentation"? Should we just remove those? For documentation, I'm mainly thinking about things like API documentation, end-user extension documentation, ResourceLoader documentation, Tool Labs documentation, etc. For communication, I'm thinking about things like API changes, extension roll-outs, getting technical feedback from the community, etc.

I'll see if I can come up with a way to split out housekeeping for local scripts and gadgets. Keep in mind that this survey will be going out to editors who are not necessarily savvy about things like user scripts and site JS, so we don't want it to be too heavy on technical details.

@Doc_James: The main purpose of this survey is to get quantitative data on satisfaction levels that we can compare year to year. That said, I'll suggest your idea and see what the Learning and Evaluation people think about it. The downside is that it would expand the cost and scope of the survey dramatically, as we would need to have responses translated from 9 different languages and we would have to figure out how to report on the responses both internally and to the community. It might be better if we first identify which areas have strong dissatisfaction and then solicit community feedback on those specific areas.

@kaldari: I would suggest to add an "ideas for improvement" field to each category. This would give you the chance to get a deeper insight in what is specifically not working well and how this could be improved. I also would suggest not only to ask how long someone has been an editor/contributor, but also in which project and in which role the person is mainly active (e.g. mainly editor of Spanish Wikipedia or mainly active in support work on Commons etc.). Different user groups might have different needs and also different access to or interest in informations on tech issues etc.

In T107217#1526020, @kaldari wrote:

@Ricordisamoa Do you have any suggestions for better wording for "Communicating about technical issues" or "Documentation"? Should we just remove those?

Communication and documentation may as well be cross-topic matters, e.g.:

But some issues could belong to several rows/columns, so a table may not fit.

In T107217#1526051, @kaldari wrote:

The main purpose of this survey is to get quantitative data on satisfaction levels that we can compare year to year.

As communities' needs change, it's not guaranteed you will get meaningful comparisons.
Verbal feedback should be seriously evaluated.

@Ricordisamoa: I don't think the survey software that Learning and Evaluation are using for this survey supports a table format like that, at least not a table that can have multiple options per cell, i.e. very satisfied to very unsatisfied + undecided in each cell. @egalvezwmf would know better than I do though.

This survey shouldn't have more than 10 questions at the most, as it's supposed to be a "quick survey". The WMF is going to be doing more extensive surveys later.

@Ricordisamoa, @Doc_James As @kaldari mentioned, I'm not sure that a matrix like that would work. That being said, we can ask different questions around these topics. So rather than only asking about satisfaction, we can ask how involved users are in each area, to see how large that volunteer community is.

About the wording of the areas, we can give examples, using (e.g...) at the end of each area to explain what we mean by "communications of technical issues" and "documentation", or we can rephrase of course. For example:

• Communication of technical issues (e.g. through tech news, blog, mailing lists, etc.)
• Documentation (e.g. end-user documentation, API documentation, etc.)

And I'm just realizing we can change the question so its more about needs being met. So:
To what extent is the WMF meeting the following technical needs of your (language) community over [time frame]?
Not at all
Slightly
Moderately
Mostly
Completely
Not applicable

Thanks!
E

@egalvezwmf Those suggestions sound good to me. Let's try to finalize the wording by the end of this week so we can move ahead with getting things translated.

I might repeat what's been written already.
I assume we are after good questions that encourage useful and thoughtful answers.

How satisfied are you overall with the technical support provided by the WMF to [your language] community?

Can respondents express their self-understanding of their "I'm mostly doing X" role in the project to allow differentiating satisfaction by activity groups (readers, editors, ...) and/or language groups (french, german, ...) and/or site groups (commons, wiktionary, ...)?

Bug fixes and maintenance of existing software

Areas welcome examples/clarification. Given that example, does that also cover "The templates and gadgets on our wiki break from time to time!" that the WMF might not even feel responsible for? Do we assume that respondents know which fields WMF might feel (not) responsible for, as the survey is about being "satisfied with the WMF"?

(very dissatisfied to very satisfied)

What is the full list of wordings/strings for the available answer options?

How will the results be analyzed and published?

Survey seems to only allow explaining how (un)satisfied respondents are but not why. Both respondents and respondents' needs will change over the years. Hence not covering the "why" might make it harder to get "satisfaction levels that we can compare year to year" (emphasis by me). Of course "why" would require evaluating free-text answers which takes way more time and it's been written this could be a potential follow-up step.

Can respondents express their self-understanding of their "I'm mostly doing X" role in the project to allow differentiating satisfaction by activity groups (readers, editors, ...) and/or language groups (french, german, ...) and/or site groups (commons, wiktionary, ...)?

Is this a question or a suggestion? If it's a question, the answer is currently no, maybe yes, and maybe yes (depending on what you're asking). Honestly, I'm not sure I really understand your question. I can parse it several different ways, but I'm not sure which is correct. Could you clarify?

Areas welcome examples/clarification. Given that example, does that also cover "The templates and gadgets on our wiki break from time to time!" that the WMF might not even feel responsible for? Do we assume that respondents know which fields WMF might feel (not) responsible for, as the survey is about being "satisfied with the WMF"?

I think many of the respondents will assume that the WMF doesn't feel responsible for a lot of the things they care about. They are free to express that as dissatisfaction. Whether or not we should care about those things is a much bigger debate that this survey is not designed to address.

What is the full list of wordings/strings for the available answer options?

@egalvezwmf Can you answer this one?

How will the results be analyzed and published?

That's covered by T107984

Survey seems to only allow explaining how (un)satisfied respondents are but not why.

That's correct. It's only intended as a simple temperature guage. Evaluating free-text responses in 9 languages and reporting on them is a very large undertaking. That said, I will discuss the possibility with the Learning and Eval team since several folks have suggested it here.

What is the full list of wordings/strings for the available answer options?

Actually, it looks like Edward already answered that above:
Not at all
Slightly
Moderately
Mostly
Completely
Not applicable

In T107217#1536184, @kaldari wrote:

Can respondents express their self-understanding of their "I'm mostly doing X" role in the project to allow differentiating satisfaction by activity groups (readers, editors, ...) and/or language groups (french, german, ...) and/or site groups (commons, wiktionary, ...)?

Is this a question or a suggestion? If it's a question, the answer is currently no, maybe yes, and maybe yes (depending on what you're asking). Honestly, I'm not sure I really understand your question. I can parse it several different ways, but I'm not sure which is correct. Could you clarify?

Will I be able to express {role, language, site}, by answering "I'm mostly an editor* on French* Wiktionary*" or "I'm mostly a template-developer* on German* Wikipedia*". (strings with stars stand for answer options).
As a variety of personas have a variety of different technical needs. Not sure how much you want to differentiate though, might be out of scope again for this survey.

@Akalpper To help answer your question, the survey will be going out to specific editors in 10 projects, so we will be able to know who is very active in which project, based on which language version of the survey they fill out. I'm pretty sure we are piloting this only with Wikipedia for this round. Also, as I mentioned previously:

we can ask different questions around these topics. So rather than only asking about satisfaction, we can ask how involved users are in each area, to see how large that volunteer community is.

I'm not sure how important this is for this survey; but I'm pretty sure only want to measure satisfaction/whether needs are being met for this round.

Per feedback from @Ricordisamoa and @egalvezwmf (in person), I've revised the topic areas to be more specific and more focused on power-user tasks (since this is more aligned with the Community Tech team's scope). How does this list sound:

• Reviewing recent changes
• Reviewing new articles
• Identifying and surfacing content problems
• Supporting WikiProjects
• Managing media
• Working with templates
• Categorizing content
• Supporting gadgets, bots, and 3rd party tools

Am I missing anything?

Also, here's the latest wording for the radio button choices:
Not at all
Slightly
Moderately
Mostly
Completely
I have no opinion

@kaldari, could you keep the content of the survey updated in the task description, please? This way it is easier to get an idea of the status of the survey text, and it is easier to apply improvements.

According to T107983#1509738, you are planning to make an extensive promotion of this survey in multiple languages, which is great. It looks like most (all?) of the replies expected are based on selecting options as opposed of writing text, which will help non-native English speakers. Is there a plan to allow these communities to translate the survey?

Also, I wonder whether this survey could be used to identify open tasks in your backlog that people care about, asking something like

Identify the 1-3 tasks most important for you in this list (add the task number as in "T12345")

1. 2. 3.

This question in the survey would have a double purpose, identify popular tasks but also introduce your backlog to these users, helping them understand how you work and how to get involved beyond the survey.

Which survey engine are you planning to use?

@Qgil: answers below...

According to T107983#1509738, you are planning to make an extensive promotion of this survey in multiple languages, which is great. It looks like most (all?) of the replies expected are based on selecting options as opposed of writing text, which will help non-native English speakers. Is there a plan to allow these communities to translate the survey?

Yes, translation is already underway.

Also, I wonder whether this survey could be used to identify open tasks in your backlog that people care about, asking something like...

You idea has merit, but I don't want to ask people's opinions on tasks if we aren't actually going to use that input for prioritization, which we aren't because we've already completed prioritization analysis of our backlog per the process published at https://www.mediawiki.org/wiki/Community_Tech_team/All_Our_Ideas/Process. There will be plenty of opportunity for community input on tasks during the wishlist survey though (https://www.mediawiki.org/wiki/Community_Tech_team/Community_Wishlist_Survey). Also, I'm trying really hard to keep this survey as simple as possible. It was originally going to be just a single question, but now it's 10 questions. It's also significantly behind schedule :(

Which survey engine are you planning to use?

Qualtrics, as documented at https://meta.wikimedia.org/wiki/Research:Tech_support_satisfaction_poll#Hosting ;)

Current state of the translations:

• Arabic: 4 sections need to be updated
• German: done
• Spanish: done
• French: 1 section needs to be updated
• Japanese: done
• Portuguese: 1 section needs to be updated
• Russian: 4 sections need to be updated
• Chinese: 4 sections need to be updated

I've re-pinged translators for Arabic, French, Portuguese, Russian, and Chinese.

All the translations are finished! Yay!