Between 2017-11-20 and 2017-12-01, the Wikimedia Foundation ran a direct response user survey of registered Toolforge users. 141 email recipients participated in the survey which represents 11% of those who were contacted.

Demographic questions

Based on responses to demographic questions, the average [1] respondent:

• Has used Toolforge for 1-3 years
• Developed 1-3 tools & actively maintains 1-2 tools
• Spends an hour or less a week maintaining their tools
• Programs using Python and/or PHP
• Does 80% or more of their development work locally
• Uses source control
• Was not a developer or maintainer on Toolserver

[1]: "Average" here means a range of responses covering 50% or more of responses to the question. This summarization is coarse, but useful as a broad generalization. Detailed demographic response data is available on wiki.

Qualitative questions

• 90% agree that services have high reliability (up time). Up from 87% last year.
• 78% agree that it is easy to write code and have it running on Toolforge. Up from 71% last year.
• 59% agree that they feel they are supported by the Toolforge team when they contact them via cloud mailing list, #wikimedia-cloud IRC channel, or Phabricator. This is down dramatically from 71% last year, but interestingly this question was left unanswered by 36% of respondents.
• 59% agree that they receive useful information via cloud-announce / cloud mailing lists. Up from 46% last year.
• 52% agree that documentation is easy to find. This is up from 46% last year and the first time crossing the 50% point. We still have a long way to go here though!
• 96% find the support they receive when using Toolforge as good or better than the support they received when using Toolserver. Up from 89% last year.
• 50% agree that Toolforge documentation is comprehensive. No change from last year.
• 53% agree that Toolforge documentation is clear. Up from 48% last year.

Free form responses

The survey included several free form response sections. Survey participants were told that we would only publicly share their responses or survey results in aggregate or anonymized form. The free form responses include comments broadly falling into these categories:

• Documentation (58 comments)
• Platform (48 comments)
• Workflow (48 comments)
• Community (17 comments)
• Support (6 comments)

Documentation

Comments on documentation included both positive recognition of work that has been done to improve our docs and areas that are still in need of additional work. Areas with multiple mentions include need for increased discoverability of current information, better getting started information, and more in depth coverage of topics such as wiki replica usage, Kubernetes, and job grid usage.

There were also comments asking for a self-service version control system and pre-installed pywikibot software. Both of these are offered currently in Toolforge, so these comments were classified as missing or difficult to find documentation.

Platform

Comments about the Toolforge platform have been subcategorized as follows:

• Software (26 comments)
  • The majority of software comments were related to a desire for newer language runtime versions (PHP, Java, nodejs, Python) and more flexibility in the Kubernetes environment.
• Database (10 comments)
  • Database comments include praise for the new Wiki Replica servers and multiple requests for a return of user managed tables colocated with the replica databases.
• Reliability (10 comments)
  • Reliability comments included praise for good uptime, complaints of poor uptime, and requests to improve limits on shared bastion systems.
• Hardware (2 comments)
  • (sample too small to summarize)

Workflow

• Deploy (12 comments)
  • The major theme here was automation for software deployment including requests for full continuous delivery pipelines.
• Debugging (10 comments)
  • People asked for better debugging tools and a way to create a more full featured local development environment.
• Monitoring (10 comments)
  • Monitoring comments included a desire for alerting based on tracked metrics and tracking of (more) metrics for each tool.
• Setup (10 comments)
  • Comments included praise for https://toolsadmin.wikimedia.org/ improvements and other work done in last year.
• Files (6 comments)
  • Improved workflows for remote editing and file transfer are desired.

Community

Comments classified as community related broadly called for more collaboration between tool maintainers and better adherence to practices that make accessing source code and reporting bugs easier.

Support

Support related comments praised current efforts, but also pointed to confusion about where to ask questions (irc, email, phabricator).