Brief Summary

This session aims to provide an open space for discussion around the reliability of Tools hosted on Toolforge, and what can be done on a platform level to improve the experience for maintainers. The WMF Developer Experience team is inviting new and experienced Toolforge users to contribute their view by sharing their own vision for what reliability should look like for Tools, so we can refine our understanding of how the platform can provide visibility into issues and enable maintainers to keep their tools running with confidence by providing improved monitoring and observability features.

Long description

Problem definition (5 minutes)
Presenters will make a quick 5 minute introduction to the subject, sharing our objectives, why we're working on this, and how we're approaching the topic. At the end, we'll propose a few questions as a starting point to guide the discussion with the audience.

Group discussion (45 minutes)
Depending on the number of participants in the session, we'll use either a Fishbowl format or breakout groups, with the aim of hearing from as many people and perspectives as possible. The goal is to answer the questions proposed during the introduction, or come up with our own questions and answers related to the topic.

Final debrief (10 minutes)
To finish up the session, we'll “merge” the discussions from the Fishbowl or breakout sessions into the meeting notes, which will contain a summary of the input from the community on the topic of Reliability on Toolforge.

Expected outcomes

We expect to leave this session with a “wishlist” of features for reliability and observability of Toolforge-hosted tools, which we'll iterate over internally and prioritize accordingly, identifying the gaps that need to be addressed along the way.

Why are you proposing this project?

Reliability of Toolforge-hosted tools is one of our focus areas for FY26-27 within Developer Experience and the recently-established Tools Platform team. So we're looking to leverage this unconference session for open and honest feedback on what the community expects from the platform, and what their vision of reliability tooling on Toolforge could look like.

Resources

• Slides
• Etherpad

---

Raw session notes

Unconference session: Tools That (Don’t) Break: A Toolforge Reliability Wishlist

Speaker: Arthur Puthin (User:APuthin-WMF), Chris Ciufo (User:CCiufo-WMF)

Notes:

The purpose of this session is to get the wishes regarding tools on Toolforge.

The growth and health of project rely on tools and bots run by volunteers, it is more and more critical that such tools and bot can run without problems to ensure nothing breaks on the projects.

Observability and lack of metrics is a common pain point, feeling the absence of something that can help in this matter.

We want to move Toolforge towards a model wher you spend less time checking if your tool is alive and running.

Fishbowl session:

• the difficulty of knowing what tools already exists
  • would you be interested in contributing to an existing tool instead of creating one?
    • [missing answer]
• we use tools a lot in our work, we are proud of tools getting fixed
• we have some information, ToolHub was built to kinda solve this problem, but it's not solved yet; the first thing that was trying to solve was what tool existed, period, with a JSON with very bareboned data about the tools; eventually, we tried to productionise that, what I hear here is the need to next level, somehow the community and others to give actionable feedback on existing tools, possibly evaluate these tools to provide guidance
• there's the angle of what can we ask to the tool creator in terms of interesting info, many people just do the tool to solve individual problems, we might ask people to include also other people's needs
• there are people who ragequit when asked to prove that their tool is alive and running, there are a lot of social rather than technical problems in maintaining a tool, the major fault of ToolHub is that no one is using that
• how do we now which tools are "good"? we need some form of community voting system
• would the social feature create the right pressure to keep the tools alive?
  • https://wikitech.wikimedia.org/wiki/User:BryanDavis/Developing_community_norms_for_critical_bots_and_tools
  • the hypothesis is that social pression is the good way to modify the attitude of a tool maintainer, and part of this is receiving appreciation and accolades for their work; having more than one maintainer, documentation and whatnot is for the "definition of done", if you want to do the right thing, do these things
• Tools that need maintainers board on Phabricator: has anyone been there? Could we have an "adopt a tool" system?
  • This is a question from end-users, not developers
  • Magnus' tools are a pain point in the community, people cannot work on a patch when he's busy, he's the one doing them
    • we're back to the social side, patches to tools are harder to push because there is no established user who could push them like in MediaWiki
  • when a tool gets traction, how do we encourage maintainers to collaborate more? your job changes from creating the tool to maintaining the community around that tool. it's like an IC being promoted to a manager, and if you don't understand it you're going to have a bad time.
  • is there a culture of sunsetting a tool? some tools are totally abandoned. should it be declared to the public as abandoned?
    • there is a "delete tool" button that will archive it. in toolhub you can point from a tool to a different tool. but we don't have real lifecycle management for tools. There is also the right to fork on Toolforge, you can hunt down the dev and get the source code to fork, and there's the abandoned tool, if broken and the dev is not responding, then you can begin a process where you become a co-maintainer and after some period you can also boot the original maintainer
    • There's also https://wikitech.wikimedia.org/wiki/Help:Toolforge/Toolforge_standards_committee to facilitate the processes above
• Getting back to the technical side, main pain point for me is logs. It would be good to post logs to a grafana instance so that it can be easy to search through the logs. Using grep and less is ok, but not intuitive. I need to log into the tool. Having a logs UI would be useful.
  • We have some starter part of that, the output of jobs by default is now going to Loki, a hopefully scalable data storage for logs. The two major challenges have been scaling the storage backend, and multitenancy. You don't want other people to see your tool's logs. Most of the open source solutions don't consider multitenancy.
• Would it be possible to have a container we can launch in our own cluster to manage logs?
  • A better solution could be that once we have all the logs streaming into loki, we can build our own wrapper framework to deliver tenanted logs in a custom Toolforge UI.
• Want to programmatically manage jobs via a public API.
  • We will let you do this through the eventual Toolforge UI, while for API/CLI access from outside of the bastion we will need to settle on a solution for authentication.
  • For a git push-to-deploy based workflow, that is coming through the components service (although web services is not available yet, but coming soon™).
• How do you know your tool/bot has stopped working?
  • Example of a bot which makes edits to Commons. Monitor the changes feed to see if the bot has gone silent.
• What about security issues that exist across tools? How do we manage getting those patched?
  • It will be relatively easy to scan images for tools that are on the build service.
  • Can we prompt tool maintainers to upgrade their tools when there are vulnerabilities or deprecated/outdated dependencies.