Introducing Phatality
Streamlined error reporting from Kibana to Phabricator

Introduction

This past week marks the release of a little tool that I've been working on for a while. In fact, it's something I've wanted to build for more than a year. But before I tell you about the solution, I need to describe the problem that I set out to solve.

Problem

Production errors are tracked with the tag Wikimedia-production-error. As a member of the Release-Engineering-Team, I've spent a significant amount of time copying details from Kibana log entries and pasting into the Production Error Report form here in Phabricator. There are several of us who do this on a regular basis, including most of my team and several others as well. I don't know precisely how much time is spent on error reporting but at least a handful of people are going through this process several times each week.

This is what lead to the idea for rPHAT Phatality: I recognized immediately that if I could streamline the process and save even a few seconds each time, the aggregate time savings could really add up quickly.

Solution

So after considering a few ways in which the process could be automated or otherwise streamlined, I finally focused on what seemed like the most practical: build a Kibana plugin that will format the log details and send them over to Phabricator, eliminating the tedious series of copy/paste operations.

Phatality has a couple of other tricks up it's sleeve but the essence of it is just that: capture all of the pertinent details from a single log message in Kibana and send it to Phabricator all at once with the click of a button in Kibana.

Phatality screenshot showing the submit and search buttons

Clicking the [Submit] button, as seen in the above screenshot, will take you to the phabricator Production Error form with all of the details pre-filled and ready to submit:

Conclusion

Now that Phatality is deployed to production and a few of us have had a chance to use it to submit error reports, I can say that I definitely think it was a worthwhile effort. The Kibana plugin wasn't terribly difficult to write, and thanks to @fgiunchedi's help, the deployment went fairly smoothly. Phatality definitely streamlines the reporting process, saving several clicks each time and ensuring accuracy in the details that get sent to Phabricator. In a future version of the tool I plan to add more features such as duplicate detection to help avoid duplicate submissions.

If you use Wikimedia's Kibana to report errors in Phabricator then I encourage you to look for the Phatality tab in the log details section and save some clicks!

What other repetitive tasks are ripe for automation? I'd love to hear suggestions and ideas in the comments.

Written by mmodell on Mon, Oct 7, 12:36 AM.
Release Engineer, Phabricator Admin
Projects
Subscribers
MarcoAurelio, Krenair, fgiunchedi and 3 others
Tokens
"Doubloon" token, awarded by phuedx."100" token, awarded by MarcoAurelio."Party Time" token, awarded by abi_."100" token, awarded by Neil_P._Quinn_WMF."Meh!" token, awarded by zeljkofilipin."Barnstar" token, awarded by thcipriani."Barnstar" token, awarded by greg."Party Time" token, awarded by jeena."Party Time" token, awarded by dduvall."Like" token, awarded by Mholloway."Love" token, awarded by Michael."Yellow Medal" token, awarded by kostajh."Love" token, awarded by hashar.

Event Timeline

Thanks. I've used it a couple of times to report errors in the Beta Cluster and it's very time-saving indeed.

What other repetitive tasks are ripe for automation? I'd love to hear suggestions and ideas in the comments.

I'd love some of the tasks we do in Cleanup could be automated, such as setting up automated mirrors for gerrit repository creations in Diffusion and GitHub for mediawiki extensions and skins. Registering the submodule on rMEXT and rSKINS would be a plus (it's often forgotten). Similarly, when archiving a Gerrit repo, I'd love to be able to archive Diffusion and delete GitHub mirrors without doing everything manually.