Page MenuHomePhabricator

Consider making a variant of the fatalmonitor CLI tool that ignores appserver timeouts
Closed, DeclinedPublic

Description

[I appreciate that this is a niche request for a mostly-ignored tool.]

As a near-continuous user of the fatalmonitor CLI tool, I'd love either a different version that hides appserver timeouts from the display (or ideally an interactive way to show/hide within the one tool).

The Kibana dashboard lets you do a bunch of this filtering, but it is (a) a very heavyweight browser instance that one can't keep open permanently, unlike a shell window, and (b) far away from the console where one deploys.

As I write this, of the top 50 grouped entries on fatalmonitor, over 20 groups (and over 300 production fatals) are of the form "entire web request took longer than 60 seconds" and similar, which obscures "real" issues, which include bad arguments to hooks, DB query timeouts, and mysterious blank lines presumably caused by T157396. Obviously when deploying, a spike of timeouts could itself be indicative of a problem, so I wouldn't want fatalmonitor hobbled to ignore timeouts entirely, but right now I find that sometimes my eyes glaze over issues I should look into.

Event Timeline

CDanis triaged this task as Medium priority.Jan 15 2019, 2:10 PM
• lmata subscribed.

Fatalmonitor no longer actively supported: https://wikitech.wikimedia.org/wiki/Wikimedia_binaries#fatalmonitor
Untagging observability, please re-tag if you need our assistance.

LSobanski subscribed.

Since the linked comment is: "(fatalmonitor) No longer exists - see logspam-watch instead" I would consider this as a won't fix.