Page MenuHomePhabricator

If aborting a scap due to test canary error rate, output some errors for reference
Open, MediumPublic

Description

Let's admit it, we're lazy.

Beta breaks, we click on the IRC link, and then onto Console Output

22:06:34 22:06:34 Executing check 'Logstash Error rate for deployment-mediawiki04.deployment-prep.eqiad.wmflabs'
22:06:34 22:06:34 Check 'Logstash Error rate for deployment-mediawiki04.deployment-prep.eqiad.wmflabs' failed: ERROR: 69% OVER_THRESHOLD (Avg. Error rate: Before: 0.36, After: 12.00, Threshold: 3.62)
22:06:34 
22:06:34 22:06:34 Unhandled error:
22:06:34 Traceback (most recent call last):
22:06:34   File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 300, in run
22:06:34     exit_status = app.main(app.extra_arguments)
22:06:34   File "/usr/lib/python2.7/dist-packages/scap/main.py", line 395, in main
22:06:34     return super(Scap, self).main(*extra_args)
22:06:34   File "/usr/lib/python2.7/dist-packages/scap/main.py", line 102, in main
22:06:34     '(rerun with --force to override this check)' % failed)
22:06:34 RuntimeError: 1 test canaries had check failures (rerun with --force to override this check)
22:06:34 22:06:34 scap failed: RuntimeError 1 test canaries had check failures (rerun with --force to override this check) (duration: 01m 36s)

Why doesn't it give me a sample of the errors? :)

Event Timeline

thcipriani triaged this task as Medium priority.Mar 20 2017, 3:33 PM
thcipriani moved this task from Needs triage to Debt on the Scap board.
thcipriani subscribed.

It does seem like showing some errors would be helpful; although possibly a little tricky to get at that information.

The reason scap doesn't output any error is because it's actually using another script call logstash_checker.py (https://github.com/wikimedia/puppet/blob/production/modules/service/files/logstash_checker.py) and doesn't have access to the errors.

logstash_checker.py itself is just looking at histogram Δ so it doesn't really know what the errors are either at this point. Seems like this is something logstash_checker.py could do.