Page MenuHomePhabricator

Set scap.cfg's canary_dashboard_url to useful beta logstash url
Closed, ResolvedPublic

Description

https://integration.wikimedia.org/ci/job/beta-scap-eqiad/160323/console

19:55:26 19:55:26 Executing check 'Logstash Error rate for deployment-mediawiki04.deployment-prep.eqiad.wmflabs'
19:55:26 19:55:26 Check 'Logstash Error rate for deployment-mediawiki04.deployment-prep.eqiad.wmflabs' failed: ERROR: 50% OVER_THRESHOLD (Avg. Error rate: Before: 0.03, After: 2.00, Threshold: 1.00)
19:55:26 
19:55:26 19:55:26 scap failed: average error rate on 1/1 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details)
19:55:26 19:55:26 Unhandled error:
19:55:26 Traceback (most recent call last):
19:55:26   File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 308, in run
19:55:26     exit_status = app.main(app.extra_arguments)
19:55:26   File "/usr/lib/python2.7/dist-packages/scap/main.py", line 427, in main
19:55:26     return super(Scap, self).main(*extra_args)
19:55:26   File "/usr/lib/python2.7/dist-packages/scap/main.py", line 117, in main
19:55:26     raise RuntimeError(canary_fail_msg)
19:55:26 RuntimeError: scap failed: average error rate on 1/1 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details)
19:55:26 19:55:26 scap failed: RuntimeError scap failed: average error rate on 1/1 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) (duration: 01m 43s)

It probably should be linking to beta logstash on beta, not production

Event Timeline

greg renamed this task from beta-scap-eqiad failed and linked to production logstash to Set scap.cfg's canary_dashboard_url to useful beta logstash url.Jul 7 2017, 10:50 PM
greg triaged this task as Medium priority.

Change 378750 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[operations/puppet@production] Beta: Scap: canary_dashboard_url to beta logstash

https://gerrit.wikimedia.org/r/378750

Change 378750 had a related patch set uploaded (by Jcrespo; owner: Thcipriani):
[operations/puppet@production] Beta: Scap: canary_dashboard_url to beta logstash

https://gerrit.wikimedia.org/r/378750

Change 378750 merged by Jcrespo:
[operations/puppet@production] Beta: Scap: canary_dashboard_url to beta logstash

https://gerrit.wikimedia.org/r/378750