Page MenuHomePhabricator
Create Task
Maniphest T269672

Traceback in icinga-status 'Host' object has no attribute 'downtime'
Closed, ResolvedPublic
Actions

Description

icinga-status currently runs into a traceback, which breaks the reboot cookbook (since it executes that script):

jmm@alert1001:~$ /usr/local/bin/icinga-status -j auth1002.eqiad.wmnet
Traceback (most recent call last):
  File "/usr/local/bin/icinga-status", line 286, in <module>
    raise SystemExit(main())
  File "/usr/local/bin/icinga-status", line 281, in main
    print(json_dumps(hosts, default=lambda o: o.to_json()))
  File "/usr/lib/python3.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/bin/icinga-status", line 281, in <lambda>
    print(json_dumps(hosts, default=lambda o: o.to_json()))
  File "/usr/local/bin/icinga-status", line 107, in __json__
    'downtime': self.downtime,
AttributeError: 'Host' object has no attribute 'downtime'

Details

SubjectRepoBranchLines +/-
icinga: add support for downtimed and notifications_enabled parametersoperations/software/spicerackmaster+17 -4
icinga_status: remove values from json until support added to spicerackoperations/puppetproduction+2 -6
icinga_status: fix type downtimed != downtimeoperations/puppetproduction+1 -1
Customize query in gerrit

Event Timeline

• MoritzMuehlenhoff created this task.Dec 8 2020, 12:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 8 2020, 12:43 PM
gerritbot added a comment.Dec 9 2020, 10:20 AM

Change 647208 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] icinga_status: fix type downtimed != downtime

https://gerrit.wikimedia.org/r/647208

gerritbot added a project: Patch-For-Review.Dec 9 2020, 10:20 AM

Change 647208 merged by Jbond:
[operations/puppet@production] icinga_status: fix type downtimed != downtime

https://gerrit.wikimedia.org/r/647208

jbond added a comment.Dec 9 2020, 10:24 AM

This should be fixed now please re open if yu still see issues

$ /usr/local/bin/icinga-status -j auth1002.eqiad.wmnet        
{"auth1002": {"name": "auth1002", "state": "UP", "optimal": true, "failed_services": [], "downtimed": false, "notifications_enabled": true}}
jbond closed this task as Resolved.Dec 9 2020, 10:24 AM
jbond claimed this task.
Maintenance_bot removed a project: Patch-For-Review.Dec 9 2020, 11:10 AM
jbond added a comment.Dec 9 2020, 1:55 PM
In T269672#6678613, @jbond wrote:

This should be fixed now please re open if yu still see issues

Spoke to soon we now see this issue

Exception raised while executing cookbook sre.hosts.reboot-single:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 226, in run
    raw_ret = runner.run()
  File "/usr/lib/python3/dist-packages/spicerack/_module_api.py", line 19, in run
    return self._run(self.args, self.spicerack)
  File "/srv/deployment/spicerack/cookbooks/sre/hosts/reboot-single.py", line 77, in run
    icinga_ok = icinga.get_status(remote_host.hosts).optimal
  File "/usr/lib/python3/dist-packages/spicerack/icinga.py", line 291, in get_status
    hosts_status[hostname] = HostStatus(**host_status)
TypeError: __init__() got an unexpected keyword argument 'downtimed'
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99)
gerritbot added a comment.Dec 9 2020, 1:56 PM

Change 647243 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] icinga_status: remove values from json until support added to spicerack

https://gerrit.wikimedia.org/r/647243

gerritbot added a project: Patch-For-Review.Dec 9 2020, 1:56 PM

Change 647245 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/software/spicerack@master] icinga: add support for downtimed and notifications_enabled parameters

https://gerrit.wikimedia.org/r/647245

gerritbot added a comment.Dec 9 2020, 3:47 PM

Change 647243 merged by Jbond:
[operations/puppet@production] icinga_status: remove values from json until support added to spicerack

https://gerrit.wikimedia.org/r/647243

jbond added a comment.Dec 9 2020, 4:37 PM

I have pushed a patch which should removes the invalid parameters from the json output until spicerack is patched. This should hopefully fix the cookbook

gerritbot added a comment.Dec 10 2020, 9:22 AM

Change 647245 merged by jenkins-bot:
[operations/software/spicerack@master] icinga: add support for downtimed and notifications_enabled parameters

https://gerrit.wikimedia.org/r/647245

Maintenance_bot removed a project: Patch-For-Review.Dec 10 2020, 10:10 AM
jbond added a comment.Dec 10 2020, 10:47 AM

I fix has been applied to both spicerack and the icingas_status script. I have checked things work with the following and all looks good to me. please re-open if you still see issues

sudo cookbook sre.hosts.reboot-single pki1001.eqiad.wmnet
jbond reopened this task as Open.Dec 10 2020, 10:47 AM
• MoritzMuehlenhoff added a comment.Dec 10 2020, 11:07 AM

Works like a charm now.

jbond closed this task as Resolved.Apr 16 2021, 10:40 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits