Over the weekend, our cronspam folder was dominated by Labs-related alerts. There were quite a few labstore-related ones, plus a few labcontrol and labstestservices ones.
Examples:
Date: Mon, 31 Oct 2016 00:00:04 +0000
From: Cron Daemon <root@labstore2004.codfw.wmnet>
To: root@labstore2004.codfw.wmnet
Subject: Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc test testsnap /dev/backup/test
2016-10-31 00:00:02,803 INFO force is enabled
2016-10-31 00:00:02,857 INFO removing testsnap
Traceback (most recent call last):
File "/usr/local/sbin/snapshot-manager", line 269, in <module>
main()
File "/usr/local/sbin/snapshot-manager", line 265, in main
actions.get(args.action, help)()
File "/usr/local/sbin/snapshot-manager", line 238, in create
ss.remove()
File "/usr/local/sbin/snapshot-manager", line 130, in remove
if not self.status() or not self.is_snapshot():
File "/usr/local/sbin/snapshot-manager", line 100, in is_snapshot
return bool(meta['lv_attr'].startswith('s'))
KeyError: 'lv_attr'---NeoMutt: =Automated/root [Msgs:120/582 Inc:4 1,4M]---(threads/date)------------------------------------------------------------------------------------(7%)--- Date: Mon, 31 Oct 2016 00:00:04 +0000 From: Cron Daemon <root@labstore2003.codfw.wmnet> To: root@labstore2003.codfw.wmnet Subject: Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 misc test testsnap /dev/backup/test 2016-10-31 00:00:04,633 INFO force is enabled opendev [/dev/backup/test]: No such file or directory do_patch: EOF(stdin)
Date: Mon, 31 Oct 2016 00:01:17 +0000 From: Cron Daemon <root@labstore2003.codfw.wmnet> To: root@labstore2003.codfw.wmnet Subject: Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 misc test testsnap /dev/backup/test 2016-10-31 00:01:07,689 INFO force is enabled 2016-10-31 00:01:07,712 INFO removing testsnap 2016-10-31 00:01:07,774 INFO removing testsnap 2016-10-31 00:01:17,442 ERROR b' Logical volume misc/testsnap in use.\n' opendev [/dev/backup/test]: No such file or directory do_patch: EOF(stdin)
Date: Mon, 31 Oct 2016 14:40:02 +0000
From: Cron Daemon <root@labtestservices2001.wikimedia.org>
To: root@labtestservices2001.wikimedia.org
Subject: Cron <root@labtestservices2001> /etc/dns-floating-ip-updater.py > /dev/null
Traceback (most recent call last):
File "/etc/dns-floating-ip-updater.py", line 63, in <module>
for server in nova_client.servers.list():
File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 608, in list
"servers")
File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 72, in _list
_resp, body = self.api.client.get(url)
File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 170, in get
return self.request(url, 'GET', **kwargs)
File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 91, in request
**kwargs)
File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 206, in request
resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 95, in request
return self.session.request(url, method, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 304, in request
auth_headers = self.get_auth_headers(auth)
File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 617, in get_auth_headers
return auth.get_headers(self, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/base.py", line 142, in get_headers
token = self.get_token(session)
File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
return self.get_access(session).auth_token
File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
self.auth_ref = self.get_auth_ref(session)
File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
authenticated=False, log=False)
File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 501, in post
return self.request(url, 'POST', **kwargs)
File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 401, in request
raise exceptions.from_response(resp, method, url)
keystoneclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-73f15447-0a0c-4d1b-8460-279cb4156dd2)Date: Mon, 31 Oct 2016 13:15:04 +0000 From: Cron Daemon <root@labcontrol1001.wikimedia.org> To: glancesync@labcontrol1001.wikimedia.org Subject: Cron <glancesync@labcontrol1001> /usr/bin/rsync --delete --delete-after -aSO /srv/glance//images/ labcontrol1002.wikimedia.org:/srv/glance//images/ ssh: connect to host labcontrol1002.wikimedia.org port 22: No route to host rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.0]