Over the weekend, our cronspam folder was dominated by Labs-related alerts. There were quite a few labstore-related ones, plus a few labcontrol and labstestservices ones.
Examples:
Date: Mon, 31 Oct 2016 00:00:04 +0000 From: Cron Daemon <root@labstore2004.codfw.wmnet> To: root@labstore2004.codfw.wmnet Subject: Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc test testsnap /dev/backup/test 2016-10-31 00:00:02,803 INFO force is enabled 2016-10-31 00:00:02,857 INFO removing testsnap Traceback (most recent call last): File "/usr/local/sbin/snapshot-manager", line 269, in <module> main() File "/usr/local/sbin/snapshot-manager", line 265, in main actions.get(args.action, help)() File "/usr/local/sbin/snapshot-manager", line 238, in create ss.remove() File "/usr/local/sbin/snapshot-manager", line 130, in remove if not self.status() or not self.is_snapshot(): File "/usr/local/sbin/snapshot-manager", line 100, in is_snapshot return bool(meta['lv_attr'].startswith('s')) KeyError: 'lv_attr'
---NeoMutt: =Automated/root [Msgs:120/582 Inc:4 1,4M]---(threads/date)------------------------------------------------------------------------------------(7%)--- Date: Mon, 31 Oct 2016 00:00:04 +0000 From: Cron Daemon <root@labstore2003.codfw.wmnet> To: root@labstore2003.codfw.wmnet Subject: Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 misc test testsnap /dev/backup/test 2016-10-31 00:00:04,633 INFO force is enabled opendev [/dev/backup/test]: No such file or directory do_patch: EOF(stdin)
Date: Mon, 31 Oct 2016 00:01:17 +0000 From: Cron Daemon <root@labstore2003.codfw.wmnet> To: root@labstore2003.codfw.wmnet Subject: Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 misc test testsnap /dev/backup/test 2016-10-31 00:01:07,689 INFO force is enabled 2016-10-31 00:01:07,712 INFO removing testsnap 2016-10-31 00:01:07,774 INFO removing testsnap 2016-10-31 00:01:17,442 ERROR b' Logical volume misc/testsnap in use.\n' opendev [/dev/backup/test]: No such file or directory do_patch: EOF(stdin)
Date: Mon, 31 Oct 2016 14:40:02 +0000 From: Cron Daemon <root@labtestservices2001.wikimedia.org> To: root@labtestservices2001.wikimedia.org Subject: Cron <root@labtestservices2001> /etc/dns-floating-ip-updater.py > /dev/null Traceback (most recent call last): File "/etc/dns-floating-ip-updater.py", line 63, in <module> for server in nova_client.servers.list(): File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 608, in list "servers") File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 72, in _list _resp, body = self.api.client.get(url) File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 170, in get return self.request(url, 'GET', **kwargs) File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 91, in request **kwargs) File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 206, in request resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 95, in request return self.session.request(url, method, **kwargs) File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner return func(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 304, in request auth_headers = self.get_auth_headers(auth) File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 617, in get_auth_headers return auth.get_headers(self, **kwargs) File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/base.py", line 142, in get_headers token = self.get_token(session) File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 200, in get_token return self.get_access(session).auth_token File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 240, in get_access self.auth_ref = self.get_auth_ref(session) File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref authenticated=False, log=False) File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 501, in post return self.request(url, 'POST', **kwargs) File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner return func(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 401, in request raise exceptions.from_response(resp, method, url) keystoneclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-73f15447-0a0c-4d1b-8460-279cb4156dd2)
Date: Mon, 31 Oct 2016 13:15:04 +0000 From: Cron Daemon <root@labcontrol1001.wikimedia.org> To: glancesync@labcontrol1001.wikimedia.org Subject: Cron <glancesync@labcontrol1001> /usr/bin/rsync --delete --delete-after -aSO /srv/glance//images/ labcontrol1002.wikimedia.org:/srv/glance//images/ ssh: connect to host labcontrol1002.wikimedia.org port 22: No route to host rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.0]