Page MenuHomePhabricator

cronspam from labstores, labcontrol, labstestservices
Closed, ResolvedPublic

Description

Over the weekend, our cronspam folder was dominated by Labs-related alerts. There were quite a few labstore-related ones, plus a few labcontrol and labstestservices ones.

Examples:

Date: Mon, 31 Oct 2016 00:00:04 +0000
From: Cron Daemon <root@labstore2004.codfw.wmnet>
To: root@labstore2004.codfw.wmnet
Subject: Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc test testsnap /dev/backup/test

2016-10-31 00:00:02,803 INFO force is enabled
2016-10-31 00:00:02,857 INFO removing testsnap
Traceback (most recent call last):
  File "/usr/local/sbin/snapshot-manager", line 269, in <module>
    main()
  File "/usr/local/sbin/snapshot-manager", line 265, in main
    actions.get(args.action, help)()
  File "/usr/local/sbin/snapshot-manager", line 238, in create
    ss.remove()
  File "/usr/local/sbin/snapshot-manager", line 130, in remove
    if not self.status() or not self.is_snapshot():
  File "/usr/local/sbin/snapshot-manager", line 100, in is_snapshot
    return bool(meta['lv_attr'].startswith('s'))
KeyError: 'lv_attr'
---NeoMutt: =Automated/root [Msgs:120/582 Inc:4 1,4M]---(threads/date)------------------------------------------------------------------------------------(7%)---
Date: Mon, 31 Oct 2016 00:00:04 +0000
From: Cron Daemon <root@labstore2003.codfw.wmnet>
To: root@labstore2003.codfw.wmnet
Subject: Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 misc test testsnap /dev/backup/test

2016-10-31 00:00:04,633 INFO force is enabled
opendev [/dev/backup/test]: No such file or directory
do_patch: EOF(stdin)
Date: Mon, 31 Oct 2016 00:01:17 +0000
From: Cron Daemon <root@labstore2003.codfw.wmnet>
To: root@labstore2003.codfw.wmnet
Subject: Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 misc test testsnap /dev/backup/test

2016-10-31 00:01:07,689 INFO force is enabled
2016-10-31 00:01:07,712 INFO removing testsnap
2016-10-31 00:01:07,774 INFO removing testsnap
2016-10-31 00:01:17,442 ERROR b'  Logical volume misc/testsnap in use.\n'
opendev [/dev/backup/test]: No such file or directory
do_patch: EOF(stdin)
Date: Mon, 31 Oct 2016 14:40:02 +0000
From: Cron Daemon <root@labtestservices2001.wikimedia.org>
To: root@labtestservices2001.wikimedia.org
Subject: Cron <root@labtestservices2001> /etc/dns-floating-ip-updater.py > /dev/null

Traceback (most recent call last):
  File "/etc/dns-floating-ip-updater.py", line 63, in <module>
    for server in nova_client.servers.list():
  File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 608, in list
    "servers")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 72, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 170, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 91, in request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 206, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 95, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 304, in request
    auth_headers = self.get_auth_headers(auth)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 617, in get_auth_headers
    return auth.get_headers(self, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/base.py", line 142, in get_headers
    token = self.get_token(session)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
    return self.get_access(session).auth_token
  File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
    authenticated=False, log=False)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 501, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 401, in request
    raise exceptions.from_response(resp, method, url)
keystoneclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-73f15447-0a0c-4d1b-8460-279cb4156dd2)
Date: Mon, 31 Oct 2016 13:15:04 +0000
From: Cron Daemon <root@labcontrol1001.wikimedia.org>
To: glancesync@labcontrol1001.wikimedia.org
Subject: Cron <glancesync@labcontrol1001> /usr/bin/rsync --delete --delete-after -aSO /srv/glance//images/ labcontrol1002.wikimedia.org:/srv/glance//images/

ssh: connect to host labcontrol1002.wikimedia.org port 22: No route to host
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.0]

Event Timeline

The "Cron <glancesync@labcontrol1001> /usr/bin/rsync --delete --delete-after -aSO /srv/glanceimages/ labcontrol1002.wikimedia.org:/srv/glanceimages/" messages are correct/not spam; labcontrol1002 was down over the weekend and the system only correctly rebooted

Change 330453 had a related patch set uploaded (by Faidon Liambotis):
labs: 2>/dev/null dns-floating-ip-updater's output

https://gerrit.wikimedia.org/r/330453

Change 330453 merged by Faidon Liambotis:
labs: 2>/dev/null dns-floating-ip-updater's output

https://gerrit.wikimedia.org/r/330453

faidon claimed this task.

I don't think there's anything else to be done for this and it seems to have been largely ignored anyway. Resolving.