Page MenuHomePhabricator

labtestservices* cron spam related to designate
Closed, ResolvedPublic

Description

Subject:	Cron <root@labtestservices2003> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
Date:	Tue, 30 Jan 2018 06:45:06 +0000
From:	Cron Daemon <root@labtestservices2003.wikimedia.org>
To:	root@labtestservices2003.wikimedia.org

/etc/cron.daily/logrotate:
error: designate-mdns:1 duplicate log entry for /var/log/designate/designate-mdns.log
error: designate-pool-manager:1 duplicate log entry for /var/log/designate/designate-pool-manager.log

This ends up being an interesting ride. Historically designate packages had issues where logrotate was not managed well across the designate packages. There are several designate log files generated and each was responsible for its own logs and for early Liberty pacakages designate-pool-manager and designate-mdns were unmanaged. Also, there was a designate-common logrotate that as ineffectual but it was removed manually in existing cases so I'm not sure what the original content was.

So we added logic to deal with this:

https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/openstack/manifests/designate/service.pp;b3ca877584ab82db9dd475b5930dfe4bbe23c84f$111-121

This is fine if every package manages their own logs, and only designate-pool-manager and designate-mdns are the only outliers. This is not the case anymore for labtestservices2001/2002/2003.

So why the difference? The Liberty designate folks saw the error of their ways and updated their packaging and we are now pulling in packages with different content logrotate wise (at least) on new servers vs old. If we were to reimage labservices1002 right now this would be the case there as well.

root@labservices1002:~# dpkg -L designate-agent | grep logrotate
/etc/logrotate.d/designate-agent
root@labservices1002:~# dpkg -L designate-api | grep logrotate
/etc/logrotate.d/designate-api
root@labservices1002:~# dpkg -L designate-sink | grep logrotate
/etc/logrotate.d/designate-sink
root@labservices1002:~# dpkg -L designate-agent | grep logrotat
/etc/logrotate.d/designate-agent

root@labservices1002:/etc/logrotate.d# ls *designate*
designate-api designate-central designate-mdns designate-pool-manager designate-sink

vs

root@labtestservices2003:~# dpkg -L designate-agent | grep logrotate
root@labtestservices2003:~# dpkg -L designate-agent | grep logrotate
root@labtestservices2003:~# dpkg -L designate-api | grep logrotate
root@labtestservices2003:~# dpkg -L designate-sink | grep logrotate
root@labtestservices2003:~# dpkg -L designate-agent | grep logrotat
root@labtestservices2003:~#

root@labtestservices2003:/etc/logrotate.d# ls *designate*
designate-common designate-mdns designate-pool-manager

Differences in package versions from original (main/prod setup vs labtest*)

root@labservices1002:~# dpkg -s designate-agent | grep Version
Version: 1:1.0.0-0ubuntu1~cloud0
root@labservices1002:~# dpkg -s designate-api | grep Version
Version: 1:1.0.0-0ubuntu1~cloud0
root@labservices1002:~# dpkg -s designate-sink | grep Version
Version: 1:1.0.0-0ubuntu1~cloud0
root@labservices1002:~# dpkg -s designate-agent | grep Version
Version: 1:1.0.0-0ubuntu1~cloud0

vs

root@labtestservices2003:~# dpkg -s designate-agent | grep Version
Version: 1:1.0.2-0ubuntu1~cloud0
root@labtestservices2003:~# dpkg -s designate-api | grep Version
Version: 1:1.0.2-0ubuntu1~cloud0
root@labtestservices2003:~# dpkg -s designate-sink | grep Version
Version: 1:1.0.2-0ubuntu1~cloud0
root@labtestservices2003:~# dpkg -s designate-agent | grep Version
Version: 1:1.0.2-0ubuntu1~cloud0

and

root@labservices1002:~# dpkg -s designate-common | grep Version
Version: 1:1.0.0-0ubuntu1~cloud0

vs

root@labtestservices2003:~# dpkg -s designate-common | grep Version
Version: 1:1.0.2-0ubuntu1~cloud0

where designate-common for labtestservices* now has /etc/logrotate.d/designate-common:

/var/log/designate/*.log {
    daily
    missingok
    compress
    delaycompress
    notifempty
    copytruncate

This should suffice and is much cleaner for all but conflicts with and in fact the comments in for the manual logrotate designation indicate they are pre-Liberty and that Liberty itself should fix. That wasn't true of first en Liberty packages however.

root@labtestservices2003:~# logrotate --force /etc/logrotate.conf

error: designate-mdns:1 duplicate log entry for /var/log/designate/designate-mdns.log
error: designate-pool-manager:1 duplicate log entry for /var/log/designate/designate-pool-manager.log
root@labtestservices2003:/etc/logrotate.d# mv designate-mdns designate-pool-manager /root/
root@labtestservices2003:/etc/logrotate.d# ls *designate*
designate-common

root@labtestservices2003:/etc/logrotate.d# logrotate --force /etc/logrotate.conf && echo $?
0

Event Timeline

We could broadly do two things:

  • Use Puppet to manage these logrotate files and ensure good state on older and newer packages
  • Upgrade the old designate packages on labservices100[12] and remove the historic fixes from Puppet

We could broadly do two things:

  • Use Puppet to manage these logrotate files and ensure good state on older and newer packages
  • Upgrade the old designate packages on labservices100[12] and remove the historic fixes from Puppet

Considering if we have to rebuild we are going to be on the newer packages, and we are already on the newer packages for labtest/labtestn/and whatever secondary environment we build in eqiad it seems upgrading is the sanest option if we don't fear it.

designate:all/trusty-updates 1:1.0.0-0ubuntu1~cloud0 upgradeable to 1:1.0.2-0ubuntu1~cloud0
designate-agent:all/trusty-updates 1:1.0.0-0ubuntu1~cloud0 upgradeable to 1:1.0.2-0ubuntu1~cloud0
designate-api:all/trusty-updates 1:1.0.0-0ubuntu1~cloud0 upgradeable to 1:1.0.2-0ubuntu1~cloud0
designate-central:all/trusty-updates 1:1.0.0-0ubuntu1~cloud0 upgradeable to 1:1.0.2-0ubuntu1~cloud0
designate-common:all/trusty-updates 1:1.0.0-0ubuntu1~cloud0 upgradeable to 1:1.0.2-0ubuntu1~cloud0
designate-doc:all/trusty-updates 1:1.0.0-0ubuntu1~cloud0 upgradeable to 1:1.0.2-0ubuntu1~cloud0
designate-sink:all/trusty-updates 1:1.0.0-0ubuntu1~cloud0 upgradeable to 1:1.0.2-0ubuntu1~cloud0
python-designate:all/trusty-updates 1:1.0.0-0ubuntu1~cloud0 upgradeable to 1:1.0.2-0ubuntu1~cloud0

apt-get install designate

@Andrew updated packages today on labservices1001 and labservices1002

Change 407459 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] designate: ensure consistent state for logrotate mess

https://gerrit.wikimedia.org/r/407459

Change 407459 merged by Rush:
[operations/puppet@production] designate: ensure consistent state for logrotate mess

https://gerrit.wikimedia.org/r/407459

chasemp claimed this task.