Page MenuHomePhabricator
Create Task
Maniphest T247013

cloudservices1003/1004: Warning: NTP not enabled!
Closed, ResolvedPublic
Actions

Description

I did some package updates on cloudservices1003/1003 and now icinga is complaining (wrongly) about NTP:

/home/andrew# /usr/lib/nagios/plugins/check_timedatectl
Warning: NTP not enabled!

This is because these boxes are showing yet a new output format from timedatectl

root@cloudservices1004:/home/andrew# timedatectl 
                      Local time: Thu 2020-03-05 18:09:59 UTC
                  Universal time: Thu 2020-03-05 18:09:59 UTC
                        RTC time: Thu 2020-03-05 18:09:59
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: yes
                 RTC in local TZ: no

Probably we just need to update check_timedatectl (as @MoritzMuehlenhoff did a year ago) but I'm not yet clear on why this is happening just on those two (Stretch!) servers.

Details

SubjectRepoBranchLines +/-
OpenStack/Rocky/Stretch, avoid upgrading systemdoperations/puppetproduction+7 -0
openstack: queens: apt pin systemd harderoperations/puppetproduction+1 -1
OpenStack/Queens/Stretch: fix pinning ruleoperations/puppetproduction+1 -1
OpenStack/Queens/Stretch, move priority to 501operations/puppetproduction+1 -1
OpenStack/Queens/Stretch, avoid upgrading systemdoperations/puppetproduction+7 -0
Customize query in gerrit

Event Timeline

Andrew created this task.Mar 5 2020, 6:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 5 2020, 6:11 PM
MoritzMuehlenhoff added a comment.Mar 5 2020, 7:40 PM

The output of timedatectl (which the Icinga check tests) changed between the systemd version in Stretch and Buster, the check has been adapted, but it does that based on OS versions. cloudservices* run systemd from stretch-backports, so this now appears to omit the new format. You can adapt the Icinga check to also use the new format on stretch-backports or revert the installation the the native stretch version (which is very much preferred as backports doesn't receive timely security fixes)

Krenair subscribed.Mar 5 2020, 9:17 PM
Andrew triaged this task as Medium priority.Mar 10 2020, 4:13 PM
Andrew moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
gerritbot added a comment.Mar 12 2020, 3:14 PM

Change 579289 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack/Queens/Stretch, avoid upgrading systemd

https://gerrit.wikimedia.org/r/579289

gerritbot added a project: Patch-For-Review.Mar 12 2020, 3:14 PM
Andrew added a comment.Mar 12 2020, 3:25 PM

Ugh, downgrading systemd turns out to be really messy! I'm going to try to pin it so this doesn't happen in more places.

@MoritzMuehlenhoff, what should I make of this?

root@cloudcontrol1003:~# apt-cache policy systemd
systemd:
  Installed: 232-25+deb9u12
  Candidate: 232-25+deb9u12
  Version table:
     241-5~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
 *** 232-25+deb9u12 500
        500 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status
     232-25+deb9u11 500
        500 http://security.debian.org/debian-security stretch/updates/main amd64 Packages

The version from 'debian' is newer than the version from 'debian-security' which makes me think I shouldn't pin it to -security, but if I hadn't looked I would've pinned to -security.

gerritbot added a comment.Mar 13 2020, 2:49 PM

Change 579289 merged by Andrew Bogott:
[operations/puppet@production] OpenStack/Queens/Stretch, avoid upgrading systemd

https://gerrit.wikimedia.org/r/579289

Maintenance_bot removed a project: Patch-For-Review.Mar 13 2020, 3:10 PM
gerritbot added a comment.Mar 13 2020, 4:08 PM

Change 579589 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack/Queens/Stretch, move priority to 501

https://gerrit.wikimedia.org/r/579589

gerritbot added a project: Patch-For-Review.Mar 13 2020, 4:08 PM

Change 579589 merged by Andrew Bogott:
[operations/puppet@production] OpenStack/Queens/Stretch, move priority to 501

https://gerrit.wikimedia.org/r/579589

Maintenance_bot removed a project: Patch-For-Review.Mar 13 2020, 4:10 PM
gerritbot added a comment.Mar 13 2020, 4:42 PM

Change 579597 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack/Queens/Stretch: fix pinning rule

https://gerrit.wikimedia.org/r/579597

gerritbot added a project: Patch-For-Review.Mar 13 2020, 4:42 PM

Change 579597 merged by Andrew Bogott:
[operations/puppet@production] OpenStack/Queens/Stretch: fix pinning rule

https://gerrit.wikimedia.org/r/579597

Andrew closed this task as Resolved.Mar 13 2020, 5:09 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 13 2020, 5:11 PM
gerritbot added a comment.Mar 26 2020, 10:20 AM

Change 583569 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: queens: apt pin systemd harder

https://gerrit.wikimedia.org/r/583569

gerritbot added a project: Patch-For-Review.Mar 26 2020, 10:20 AM
gerritbot added a comment.Mar 26 2020, 11:44 AM

Change 583569 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: queens: apt pin systemd harder

https://gerrit.wikimedia.org/r/583569

Maintenance_bot removed a project: Patch-For-Review.Mar 26 2020, 12:10 PM
Stashbot added a comment.Mar 26 2020, 12:32 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-26T12:32:18Z] <arturo> [codfw1dev] downgraded systemd, libsystemd0, udev and friends to the non-backports versions (T247013)

gerritbot added a comment.Apr 4 2020, 2:23 AM

Change 585848 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack/Rocky/Stretch, avoid upgrading systemd

https://gerrit.wikimedia.org/r/585848

gerritbot added a project: Patch-For-Review.Apr 4 2020, 2:23 AM

Change 585848 merged by Andrew Bogott:
[operations/puppet@production] OpenStack/Rocky/Stretch, avoid upgrading systemd

https://gerrit.wikimedia.org/r/585848

Maintenance_bot removed a project: Patch-For-Review.Apr 4 2020, 3:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL