cloudservices1003/1004: Warning: NTP not enabled!
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Andrew
	Mar 5 2020, 6:11 PM

Description

I did some package updates on cloudservices1003/1003 and now icinga is complaining (wrongly) about NTP:

/home/andrew# /usr/lib/nagios/plugins/check_timedatectl
Warning: NTP not enabled!

This is because these boxes are showing yet a new output format from timedatectl

root@cloudservices1004:/home/andrew# timedatectl 
                      Local time: Thu 2020-03-05 18:09:59 UTC
                  Universal time: Thu 2020-03-05 18:09:59 UTC
                        RTC time: Thu 2020-03-05 18:09:59
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: yes
                 RTC in local TZ: no

Probably we just need to update check_timedatectl (as @MoritzMuehlenhoff did a year ago) but I'm not yet clear on why this is happening just on those two (Stretch!) servers.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
openstack serverpackages: don't pin systemd	operations/puppet	production	+0 -7
OpenStack/Rocky/Stretch, avoid upgrading systemd	operations/puppet	production	+7 -0
openstack: queens: apt pin systemd harder	operations/puppet	production	+1 -1
OpenStack/Queens/Stretch: fix pinning rule	operations/puppet	production	+1 -1
OpenStack/Queens/Stretch, move priority to 501	operations/puppet	production	+1 -1
OpenStack/Queens/Stretch, avoid upgrading systemd	operations/puppet	production	+7 -0

Customize query in gerrit

Event Timeline

Andrew created this task.Mar 5 2020, 6:11 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 5 2020, 6:11 PM

The output of timedatectl (which the Icinga check tests) changed between the systemd version in Stretch and Buster, the check has been adapted, but it does that based on OS versions. cloudservices* run systemd from stretch-backports, so this now appears to omit the new format. You can adapt the Icinga check to also use the new format on stretch-backports or revert the installation the the native stretch version (which is very much preferred as backports doesn't receive timely security fixes)

Krenair subscribed.Mar 5 2020, 9:17 PM

Andrew triaged this task as Medium priority.Mar 10 2020, 4:13 PM

Andrew moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Change 579289 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack/Queens/Stretch, avoid upgrading systemd

https://gerrit.wikimedia.org/r/579289

gerritbot added a project: Patch-For-Review.Mar 12 2020, 3:14 PM

Ugh, downgrading systemd turns out to be really messy! I'm going to try to pin it so this doesn't happen in more places.

@MoritzMuehlenhoff, what should I make of this?

root@cloudcontrol1003:~# apt-cache policy systemd
systemd:
  Installed: 232-25+deb9u12
  Candidate: 232-25+deb9u12
  Version table:
     241-5~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
 *** 232-25+deb9u12 500
        500 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status
     232-25+deb9u11 500
        500 http://security.debian.org/debian-security stretch/updates/main amd64 Packages

The version from 'debian' is newer than the version from 'debian-security' which makes me think I shouldn't pin it to -security, but if I hadn't looked I would've pinned to -security.

Change 579289 merged by Andrew Bogott:
[operations/puppet@production] OpenStack/Queens/Stretch, avoid upgrading systemd

https://gerrit.wikimedia.org/r/579289

Maintenance_bot removed a project: Patch-For-Review.Mar 13 2020, 3:10 PM

Change 579589 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack/Queens/Stretch, move priority to 501

https://gerrit.wikimedia.org/r/579589

Change 579589 merged by Andrew Bogott:
[operations/puppet@production] OpenStack/Queens/Stretch, move priority to 501

https://gerrit.wikimedia.org/r/579589

Maintenance_bot removed a project: Patch-For-Review.Mar 13 2020, 4:10 PM

Change 579597 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack/Queens/Stretch: fix pinning rule

https://gerrit.wikimedia.org/r/579597

Change 579597 merged by Andrew Bogott:
[operations/puppet@production] OpenStack/Queens/Stretch: fix pinning rule

https://gerrit.wikimedia.org/r/579597

Andrew closed this task as Resolved.Mar 13 2020, 5:09 PM

Maintenance_bot removed a project: Patch-For-Review.Mar 13 2020, 5:11 PM

Change 583569 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: queens: apt pin systemd harder

https://gerrit.wikimedia.org/r/583569

gerritbot added a project: Patch-For-Review.Mar 26 2020, 10:20 AM

Change 583569 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: queens: apt pin systemd harder

https://gerrit.wikimedia.org/r/583569

Maintenance_bot removed a project: Patch-For-Review.Mar 26 2020, 12:10 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-26T12:32:18Z] <arturo> [codfw1dev] downgraded systemd, libsystemd0, udev and friends to the non-backports versions (T247013)

Change 585848 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack/Rocky/Stretch, avoid upgrading systemd

https://gerrit.wikimedia.org/r/585848

Change 585848 merged by Andrew Bogott:
[operations/puppet@production] OpenStack/Rocky/Stretch, avoid upgrading systemd

https://gerrit.wikimedia.org/r/585848

Maintenance_bot removed a project: Patch-For-Review.Apr 4 2020, 3:10 AM

Change #1180913 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] openstack serverpackages: don't pin systemd

https://gerrit.wikimedia.org/r/1180913

Restricted Application edited projects, added cloud-services-team; removed cloud-services-team (Kanban). · View Herald TranscriptAug 21 2025, 5:23 PM

gerritbot added a project: Patch-For-Review.Aug 21 2025, 5:23 PM

Change #1180913 merged by Andrew Bogott: