Page MenuHomePhabricator

ceph: check time sync setup
Open, MediumPublic

Description

While doing a full reboot of the fleet saw this message in the ceph status logs:

2021-02-26 10:53:26.200166 mon.cloudcephmon2002-dev [WRN] message from mon.1 was stamped 0.080681s in the future, clocks not synchronized

Maybe we are not configuring ntp/whatever we use to sync time properly in those machines.

Event Timeline

dcaro triaged this task as Medium priority.Feb 26 2021, 10:55 AM
dcaro created this task.

This is also happening in eqiad:

2021-02-26 11:10:09.025246 mon.cloudcephmon1003 [WRN] 1 clock skew 0.0789936s > max 0.05s
2021-02-26 11:10:09.029022 mon.cloudcephmon1003 [WRN] Health check update: clock skew detected on mon.cloudcephmon1002 (MON_CLOCK_SKEW)
2021-02-26 11:10:09.029091 mon.cloudcephmon1003 [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum cloudcephmon1002,cloudcephmon1001)
2021-02-26 11:10:09.032579 mon.cloudcephmon1003 [WRN] message from mon.1 was stamped 0.088884s in the future, clocks not synchronized
2021-02-26 11:10:09.037745 mon.cloudcephmon1003 [WRN] overall HEALTH_WARN noout,norebalance flag(s) set; clock skew detected on mon.cloudcephmon1002
2021-02-26 11:10:14.420896 mon.cloudcephmon1003 [WRN] message from mon.1 was stamped 0.077498s in the future, clocks not synchronized
2021-02-26 11:10:40.954719 mon.cloudcephmon1003 [INF] Health check cleared: MON_CLOCK_SKEW (was: clock skew detected on mon.cloudcephmon1002)
dcaro renamed this task from [ceph][codfw] Check time sync setup to ceph: check time sync setup.Apr 28 2021, 8:21 AM