While doing a full reboot of the fleet saw this message in the ceph status logs:
2021-02-26 10:53:26.200166 mon.cloudcephmon2002-dev [WRN] message from mon.1 was stamped 0.080681s in the future, clocks not synchronized
Maybe we are not configuring ntp/whatever we use to sync time properly in those machines.
This is also happening in eqiad:
2021-02-26 11:10:09.025246 mon.cloudcephmon1003 [WRN] 1 clock skew 0.0789936s > max 0.05s 2021-02-26 11:10:09.029022 mon.cloudcephmon1003 [WRN] Health check update: clock skew detected on mon.cloudcephmon1002 (MON_CLOCK_SKEW) 2021-02-26 11:10:09.029091 mon.cloudcephmon1003 [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum cloudcephmon1002,cloudcephmon1001) 2021-02-26 11:10:09.032579 mon.cloudcephmon1003 [WRN] message from mon.1 was stamped 0.088884s in the future, clocks not synchronized 2021-02-26 11:10:09.037745 mon.cloudcephmon1003 [WRN] overall HEALTH_WARN noout,norebalance flag(s) set; clock skew detected on mon.cloudcephmon1002 2021-02-26 11:10:14.420896 mon.cloudcephmon1003 [WRN] message from mon.1 was stamped 0.077498s in the future, clocks not synchronized 2021-02-26 11:10:40.954719 mon.cloudcephmon1003 [INF] Health check cleared: MON_CLOCK_SKEW (was: clock skew detected on mon.cloudcephmon1002)
Mentioned in SAL (#wikimedia-cloud) [2021-04-28T08:21:22Z] <dcaro> The clock skew seems intermittent, there's another task to follw it T275860 (T280641)