Page MenuHomePhabricator

systemd-logind fails with result 'timeout' in db2093 and dns4001
Open, LowPublic

Description

After upgrading the following packages on June 7th:

2018-06-07 10:38:38 status installed libssl1.0.2:amd64 1.0.2l-2+deb9u3
2018-06-07 10:38:38 status installed libc-bin:amd64 2.24-11+deb9u3
2018-06-07 10:38:38 status installed libssl1.1:amd64 1.1.0f-3+deb9u2
2018-06-07 10:38:38 status installed openssl:amd64 1.1.0f-3+deb9u2
2018-06-07 10:38:39 status installed man-db:amd64 2.7.6.1-2
2018-06-07 10:38:54 status installed linux-image-4.9.0-6-amd64:amd64 4.9.88-1+deb9u1
2018-06-07 10:38:54 status installed libc-bin:amd64 2.24-11+deb9u3
db2093
vgutierrez@db2093:~$ grep logind /var/log/daemon.log | tail
Jun 26 13:38:28 db2093 systemd[1]: systemd-logind.service: Failed with result 'timeout'.
Jun 26 13:38:28 db2093 systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Jun 26 13:39:58 db2093 systemd[1]: systemd-logind.service: Start operation timed out. Terminating.
Jun 26 13:39:58 db2093 systemd[1]: systemd-logind.service: Unit entered failed state.
Jun 26 13:39:58 db2093 systemd[1]: systemd-logind.service: Failed with result 'timeout'.
Jun 26 13:39:58 db2093 systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Jun 26 13:41:28 db2093 systemd[1]: systemd-logind.service: Start operation timed out. Terminating.
Jun 26 13:41:28 db2093 systemd[1]: systemd-logind.service: Unit entered failed state.
Jun 26 13:41:28 db2093 systemd[1]: systemd-logind.service: Failed with result 'timeout'.
Jun 26 13:41:28 db2093 systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
dns4001
vgutierrez@dns4001:~$ grep logind /var/log/daemon.log | tail
Jun 26 13:39:57 dns4001 systemd[1]: systemd-logind.service: Failed with result 'timeout'.
Jun 26 13:39:57 dns4001 systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Jun 26 13:41:27 dns4001 systemd[1]: systemd-logind.service: Start operation timed out. Terminating.
Jun 26 13:41:27 dns4001 systemd[1]: systemd-logind.service: Unit entered failed state.
Jun 26 13:41:27 dns4001 systemd[1]: systemd-logind.service: Failed with result 'timeout'.
Jun 26 13:41:27 dns4001 systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Jun 26 13:42:57 dns4001 systemd[1]: systemd-logind.service: Start operation timed out. Terminating.
Jun 26 13:42:57 dns4001 systemd[1]: systemd-logind.service: Unit entered failed state.
Jun 26 13:42:57 dns4001 systemd[1]: systemd-logind.service: Failed with result 'timeout'.
Jun 26 13:42:57 dns4001 systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.

login is painfully slow on the affected servers. I've checked our whole fleet and apparently only these two servers are affected.
Upstream bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=823987

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 26 2018, 1:47 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-27T07:37:28Z] <vgutierrez> Depool dns4001 for server restart - T198215

Apparently the server restart solved the issue for dns4001, I'll monitor it for a while to be sure.

Vgutierrez triaged this task as Low priority.Jun 27 2018, 7:53 AM