Page MenuHomePhabricator
Create Task
Maniphest T199911

Systemd session creation fails under I/O load
Closed, ResolvedPublic
Actions

Description

We had some occurrences of failure of the debmonitor unprivileged script that runs as APT hook on swift backend hosts, where systemd fails to start the session.

$ sudo systemctl status session-22600.scope
● session-22600.scope - Session 22600 of user debmonitor
   Loaded: loaded (/run/systemd/transient/session-22600.scope; transient; vendor preset: enabled)
Transient: yes
   Active: failed (Result: resources)

From syslog:

Jul 16 09:29:16 ms-be1028 systemd[1]: Created slice User Slice of debmonitor.
Jul 16 09:29:16 ms-be1028 systemd[1]: Starting User Manager for UID 499...
Jul 16 09:29:16 ms-be1028 systemd[1]: session-22600.scope: Failed to add PIDs to scope's control group: No such process
Jul 16 09:29:16 ms-be1028 systemd[1]: Failed to start Session 22600 of user debmonitor.
Jul 16 09:29:16 ms-be1028 systemd[1]: session-22600.scope: Unit entered failed state.
...
Jul 16 09:29:19 ms-be1028 systemd[1]: Stopped User Manager for UID 499.
Jul 16 09:29:19 ms-be1028 systemd[1]: Removed slice User Slice of debmonitor.

For a quick fix a sudo systemctl reset-failed session-22600.scope does the trick, but this task to investigate the root cause and try to find a more permanent solution.

What the script does, is dropping privileges, with su - debmonitor -c "..."

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+3 -1Disable scope cleanup cron on Thanos backends
operations/puppetproduction+5 -1dbprov: Apply the cleanup stale scope sessions to the right set of hosts
operations/puppetproduction+2 -0dbstore_multiinstance: Add module to cleanup stale scope sessions
operations/puppetproduction+2 -0swift: cleanup stale scope sessions
operations/puppetproduction+13 -0toil: cleanup failed systemd scope units
Customize query in gerrit

Related Objects
Search...

Event Timeline

Volans created this task.Jul 18 2018, 1:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 18 2018, 1:23 PM
herron triaged this task as Medium priority.Jul 18 2018, 6:26 PM
fgiunchedi added a comment.Jul 19 2018, 8:46 AM

It doesn't seem specific to debmonitor afaics, e.g. on ms-be1022 there's an occurrence where a session for root is involved instead:

syslog.1:Jul 18 11:29:14 ms-be1022 systemd[1]: Created slice User Slice of root.
syslog.1:Jul 18 11:29:14 ms-be1022 systemd[1]: Starting User Manager for UID 0...
syslog.1:Jul 18 11:29:14 ms-be1022 systemd[1]: session-269888.scope: Failed to add PIDs to scope's control group: No such process
syslog.1:Jul 18 11:29:14 ms-be1022 systemd[1]: Failed to start Session 269888 of user root.
syslog.1:Jul 18 11:29:14 ms-be1022 systemd[1]: session-269888.scope: Unit entered failed state.
auth.log:Jul 18 11:29:13 ms-be1022 sudo:   nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/lib/nagios/plugins/check_ferm
auth.log:Jul 18 11:29:13 ms-be1022 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
auth.log:Jul 18 11:29:13 ms-be1022 sudo: pam_unix(sudo:session): session closed for user root
auth.log:Jul 18 11:29:14 ms-be1022 systemd-logind[1229]: New session 269888 of user root.
auth.log:Jul 18 11:29:14 ms-be1022 systemd: pam_unix(systemd-user:session): session opened for user root by (uid=0)
auth.log:Jul 18 11:29:14 ms-be1022 systemd-logind[1229]: Removed session 269888.
auth.log:Jul 18 11:29:14 ms-be1022 sudo:   nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/lib/nagios/plugins/check_raid md
auth.log:Jul 18 11:29:14 ms-be1022 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
auth.log:Jul 18 11:29:14 ms-be1022 systemd: pam_unix(systemd-user:session): session closed for user root
auth.log:Jul 18 11:29:14 ms-be1022 sudo: pam_unix(sudo:session): session closed for user root

My guess is that in cases where the host is loaded, a session is created but by the time systemd wants to add the pid to the newly created cgroup the process has already finished.

fgiunchedi added a comment.Nov 8 2018, 8:56 AM

Current bandaid I'm using:

systemctl reset-failed $(systemctl show --failed -p Id --value *.scope)

gerritbot added a subscriber: gerritbot.Nov 22 2018, 11:08 AM

Change 475306 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] toil: cleanup failed systemd scope units

https://gerrit.wikimedia.org/r/475306

gerritbot added a project: Patch-For-Review.Nov 22 2018, 11:08 AM
gerritbot added a comment.Nov 22 2018, 5:11 PM

Change 475348 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] swift: cleanup stale scope sessions

https://gerrit.wikimedia.org/r/475348

gerritbot added a comment.Nov 22 2018, 5:17 PM

Change 475306 merged by Filippo Giunchedi:
[operations/puppet@production] toil: cleanup failed systemd scope units

https://gerrit.wikimedia.org/r/475306

gerritbot added a comment.Nov 22 2018, 5:18 PM

Change 475348 merged by Filippo Giunchedi:
[operations/puppet@production] swift: cleanup stale scope sessions

https://gerrit.wikimedia.org/r/475348

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:11 PM
Volans added a comment.Feb 14 2019, 5:34 PM

@fgiunchedi with the bandaid the problem doesn't show up anymore, do we want to keep this open for tracking or it's ok to resolve it?

fgiunchedi added a comment.Feb 14 2019, 5:36 PM
In T199911#4954809, @Volans wrote:

@fgiunchedi with the bandaid the problem doesn't show up anymore, do we want to keep this open for tracking or it's ok to resolve it?

I'm ok to resolve, we can revisit the toil module in the future if needed, Good Enough™ for now

Volans closed this task as Resolved.Feb 14 2019, 6:47 PM
Volans claimed this task.
Volans removed a project: Patch-For-Review.
Marostegui mentioned this in T265323: Add toil::systemd_scope_cleanup to dbprov hosts.Oct 13 2020, 7:36 AM
jcrespo closed subtask T265323: Add toil::systemd_scope_cleanup to dbprov hosts as Declined.Oct 21 2020, 9:17 AM
Marostegui reopened subtask T265323: Add toil::systemd_scope_cleanup to dbprov hosts as Open.Oct 27 2020, 6:08 AM
gerritbot added a comment.Oct 27 2020, 10:19 AM

Change 636633 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbstore_multiinstance: Add profile to cleanup stale scope sessions

https://gerrit.wikimedia.org/r/636633

gerritbot added a project: Patch-For-Review.Oct 27 2020, 10:19 AM
gerritbot added a comment.Oct 27 2020, 11:22 AM

Change 636633 merged by Jcrespo:
[operations/puppet@production] dbstore_multiinstance: Add module to cleanup stale scope sessions

https://gerrit.wikimedia.org/r/636633

gerritbot added a comment.Oct 27 2020, 11:41 AM

Change 636642 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbprov: Apply the cleanup stale scope sessions to the right set of hosts

https://gerrit.wikimedia.org/r/636642

gerritbot added a comment.Oct 27 2020, 11:50 AM

Change 636642 merged by Jcrespo:
[operations/puppet@production] dbprov: Apply the cleanup stale scope sessions to the right set of hosts

https://gerrit.wikimedia.org/r/636642

Maintenance_bot removed a project: Patch-For-Review.Oct 27 2020, 12:10 PM
jcrespo closed subtask T265323: Add toil::systemd_scope_cleanup to dbprov hosts as Resolved.Oct 27 2020, 1:23 PM
gerritbot added a comment.Sep 14 2021, 9:54 AM

Change 720940 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Disable scope cleanup cron on Thanos backends

https://gerrit.wikimedia.org/r/720940

Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptSep 14 2021, 9:54 AM
gerritbot added a project: Patch-For-Review.Sep 14 2021, 9:54 AM
gerritbot added a comment.Sep 16 2021, 7:13 AM

Change 720940 merged by Muehlenhoff:

[operations/puppet@production] Disable scope cleanup cron on Thanos backends

https://gerrit.wikimedia.org/r/720940

MoritzMuehlenhoff added a comment.Sep 16 2021, 7:55 AM

Since the Thanos hosts run Buster and a more recent kernel/glibc/systemd, I disabled the cleanup cron job on these hosts, so that we can check whether this got fixed. If Buster is still affected we can add the cron job back.

Maintenance_bot removed a project: Patch-For-Review.Sep 16 2021, 8:10 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL