Page MenuHomePhabricator
Create Task
Maniphest T363957

deployment_server bullseye - mw-cgroup.service: Failed
Closed, ResolvedPublicBUG REPORT
Actions

Description

Steps to replicate the issue (include links if applicable):

  • Create a bullseye VM in cloud VPS and apply the puppet role deployment_server (and the needed Hiera data)

What happens?:

When the puppet agent runs, among other issues, systemd fails to start the service mw-cgroup due to "/sys/fs/cgroup/memory/release_agent: Permission denied"

May 01 22:23:36 deploy-1006 systemd[1]: Starting "Mw-cgroup"...
May 01 22:23:36 deploy-1006 mw-cgroup[735545]: /bin/bash: line 1: /sys/fs/cgroup/memory/release_agent: Permission denied
May 01 22:23:36 deploy-1006 systemd[1]: mw-cgroup.service: Main process exited, code=exited, status=1/FAILURE
May 01 22:23:36 deploy-1006 systemd[1]: mw-cgroup.service: Failed with result 'exit-code'.
May 01 22:23:36 deploy-1006 systemd[1]: Failed to start "Mw-cgroup".

What should have happened instead?:

The service mw-cgroup should start normally (or puppet should not try to start it).

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):

Debian bullseye

Other information (browser name/version, screenshots, etc.):

https://gerrit.wikimedia.org/r/c/operations/puppet/+/991347/5/modules/mediawiki/manifests/cgroup.pp seems related but the grub::bootparam doesn't apply in cloud? is it only an issue in cloud VPS and not production?

Related Objects
Search...

Event Timeline

Dzahn created this task.May 1 2024, 10:36 PM
Dzahn added a comment.May 1 2024, 10:38 PM

I now found T325228#9445729 which seems like the exact same issue on snapshot hosts.

Dzahn mentioned this in T325228: Migrate Dumps Snapshot hosts from Buster to Bullseye.May 1 2024, 10:41 PM
Stashbot added a comment.May 1 2024, 10:45 PM

Mentioned in SAL (#wikimedia-cloud) [2024-05-01T22:45:10Z] <mutante> rebooting deploy-1006 to see if issue with starting mw-cgroup goes away - .. and it did! - it must have been the grub config from https://gerrit.wikimedia.org/r/c/operations/puppet/+/991347 but needed one reboot - T363957

Dzahn added a comment.May 1 2024, 10:49 PM

I rebooted the VM and the issue went away! The grub config from the change above was applied apparently:

root@deploy-1006:/# grep -Eo systemd.unified_cgroup_hierarchy=0 /boot/grub/grub.cfg 
systemd.unified_cgroup_hierarchy=0
Notice: /Stage[main]/Mediawiki::Cgroup/Base::Service_unit[mw-cgroup]/Service[mw-cgroup]/ensure: ensure changed 'stopped' to 'running'

You just have to know you need that one extra reboot after applying certain roles.

Dzahn closed this task as Resolved.May 1 2024, 10:51 PM
Dzahn claimed this task.
Dzahn mentioned this in T363415: add bullseye support to deployment server puppet role - upgrade deployment server in devtools.
Dzahn mentioned this in T364656: replace production buster deployment servers.May 10 2024, 9:05 PM
thcipriani subscribed.Jul 12 2024, 9:00 PM

Thanks for documenting this, ran into the same thing in deployment prep (T327742), reboot also fixed it there.

Dzahn awarded a token.Jul 12 2024, 10:46 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits