Page MenuHomePhabricator
Create Task
Maniphest T273563

Helm install fails in CI namespace: apparmor failed to apply profile
Closed, ResolvedPublic
Actions

Description

Attempting to install the blubberoid helm chart in the ci namespace is failing.

kubectl describe pod reveals this error message:

Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "blubberoid-4ipjhmel-66d65bd7cd-pbr9x": Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"apparmor failed to apply profile: no such file or directory\\\"\"\n"

Details

SubjectRepoBranchLines +/-
k8s::kubelet: Ensure apparmor is purged on old k8s nodesoperations/puppetproduction+7 -2
Customize query in gerrit

Related Objects

Event Timeline

jeena created this task.Feb 2 2021, 12:16 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 2 2021, 12:16 AM
jeena added a subscriber: JMeybohm.Feb 2 2021, 12:24 AM
jeena renamed this task from Helm install fails in CI namespace to Helm install fails in CI namespace: apparmor failed to apply profile.Feb 2 2021, 12:27 AM
JMeybohm claimed this task.Feb 2 2021, 9:02 AM
JMeybohm edited projects, added serviceops, Prod-Kubernetes; removed SRE.

Most likely my fault as I installed the apparmor package as part of T228967.
Will take a look!

Joe triaged this task as High priority.Feb 2 2021, 9:10 AM
gerritbot added a comment.Feb 2 2021, 11:27 AM

Change 661083 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] k8s::kubelet: Ensure apparmor is purged on old k8s nodes

https://gerrit.wikimedia.org/r/661083

gerritbot added a project: Patch-For-Review.Feb 2 2021, 11:27 AM
JMeybohm added a comment.Feb 2 2021, 11:39 AM

Fortunately, this issue had not effected prod clusters. So it must be something in the combination of kernel / k8s version / apparmor that goes wrong here.
As we're moving away from that combination I think it's not worth it to spend more time investigating. We'll just purge the apparmor package from old k8s nodes again.

gerritbot added a comment.Feb 2 2021, 11:47 AM

Change 661083 merged by JMeybohm:
[operations/puppet@production] k8s::kubelet: Ensure apparmor is purged on old k8s nodes

https://gerrit.wikimedia.org/r/661083

JMeybohm closed this task as Resolved.Feb 2 2021, 11:55 AM

Did a puppet run on the affected nodes and scaled blubberoid pods on each of them. Looks good to me.

Maintenance_bot removed a project: Patch-For-Review.Feb 2 2021, 12:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL