Page MenuHomePhabricator
Create Task
Maniphest T341051

Fix Puppet agent provisioning on Jenkins agent instances
Closed, DeclinedPublic
Actions

Description

When creating the new integration-agent-docker-1041 with role::ci::slave::labs::docker Puppet complains cause docker is started before /var/lib/docker is partitioned.

That comes from profile::ci::docker definining Service[docker] but lacking a require on the mount.

Notice: /Stage[main]/Labs_lvm/Exec[create-volume-group]/returns: executed successfully
...
Notice: /Stage[main]/Docker::Configuration/File[/etc/docker]/ensure: created
Notice: /Stage[main]/Docker::Configuration/File[/etc/docker/daemon.json]/ensure: defined content as '{md5}fefc6b0dc2af089592fa3b7bd0b4ad29'
Error: Could not enable docker: 
Error: /Stage[main]/Profile::Ci::Docker/Service[docker]/enable: change from 'false' to 'true' failed: Could not enable docker: 
...

Info: Class[Profile::Ci::Docker]: Unscheduling all events on Class[Profile::Ci::Docker]
Notice: /Stage[main]/Profile::Ci::Dockervolume/Labs_lvm::Volume[docker]/Exec[available-space-docker]/returns: executed successfully
Notice: /Stage[main]/Profile::Ci::Dockervolume/Labs_lvm::Volume[docker]/Exec[create-vd-docker]/returns: executed successfully
Notice: /Stage[main]/Profile::Ci::Dockervolume/Labs_lvm::Volume[docker]/Mount[/var/lib/docker]/ensure: defined 'ensure' as 'mounted'
Info: Computing checksum on file /etc/fstab
Info: /Stage[main]/Profile::Ci::Dockervolume/Labs_lvm::Volume[docker]/Mount[/var/lib/docker]: Scheduling refresh of Mount[/var/lib/docker]
Info: Mount[/var/lib/docker](provider=parsed): Remounting
Notice: /Stage[main]/Profile::Ci::Dockervolume/Labs_lvm::Volume[docker]/Mount[/var/lib/docker]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Profile::Ci::Dockervolume/Labs_lvm::Volume[docker]/Mount[/var/lib/docker]: Scheduling refresh of Mount[/var/lib/docker]
Notice: /Stage[main]/Profile::Ci::Dockervolume/Labs_lvm::Volume[docker]/File[/var/lib/docker]/mode: mode changed '0755' to '0711'

Details

SubjectRepoBranchLines +/-
ci: setup dockervolume before Docker daemonoperations/puppetproduction+1 -1
ci: enabling docker requires the docker-ce packageoperations/puppetproduction+3 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Jul 4 2023, 10:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 4 2023, 10:47 AM
hashar added a parent task: T340070: Rebuild WMCS integration instances to larger flavor.Jul 4 2023, 10:47 AM
gerritbot added a comment.Jul 4 2023, 11:41 AM

Change 935405 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] ci: setup dockervolume before Docker daemon

https://gerrit.wikimedia.org/r/935405

gerritbot added a project: Patch-For-Review.Jul 4 2023, 11:41 AM
hashar added a comment.EditedJul 4 2023, 3:04 PM

With https://gerrit.wikimedia.org/r/c/operations/puppet/+/935405 applied:

Notice: /Stage[main]/Labs_lvm/Exec[create-volume-group]/returns: executed successfully
Notice: /Stage[main]/Profile::Ci::Docker/Package[acl]/ensure: created
Notice: /Stage[main]/Docker::Configuration/File[/etc/docker]/ensure: created
Notice: /Stage[main]/Docker::Configuration/File[/etc/docker/daemon.json]/ensure: defined content as '{md5}fefc6b0dc2af089592fa3b7bd0b4ad29'
Error: Could not enable docker: 
Error: /Stage[main]/Profile::Ci::Docker/Service[docker]/enable: change from 'false' to 'true' failed: Could not enable docker: 
Notice: /Stage[main]/Profile::Ci::Docker/File[/usr/local/bin/docker-credential-environment]/ensure: defined content as '{md5}a48067b5809a2703033c7bf7b89c98a8'
Notice: /Stage[main]/Java/Java::Package[openjdk-jre-headless-11]/Package[openjdk-11-jre-headless]/ensure: created
Notice: /Stage[main]/Profile::Ci::Dockervolume/Labs_lvm::Volume[docker]/Exec[available-space-docker]/returns: executed successfully
Notice: /Stage[main]/Profile::Ci::Dockervolume/Labs_lvm::Volume[docker]/Exec[create-vd-docker]/returns: executed successfully
Notice: /Stage[main]/Profile::Ci::Dockervolume/Labs_lvm::Volume[docker]/Exec[create-mountpoint-docker]/returns: executed successfully
Notice: /Stage[main]/Profile::Ci::Dockervolume/Labs_lvm::Volume[docker]/Mount[/var/lib/docker]/ensure: defined 'ensure' as 'mounted'
...
Notice: /Stage[main]/Profile::Ci::Docker/Package[docker-ce]/ensure: created

Not quite there unfortunately :)

The issue would be service { enable => true } is invoked BEFORE the docker-ce package is installed.

gerritbot added a comment.Jul 4 2023, 3:55 PM

Change 935471 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] ci: enabling docker require the docker-ce package

https://gerrit.wikimedia.org/r/935471

hashar claimed this task.Jul 4 2023, 3:56 PM
gerritbot added a comment.Sep 11 2023, 12:18 PM

Change 935471 abandoned by Hashar:

[operations/puppet@production] ci: enabling docker requires the docker-ce package

Reason:

https://gerrit.wikimedia.org/r/935471

gerritbot added a comment.Sep 11 2023, 12:18 PM

Change 935405 abandoned by Hashar:

[operations/puppet@production] ci: setup dockervolume before Docker daemon

Reason:

https://gerrit.wikimedia.org/r/935405

Maintenance_bot removed a project: Patch-For-Review.Sep 11 2023, 12:30 PM
dancy added a subscriber: dancy.Sep 13 2023, 5:20 PM

@hashar Note that you still have a local cherry-pick related to this ticket in integration-puppetmaster-02.integration.eqiad.wmflabs:/var/lib/git/operations/puppet.

hashar closed this task as Declined.Sep 18 2023, 8:41 AM

I have removed the cherry pick from the integration Puppet master. I am bailing out on ensuring an initial provisioning works on the first time given we usually rebuild those instances every couple years :)

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL