Page MenuHomePhabricator
Create Task
Maniphest T226233

Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch
Closed, ResolvedPublic
Actions

Description

I would like to rebuild the whole fleet of integration-slave-docker instances for a few reasons:

  1. upgrade from Jessie to Stretch
  2. reduce RAM from 32 to 24 by switching the flavor from bigram to mediumram (available since T225025)
  3. in the hostname, replace slave by agent
  4. use /srv/jenkins/workspace (instead of jenkins-workspace)

Steps:

Wait 3 or 4 minutes for the instance to be fully provisioned. Then on the instance:

  • rm -fR /var/lib/puppet/ssl && puppet agent -tv
  • if that complains:
    • get the instance fully qualified domain name hostname --fqdn
    • on integration-puppetmaster01.integration.eqiad.wmflabs: sudo puppet cert clean <FQDN OF INSTANCE HERE>

Apply the puppet role

  • Run puppet on the instance
  • Make sure there is a /var/lib/docker partition for Docker
  • Upgrade and reboot: sudo apt -y dist-upgrade && /usr/sbin/reboot

Add the instance to Jenkins

  • Create node copying integration-agent-docker-1001
  • Remote root directory is /srv/jenkins/workspace
  • Change the IP address
  • Once the master tries to connect, manually accept the ssh key from the side bar: Trust SSH Host Key

Details

SubjectRepoBranchLines +/-
contint: pin Docker on stretchoperations/puppetproduction+5 -1
contint: remove zuul-cloner from Docker agentoperations/puppetproduction+0 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Jun 21 2019, 7:56 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 21 2019, 7:56 AM
hashar updated the task description. (Show Details)Jun 21 2019, 8:14 AM
hashar added a comment.Jun 21 2019, 8:29 AM
E: Version '18.06.2~ce~3-0~debian' for 'docker-ce' was not found

On Jessie we have:

$ apt-cache policy docker-ce
docker-ce:
  Installed: 18.06.2~ce~3-0~debian
  Candidate: 18.06.2~ce~3-0~debian
  Version table:
 *** 18.06.2~ce~3-0~debian 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/thirdparty/ci amd64 Packages
        100 /var/lib/dpkg/status
gerritbot added a comment.Jun 21 2019, 8:46 AM

Change 518222 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] contint: remove zuul-cloner from Docker agent

https://gerrit.wikimedia.org/r/518222

gerritbot added a project: Patch-For-Review.Jun 21 2019, 8:46 AM
hashar added a comment.Jun 21 2019, 8:49 AM

I have also removed the zuul package from the Docker instances.

hashar@integration-cumin:~$ sudo cumin --force 'name:docker' 'apt-get -y remove --purge zuul'
hashar@integration-cumin:~$ sudo cumin --force 'name:docker' 'rm -fR /var/lib/zuul /usr/share/python/zuul'

Jdforrester-WMF awarded a token.Jun 21 2019, 5:05 PM
MoritzMuehlenhoff closed subtask T226236: Upload docker-ce 18.06.3 upstream package for Stretch as Resolved.Jun 28 2019, 6:09 AM
hashar reopened subtask T226236: Upload docker-ce 18.06.3 upstream package for Stretch as Open.Jun 28 2019, 7:57 AM
hashar added a parent task: T224943: Migrate debian-glue jobs to Buster instances.Jul 1 2019, 4:21 PM
hashar mentioned this in T224943: Migrate debian-glue jobs to Buster instances.
greg added a project: Release-Engineering-Team-TODO.Jul 1 2019, 9:28 PM
greg moved this task from Should be empty (use Release-Engineering-Team) to Soon-ish on the Release-Engineering-Team-TODO board.Jul 1 2019, 9:30 PM
greg removed a project: Release-Engineering-Team (Kanban).Jul 1 2019, 9:31 PM
greg triaged this task as Medium priority.Jul 6 2019, 5:01 AM
greg moved this task from Soon-ish to Next on the Release-Engineering-Team-TODO board.
greg edited projects, added Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (201907); removed Release-Engineering-Team-TODO.Jul 6 2019, 5:16 AM
greg moved this task from INBOX to Ready on the Release-Engineering-Team-TODO (201907) board.
Jdforrester-WMF subscribed.Jul 8 2019, 11:26 PM

Given that as of two days ago Buster is now officially stable and so per https://wikitech.wikimedia.org/wiki/Operating_system_upgrade_policy is allowed to be used in production, do we want to upgrade straight to that?

hashar added a comment.Jul 9 2019, 4:17 PM

I would rather not be one of the first adopters of Buster :-]

gerritbot added a comment.Jul 15 2019, 10:20 AM

Change 518222 merged by Alexandros Kosiaris:
[operations/puppet@production] contint: remove zuul-cloner from Docker agent

https://gerrit.wikimedia.org/r/518222

Maintenance_bot removed a project: Patch-For-Review.Jul 15 2019, 11:11 AM
MoritzMuehlenhoff closed subtask T226236: Upload docker-ce 18.06.3 upstream package for Stretch as Resolved.Jul 16 2019, 11:58 AM
hashar claimed this task.Jul 17 2019, 3:39 PM
Legoktm subscribed.Jul 17 2019, 4:45 PM

Yay stretch! Please make sure to test that the less RAM will still be enough for phan to run relatively quickly.

hashar reopened subtask T226236: Upload docker-ce 18.06.3 upstream package for Stretch as Open.Jul 18 2019, 1:16 PM
Jdforrester-WMF edited projects, added Release-Engineering-Team-TODO (201908); removed Release-Engineering-Team-TODO (201907).Jul 29 2019, 4:35 PM
Jdforrester-WMF moved this task from INBOX to Ready on the Release-Engineering-Team-TODO (201908) board.
Paladox subscribed.Aug 2 2019, 10:17 PM
hashar edited projects, added Release-Engineering-Team-TODO (201909); removed Release-Engineering-Team-TODO (201908).Sep 5 2019, 5:13 PM
hashar renamed this task from Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch to Rebuild integration-slave-docker-* instances to use less RAM, new name.Sep 19 2019, 9:50 AM
hashar closed subtask T226236: Upload docker-ce 18.06.3 upstream package for Stretch as Declined.
hashar updated the task description. (Show Details)

Will do the Stretch upgrade later on when I can also handle the upgrade to a more recent Docker daemon. Lets stick to the current stack for now.

gerritbot added a comment.Sep 20 2019, 1:01 PM

Change 538259 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] contint: pin Docker on stretch

https://gerrit.wikimedia.org/r/538259

gerritbot added a project: Patch-For-Review.Sep 20 2019, 1:01 PM
hashar renamed this task from Rebuild integration-slave-docker-* instances to use less RAM, new name to Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch.Sep 20 2019, 1:02 PM
hashar updated the task description. (Show Details)
hashar updated the task description. (Show Details)
hashar updated the task description. (Show Details)Sep 20 2019, 1:13 PM

Eventually I wanted to reuse the exact same Docker package on Stretch (T226236) which got rejected. After some madness that seems to work (so far). The Stretch instances would receive Docker 18.09.7 from thirdparty/ci instead of 18.06.2 on Jessie.

There is now:

integration-agent-docker-1001172.16.7.14424G RAM8vCPUs

https://integration.wikimedia.org/ci/computer/integration-agent-docker-1001/

Stashbot added a comment.Sep 20 2019, 1:14 PM

Mentioned in SAL (#wikimedia-releng) [2019-09-20T13:14:32Z] <hashar> Pooled integration-agent-docker-1001 based on Stretch # T226233

gerritbot added a comment.Sep 20 2019, 1:33 PM

Change 538259 merged by Muehlenhoff:
[operations/puppet@production] contint: pin Docker on stretch

https://gerrit.wikimedia.org/r/538259

Maintenance_bot removed a project: Patch-For-Review.Sep 20 2019, 2:10 PM
Stashbot added a comment.Sep 20 2019, 2:47 PM

Mentioned in SAL (#wikimedia-releng) [2019-09-20T14:47:27Z] <hashar> Pooled integration-agent-docker-1002 integration-agent-docker-1003 based on Stretch # T226233

Stashbot added a comment.Sep 20 2019, 3:14 PM

Mentioned in SAL (#wikimedia-releng) [2019-09-20T15:14:26Z] <hashar> Pooled integration-agent-docker-1004 based on Stretch # T226233

Stashbot added a comment.Sep 20 2019, 3:16 PM

Mentioned in SAL (#wikimedia-releng) [2019-09-20T15:16:20Z] <hashar> Pooled integration-agent-docker-1005 based on Stretch # T226233

hashar mentioned this in T233440: Raise quota for integration project.Sep 20 2019, 5:06 PM
hashar added a comment.Sep 23 2019, 6:37 PM

I have updated https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup

Stashbot added a comment.Sep 23 2019, 6:43 PM

Mentioned in SAL (#wikimedia-releng) [2019-09-23T18:43:52Z] <hashar> Added integration-agent-docker-1006 # T226233

Stashbot added a comment.Sep 23 2019, 7:06 PM

Mentioned in SAL (#wikimedia-releng) [2019-09-23T19:06:37Z] <hashar> Added integration-agent-docker-1007 # T226233

Stashbot added a comment.Sep 23 2019, 7:09 PM

Mentioned in SAL (#wikimedia-releng) [2019-09-23T19:09:11Z] <hashar> Added integration-agent-docker-1008 # T226233

Stashbot added a comment.Sep 23 2019, 8:23 PM

Mentioned in SAL (#wikimedia-releng) [2019-09-23T20:23:12Z] <hashar> Added integration-agent-docker-1009 # T226233

Stashbot added a comment.Sep 23 2019, 8:37 PM

Mentioned in SAL (#wikimedia-releng) [2019-09-23T20:37:22Z] <hashar> Added integration-agent-docker-1010 # T226233

Stashbot added a comment.Sep 23 2019, 8:38 PM

Mentioned in SAL (#wikimedia-releng) [2019-09-23T20:38:21Z] <hashar> Added integration-agent-docker-1011 # T226233

Stashbot added a comment.Sep 23 2019, 8:42 PM

Mentioned in SAL (#wikimedia-releng) [2019-09-23T20:42:56Z] <hashar> Added integration-agent-docker-1012 # T226233

hashar closed this task as Resolved.Sep 23 2019, 8:49 PM

All the bigram instances are gone \o/

Krinkle awarded a token.Sep 30 2019, 10:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL