Page MenuHomePhabricator
Create Task
Maniphest T224591

Migrate contint* hosts to Buster
Closed, ResolvedPublic
Actions

Description

The CI production servers are using Debian Jessie and have to be upgrade. We will go for Buster. The hosts are:

HostRole
contint1001.wikimedia.orgPrimary
contint2001.wikimedia.orgSpare

The services to be migrated are:

  • docker-pkg
    • Buster is supported since Gerrit 585451
    • Updated on April 2nd. Will need to be redeployed after upgrade: scap deploy --limit contint.*
  • Docker
    • We can afford to loose the containers. They will be redownloaded from the registry if need be.
  • Pipeline containers building

Migration

The overall sequence is to upgrade contint2001, migrate the services to it, upgrade contint1001, move the services back to contint1001.

Zuul to scap

Zuul has been deployed using a Debian package but that methods is painful for everyone. On Buster we will deploy it using scap. We can do all the scap related work before upgrading from Jessie to Buster, we need to feature switch based on the target distribution so that a host still reies on the Debian package as long as it is still using Jessie.

  • Craft a scap deployment repository for Zuul
  • Get puppet patches to vary based on the Distribution

The deployment on a host can only be done after it has been upgraded to Buster.

contint2001 upgrade

zuul-merger
~~~~~~~~~

The sole production service being run on contint2001 is zuul-merger:

docker-pkg
~~~~~~~~

For contint2001:

Puppet run on contint2001 without errors / missing packages built:

  • E: Unable to locate package blubber
  • E: Unable to locate package zuul
  • E: Unable to locate package helm
  • E: Unable to locate package helmfile
  • E: Unable to locate package helm-diff
  • E: Unable to locate package kubernetes-client
  • mod_php_7.3 - ERROR: Module mpm_event is enabled - cannot proceed due to conflicts. It needs to be disabled first (known issue -> https://gerrit.wikimedia.org/r/c/operations/puppet/+/451206)

Jenkins agent
~~~~~~~~~~~

  • Add contint2001 as a Jenkins agent (copying contint1001)
  • Disable contint1001 agent
  • Update jobs in integration/config to point to contint2001

Migrate

  • Stop zuul, zuul, jenkins on contint1001
  • rsync data
  • change DNS backend for contint.wikimedia.org
  • update fabfile to have zuul reloaded on contint2001 instead of contint1001
  • Set contint2001 as master in Puppet / Hiera
  • Start Jenkins, verify that agents are connected and jobs set
  • Start Zuul scheduler

contint1001 upgrade

  • reinstall with Buster
  • deploy the zuul scap to the machine: scap deploy --limit contint1001 - deployed as part of
  • redeploy docker-pkg scap deploy --limit contint1001
  • update the fabfile.py for deploy_docker We now use contint.wikimedia.org
  • Stop zuul, zuul-merger, jenkins on contint2001
  • rsync data
  • change DNS backend for contint.wikimedia.org
  • update fabfile to have Zuul reloaded on contint1001 instead of contint2001 We now use contint.wikimedia.org
  • Set contint1001 as master in Puppet / Hiera
  • Start Jenkins, verify that agents are connected and jobs set
  • Start Zuul scheduler

Details

Due Date
Mar 29 2020, 10:00 PM
SubjectRepoBranchLines +/-
zuul: replace user/group with systemd-sysuser and reserved UIDoperations/puppetproduction+13 -17
jenkins: replace system user/group with systemd-sysuseroperations/puppetproduction+13 -15
admins: add system user for jenkins, reserve UID 903operations/puppetproduction+4 -1
contint: fix git cloning of docroot for integration.wm.orgoperations/puppetproduction+5 -6
ci:master: prepare for contint switchoveroperations/puppetproduction+2 -2
contint: move Docker data to /srv/dockeroperations/puppetproduction+1 -1
partman: switch contint1001 to raid10-4dev, previous recipe is goneoperations/puppetproduction+1 -1
DHCP: switch contint1001 from jessie to buster installeroperations/puppetproduction+0 -1
zuul: add convenience link to 'zuul' binoperations/puppetproduction+4 -0
jenkins: master should stick to java 8operations/puppetproduction+7 -37
jenkins/icinga: fix process monitoring after change in command lineoperations/puppetproduction+1 -1
Initially use Java 8 for contint on Busteroperations/puppetproduction+13 -1
jenkins: add missing /srv/jenkins diroperations/puppetproduction+7 -0
switch contint from 1001 to 2001operations/dnsmaster+1 -1
contint: switch jenkins/zuul/gearman to contint2001operations/puppetproduction+7 -7
fab: deploy zuul config on contint2001integration/configmaster+1 -1
Switch jobs to contint2001integration/configmaster+13 -13
contint: move common and default Hiera settings to role leveloperations/puppetproduction+10 -15
contint: move Docker data out of / on contint2001operations/puppetproduction+1 -0
contint: ignore more Docker partitions disk checksoperations/puppetproduction+1 -1
tlsproxy::envoy: allow limiting firewall srangeoperations/puppetproduction+6 -1
Rebuild for Busterblubberdebian+6 -0
ATS: use contint service alias as backend for integration.wm.orgoperations/puppetproduction+1 -1
add contint.wikimedia.org service alias for contint machinesoperations/dnsmaster+2 -0
contint: allow masters to ssh to themselvesoperations/puppetproduction+16 -1
contint: enable zuul-merger on contint2001operations/puppetproduction+0 -1
zuul: inject Gerrit ssh public key to known_hostsoperations/puppetproduction+9 -0
Profile to inject Gerrit ssh public key to known_hostsoperations/puppetproduction+18 -8
zuul: add missing ssh public key for zuul-mergeroperations/puppetproduction+12 -0
zuul: create /var/log/zuuloperations/puppetproduction+7 -0
zuul: use zuul-deployers not contint-rootoperations/puppetproduction+8 -2
scap: fix deploy_userintegration/zuul/deploymaster+1 -1
scap: strip 'integration' from git_repointegration/zuul/deploymaster+1 -1
fab: use contint2001 to build Docker imagesintegration/configmaster+3 -2
ci: remove blubber Debian packageoperations/puppetproduction+0 -5
zuul: fix dependency on /etc/zuul and package if on busteroperations/puppetproduction+13 -1
Sync jenkins to thirdparty/cioperations/puppetproduction+2 -0
contint: use package_from_component, stop using docker classoperations/puppetproduction+10 -17
ci: Use docker.io on Busteroperations/puppetproduction+18 -10
ci: add parameter to stop or mask zuul-merger service in codfwoperations/puppetproduction+21 -5
ci: allow rsyncing of data dirs for server migrationsoperations/puppetproduction+38 -0
ci::docker: add same docker version also for busteroperations/puppetproduction+1 -0
ci::httpd: add support for buster PHP 7.3operations/puppetproduction+2 -0
install_server: switch contint2001 from jessie to busteroperations/puppetproduction+1 -1
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedakosiarisT249724 Track and remove jessie based container images from production
 ResolvedJdforrester-WMFT224908 Drop jessie testing support
 ResolvedJdforrester-WMFT224906 Drop php56 testing support
 ResolvedJdforrester-WMFT211784 Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects
 InvalidJdforrester-WMFT211785 Upgrade the mobileapps CI job from npm3 to npm6
 ResolvedMoritzMuehlenhoffT203239 Create Debian packages for Node.js 10 upgrade
 ResolvedKrinkleT213944 Jenkins jobs for npm-test fail on project with deps on node-gyp which requires python2.7
 ResolvedKrinkleT215562 npm 6 consistently fails with "Z_DATA_ERROR: invalid distance too far back" on some repos
 ResolvedhasharT217545 Update selenium-daily-beta-* jobs to node10/npm6
 ResolvedJdforrester-WMFT222406 Switch quibble-based CI jobs from node6 to node10
 ResolvedJdforrester-WMFT224983 mediawiki-phpunit-coverage-patch-docker fails to install fibers@3.1.1
 DeclinedJdforrester-WMFT224997 Update MobileFrontend-npm-run-lint-modules-docker to run node10
 DuplicateNoneT224978 WikibaseMediaInfo selenium tests failing when run against beta commons
 ResolvedMilimetricT228451 Fix the analytics/mediawiki-storage repo to work on node10
 ResolvedMilimetricT228452 Fix the analytics/wikistats2 repo to work on node10
 ResolvedLadsgroupT228453 Fix the data-values/value-view repo to work on node10
 ResolvedakosiarisT218733 Migrate mobileapps to k8s and node 10
 DeclinedNoneT215539 Node.js 10 changes encoding for at least one Georgian character
 Resolved• MhollowayT258186 Investigate why mobileapps in k8s "/{domain}/v1/data/css/mobile/site" endpoint takes way longer than on scb to complete
 DuplicateNoneT225107 Migrate recommendation-api to node 10
 ResolvedNoneT225678 Migrate 3d2png to k8s
 ResolvedNoneT267327 Run latest Thumbor on Docker with Buster + Python 3
 DeclinedNoneT269215 Blubber "copies" and "builder command" steps should run in the opposite order
 ResolvedMSantosT217114 Migrate Proton to k8s and nodejs 10
 DuplicateNoneT228907 Migrate the wikimedia-portals-build timed CI job to node10
 ResolvedjeenaT213806 Migrate wikimedia-portals-build to Docker container
 ResolvedJdforrester-WMFT237479 Update the wikimedia-portals repo's CI/linting code for various security issues
 ResolvedJdrewniakT247996 Fix issues with Gulp 4 migration
 DuplicateNoneT229276 Fix the data-values/value-view repo to work on node10
 ResolvedJdforrester-WMFT230841 Migrate documentation generation to Node 10.15.2 from node 6.11.0
 ResolvedJdforrester-WMFT235570 Move the OOUI repo to a new custom docker image for node10 and php72
 ResolvedJdforrester-WMFT247536 Migrate mediawiki-core-jsduck-docker-publish off node 6 so it works again
 ResolvedMoritzMuehlenhoffT224549 Track remaining jessie systems in production
 ResolvedhasharT249268 Reduce size of artifacts stored on the CI Jenkins master
 ResolvedhasharT224591 Migrate contint* hosts to Buster
 DeclinedMoritzMuehlenhoffT226236 Upload docker-ce 18.06.3 upstream package for Stretch
 InvalidthciprianiT239880 Replacement hardware for buster/stretch upgrade of contint1001 and contint2001
 Resolved• mmodellT215458 Convert zuul to use scap
 ResolvedhasharT240551 Remove Zuul Debian package from WMCS instances
 ResolvedJoeT249110 Build and publish a python2 based container to build wheels
 ResolvedJMeybohmT249812 Rebuild helm/helm-diff for buster-wikimedia
 ResolvedJMeybohmT250479 Rebuild helmfile for buster-wikimedia

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.May 5 2020, 1:33 PM

Change 594477 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] contint: switch jenkins/zuul from contint1001 to contint2001

https://gerrit.wikimedia.org/r/594477

gerritbot added a comment.May 5 2020, 1:43 PM

Change 594480 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] switch contint from 1001 to 2001

https://gerrit.wikimedia.org/r/594480

gerritbot added a comment.May 6 2020, 2:55 PM

Change 594475 merged by Dzahn:
[operations/puppet@production] contint: move common and default Hiera settings to role level

https://gerrit.wikimedia.org/r/594475

Dzahn added a comment.May 7 2020, 7:51 AM

@hashar So.. when do we schedule the maintenance window early next week? Monday?

Dzahn updated the task description. (Show Details)May 11 2020, 8:27 AM
Stashbot added a comment.May 11 2020, 8:32 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-11T08:32:36Z] <mutante> rsynced data from contint1001 to contint2001 - pathes per T224591#6039192 for the migration later today

Stashbot added a comment.May 11 2020, 9:07 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-11T09:07:32Z] <mutante> contint1001 - rsync -avpz --delete /srv/jenkins/ rsync://contint2001.wikimedia.org/ci--srv-/jenkins/ (T224591)

hashar updated the task description. (Show Details)May 11 2020, 12:04 PM
gerritbot added a comment.May 11 2020, 12:09 PM

Change 595511 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Switch jobs to contint2001

https://gerrit.wikimedia.org/r/595511

gerritbot added a comment.May 11 2020, 12:10 PM

Change 595512 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] fab: deploy zuul config on contint2001

https://gerrit.wikimedia.org/r/595512

gerritbot added a comment.May 11 2020, 12:14 PM

Change 595511 merged by jenkins-bot:
[integration/config@master] Switch jobs to contint2001

https://gerrit.wikimedia.org/r/595511

gerritbot added a comment.May 11 2020, 12:14 PM

Change 595512 merged by jenkins-bot:
[integration/config@master] fab: deploy zuul config on contint2001

https://gerrit.wikimedia.org/r/595512

Stashbot added a comment.May 11 2020, 12:14 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-11T12:14:47Z] <hashar> shutting down Zuul and Jenkins for system switch # T224591

gerritbot added a comment.May 11 2020, 12:27 PM

Change 594480 merged by Dzahn:
[operations/dns@master] switch contint from 1001 to 2001

https://gerrit.wikimedia.org/r/594480

hashar updated the task description. (Show Details)May 11 2020, 12:28 PM
gerritbot added a comment.May 11 2020, 12:29 PM

Change 594477 merged by Dzahn:
[operations/puppet@production] contint: switch jenkins/zuul/gearman to contint2001

https://gerrit.wikimedia.org/r/594477

Stashbot added a comment.May 11 2020, 12:50 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-11T12:50:26Z] <hashar> Pointing CI Jenkins to contint2001 Gearman server T224591

gerritbot added a comment.May 11 2020, 12:57 PM

Change 595521 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] jenkins: add missing /srv/jenkins dir

https://gerrit.wikimedia.org/r/595521

Stashbot added a comment.May 11 2020, 1:36 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-11T13:36:31Z] <hashar> Rolling back CI system switch to previous known state # T224591

gerritbot added a comment.May 11 2020, 1:42 PM

Change 595525 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] contint: fix git cloning of docroot for integration.wm.org

https://gerrit.wikimedia.org/r/595525

hashar added a comment.May 11 2020, 2:02 PM

The upgrade itself went well:

  • the rsync of data from contint1001 to contint2001 finished just in time
  • our rsync configuration cause file ownership to be transferred by uid instead of name which caused us to do a lot of chown to fix them
  • the zuul scheduler deployed from scap did process events. I have tested it locally.

In Jenkins, the jobs requests landed in its build queue but the agent executors rejected them. Each claiming they were busy. It might be due to the different java version being used (8 vs 11) which in turn might point at the Gearman plugin :-\ We have thus rolled back to contint1001 which took just a couple of minutes.

Follow up actions

Try to reproduce the issue with same zuul/jenkins/java11

Moritz pointed out we have a java 8 package for Buster: deb http://apt.wikimedia.org/wikimedia buster-wikimedia component/jdk8

Find a fix for rsync to keep the user/group either:

  • ensure a consistent uid accross the fleet for jenkins and zuul users
  • update rsyncd.conf which has use chroot which defaults to force numeric ids and thus prevent the name/id mapping to occur
gerritbot added a comment.May 11 2020, 2:09 PM

Change 595531 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Initially use Java 8 for contint on Buster

https://gerrit.wikimedia.org/r/595531

hashar mentioned this in T252310: pywikibot get merge rejections due to zuul-merger not being able to update tags.May 11 2020, 2:54 PM
gerritbot added a comment.May 12 2020, 7:37 AM

Change 595531 merged by Dzahn:
[operations/puppet@production] Initially use Java 8 for contint on Buster

https://gerrit.wikimedia.org/r/595531

gerritbot added a comment.May 12 2020, 8:19 AM

Change 595521 merged by Dzahn:
[operations/puppet@production] jenkins: add missing /srv/jenkins dir

https://gerrit.wikimedia.org/r/595521

hashar mentioned this in T252428: Make helm upgrades atomic.May 13 2020, 7:00 PM
Stashbot added a comment.May 18 2020, 9:46 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-18T09:46:37Z] <mutante> contint2001 - apt-get remove --purge openjdk-11-* - T224591

gerritbot added a comment.May 18 2020, 2:40 PM

Change 597090 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] jenkins: master should stick to java 8

https://gerrit.wikimedia.org/r/597090

gerritbot added a comment.May 19 2020, 7:37 AM

Change 597090 merged by Dzahn:
[operations/puppet@production] jenkins: master should stick to java 8

https://gerrit.wikimedia.org/r/597090

gerritbot added a comment.May 19 2020, 10:40 AM

Change 597255 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] jenkins/icinga: fix process monitoring after change in command line

https://gerrit.wikimedia.org/r/597255

gerritbot added a comment.May 19 2020, 10:46 AM

Change 597255 merged by Dzahn:
[operations/puppet@production] jenkins/icinga: fix process monitoring after change in command line

https://gerrit.wikimedia.org/r/597255

hashar added a comment.May 20 2020, 5:31 AM

We now have Jenkins pinned to java8 so I guess we to do the switch over again.

@Dzahn would you be available at the beginning of next week (Monday/Tuesday)? (we can sync up over irc to find a good time)

Dzahn added a comment.May 20 2020, 7:46 AM
In T224591#6150943, @hashar wrote:

@Dzahn would you be available at the beginning of next week (Monday/Tuesday)? (we can sync up over irc to find a good time)

Yes, some time between Monday 9 to 5 or Tuesday 9 to 4. Or simply tomorrow, Thursday, even preferable.

hashar added a comment.May 22 2020, 2:01 PM

After discussion with @Dzahn we will do the switch Wednesday May 27th at 9:30 CEST (7:30 UTC).

gerritbot added a comment.May 22 2020, 3:25 PM

Change 598068 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] zuul: add convenience link to 'zuul' bin

https://gerrit.wikimedia.org/r/598068

gerritbot added a comment.May 26 2020, 7:18 AM

Change 598068 merged by Dzahn:
[operations/puppet@production] zuul: add convenience link to 'zuul' bin

https://gerrit.wikimedia.org/r/598068

Stashbot added a comment.May 27 2020, 9:39 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-27T09:39:11Z] <hashar> Stopping Zuul and Jenkins CI for scheduled maintenance # T224591

Dzahn added a comment.May 27 2020, 10:22 AM
2020-05-27

    10:15 hashar: contint2001: starting zuul
    10:15 hashar: contint2001: started jenkins
    10:03 mutante: contint2001 - find /var/lib/jenkins -user statsite -exec chown -h jenkins:jenkins {} \;
    10:02 mutante: repeated rsync of /var/lib/jenkins with -p ; find /var/lib/jenkins -group bacula -user statsite -exec chown -h jenkins:jenkins {} \;
    09:55 hashar: contint2001: starting jenkins
    09:54 hashar: contint1001 / contint2001 : deleted obsolete files /var/lib/jenkins/.git and /var/lib/jenkins/jobs/_shared/
    09:52 mutante: contint2001 - find /var/lib/jenkins -user statsite -exec chown -h jenkins:jenkins {} \;
    09:49 mutante: contint2001 - find /var/lib/jenkins -group bacula -user statsite -exec chown jenkins:jenkins {} \;
    09:48 hashar: contint2001: unmasked jenkins and started it
    09:42 mutante: switching CI backend from contint1001 to contint2001
    09:40 mutante: repeated rsync -avp --delete /var/lib/zuul/ rsync://contint2001.wikimedia.org/ci--var-lib-zuul-
    09:40 hashar: contint1001: masked jenkins and zuul
    09:39 mutante: repeated rsync -avp --delete /var/lib/jenkins/ rsync://contint2001.wikimedia.org/ci--var-lib-jenkins-
    09:39 hashar: Stopping Zuul and Jenkins CI for scheduled maintenance # T224591
    08:52 hashar: contint1001: find /srv/jenkins/builds/operations-puppet-wmf-style-guide -type f -name '*.tmp' -delete # T253729
    08:08 hashar: contint1001 / contint2001 : deleted unused /var/lib/zuul/git (the real one is /srv/zuul/git )
    08:02 mutante: contint2001 - chown root:root /var/lib/zuul/git
    07:45 hashar: contint2001 also fixing symlink permissions: sudo find /var/lib/jenkins -not -user jenkins -exec chown -h jenkins:jenkins {} +
    07:35 mutante: contint2001 - find /var/lib/jenkins -group bacula -user jenkins -exec chown jenkins:jenkins {} \;
    07:30 mutante: contint2001 - find /var/lib/jenkins -user statsite -exec chown jenkins {} \;
    07:26 mutante: contint2001 - chown -R zuul:zuul /var/lib/zuul/
    07:26 mutante: contint1001:~# rsync -avpz --delete /srv/jenkins/ rsync://contint2001.wikimedia.org/ci--srv-/jenkins/
    07:25 mutante: contint1001:~# rsync -avp --delete /var/lib/jenkins/ rsync://contint2001.wikimedia.org/ci--var-lib-jenkins-
    07:25 mutante: contint1001:~# rsync -avp --delete /var/lib/zuul/ rsync://contint2001.wikimedia.org/ci--var-lib-zuul-
hashar added a comment.May 27 2020, 10:25 AM

I forgot about having the data synchronized ahead of the maintenance window which caused a 2 hours delay. We then had to fight with file permissions changes introduced by rsync. Eventually the service got switched and seems operational now. I will continue monitoring but I am guessing it will be fine from now on.

Dzahn raised the priority of this task from Medium to High.May 27 2020, 10:34 AM
Stashbot added a comment.May 27 2020, 3:33 PM

Mentioned in SAL (#wikimedia-releng) [2020-05-27T15:33:39Z] <James_F> Zuul: Re-pulling config forwards ~2 weeks on contint2001; forgotten in T224591?

Jdforrester-WMF added a subscriber: Reedy.May 27 2020, 3:38 PM

Sorry, my mistake, not two weeks, just lots and lots of semi-familiar changes, apparently all merged but not deployed so far today by @hashar and one by @Reedy: abcca40b..9fb87190.

Dzahn added a comment.May 28 2020, 10:10 AM

Hashar confirmed things are working now.

We agreed that next week I will reimage contint1001 to buster.

Jdforrester-WMF updated the task description. (Show Details)May 28 2020, 3:44 PM
gerritbot added a comment.Jun 2 2020, 7:57 AM

Change 601645 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] DHCP: switch contint1001 from jessie to buster installer

https://gerrit.wikimedia.org/r/601645

gerritbot added a comment.Jun 2 2020, 8:05 AM

Change 601645 merged by Dzahn:
[operations/puppet@production] DHCP: switch contint1001 from jessie to buster installer

https://gerrit.wikimedia.org/r/601645

ops-monitoring-bot added a comment.Jun 2 2020, 8:09 AM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

contint1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202006020808_dzahn_28298_contint1001_wikimedia_org.log.

ops-monitoring-bot added a comment.Jun 2 2020, 9:09 AM

Completed auto-reimage of hosts:

['contint1001.wikimedia.org']

Of which those FAILED:

['contint1001.wikimedia.org']
gerritbot added a comment.Jun 2 2020, 9:23 AM

Change 601666 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] partman: switch contint1001 to raid10-4dev, previous recipe is gone

https://gerrit.wikimedia.org/r/601666

gerritbot added a comment.Jun 2 2020, 9:29 AM

Change 601666 merged by Dzahn:
[operations/puppet@production] partman: switch contint1001 to raid10-4dev, previous recipe is gone

https://gerrit.wikimedia.org/r/601666

ops-monitoring-bot added a comment.Jun 2 2020, 9:33 AM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

contint1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202006020932_dzahn_98476_contint1001_wikimedia_org.log.

ops-monitoring-bot added a comment.Jun 2 2020, 9:33 AM

Completed auto-reimage of hosts:

['contint1001.wikimedia.org']

Of which those FAILED:

['contint1001.wikimedia.org']
ops-monitoring-bot added a comment.Jun 2 2020, 9:37 AM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

contint1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202006020937_dzahn_104332_contint1001_wikimedia_org.log.

ops-monitoring-bot added a comment.Jun 2 2020, 9:37 AM

Completed auto-reimage of hosts:

['contint1001.wikimedia.org']

Of which those FAILED:

['contint1001.wikimedia.org']
ops-monitoring-bot added a comment.Jun 2 2020, 9:39 AM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

contint1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202006020938_dzahn_105219_contint1001_wikimedia_org.log.

ops-monitoring-bot added a comment.Jun 2 2020, 9:44 AM

Completed auto-reimage of hosts:

['contint1001.wikimedia.org']

Of which those FAILED:

['contint1001.wikimedia.org']
ops-monitoring-bot added a comment.Jun 2 2020, 9:45 AM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

contint1001.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202006020945_dzahn_110360_contint1001_wikimedia_org.log.

ops-monitoring-bot added a comment.Jun 2 2020, 11:09 AM

Completed auto-reimage of hosts:

['contint1001.wikimedia.org']

and were ALL successful.

Dzahn added a comment.Jun 2 2020, 11:10 AM

@hashar contint1001 is now on buster

Dzahn reassigned this task from Dzahn to hashar.Jun 2 2020, 11:25 AM
Dzahn updated the task description. (Show Details)

re-assigning for the deployment steps that are next

hashar updated the task description. (Show Details)Jun 2 2020, 2:43 PM
hashar updated the task description. (Show Details)Jun 2 2020, 2:47 PM
hashar added a comment.Jun 2 2020, 3:08 PM

Partitions on contint1001 have changed. It used to have a secondary volume group with a 250G volume mounted at /mnt/docker. That followed up the addition of two extra SSD that went with their own lvm group T207707

Now we have four SSD on the same RAID (md0) and a single volume group. It has a 80G logical volume for / and the rest (1.4T) for /srv. I guess it is nicer this way, we just have to move Docker to write under /srv instead of /mnt/docker.

gerritbot added a comment.Jun 2 2020, 3:12 PM

Change 601760 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] contint: move Docker data to /srv/docker

https://gerrit.wikimedia.org/r/601760

gerritbot added a comment.Jun 2 2020, 3:43 PM

Change 601760 merged by Dzahn:
[operations/puppet@production] contint: move Docker data to /srv/docker

https://gerrit.wikimedia.org/r/601760

Stashbot added a comment.Jun 2 2020, 3:45 PM

Mentioned in SAL (#wikimedia-operations) [2020-06-02T15:45:54Z] <mutante> contint1001 - restarting docker afer changed data-root path (T224591)

Stashbot added a comment.Jun 2 2020, 3:48 PM

Mentioned in SAL (#wikimedia-operations) [2020-06-02T15:48:24Z] <mutante> contint1001 - rm -rf /mnt/docker (T224591)

Dzahn added a comment.Jun 11 2020, 1:57 PM

Technically this is done because both contint hosts are on buster.

Whether contint2001 or contint1001 is the "active" server could be seen as unrelated.

hashar lowered the priority of this task from High to Medium.Jun 15 2020, 8:57 AM

I would like to move the service back to contint1001. But before doing that, I am going to dig into the rsync setup to saves us from having to manually fix up the files ownerships. It is a bit of a rabbit hole, but it affected us for gerrit-test as well so it is probably worth investigating and fixing ;)

Dzahn added a comment.EditedJun 16 2020, 3:39 PM

We went into this issue several times before. Let's solve it by using the same UID for our system users (also see recent mail from Moritz about systemd-sysusers). Let's not try to solve it with rsync parameters again.

Also would prefer if it was a separate ticket. It's not really related to buster upgrade. It's a general issue we have had since the beginning of Wikimedia time when https://wikitech.wikimedia.org/wiki/UID was created.

MoritzMuehlenhoff added a comment.Jun 16 2020, 5:58 PM

Definitely, if you need a system user with fixed IDs across servers, use one in the 9xx range and reserve it in data.yaml. everything else will cause problems down the road.

hashar added a comment.Jun 18 2020, 7:53 AM

The easy trick is to have the rsync daemon to run as root without chroot, it is then able do:

  • map names (being outside of a chroot it has access to eg getpwname())
  • adjust the files ownership (being root)

Which in rsync configuration can be achieved with:

uid = root
use chroot no

Which is scary. An alternative is to keep the chroot and explicitly enable mapping with numeric ids = no, but that requires copying libraries into the chroot and (per doc) make sure they can't be written to. So that sounds all a bit too complicated.

So yeah lets reserve a uid.

hashar added a comment.Jun 18 2020, 9:00 AM

Actually we do not need any name mapping. The uid and gid are the exact same ones on each servers

$ getent passwd zuul jenkins jenkins-slave
zuul:x:497:498::/var/lib/zuul:/bin/bash
jenkins:x:499:499::/var/lib/jenkins:/bin/bash
jenkins-slave:x:498:1001::/var/lib/jenkins-slave:/bin/bash
$ getent group zuul jenkins jenkins-slave
zuul:x:498:
jenkins:x:499:
jenkins-slave:x:1001:

So I do not get why we would get the jenkins user becoming statsite or the jenkins group becoming bacula. That does not make any sense :-\

gerritbot added a comment.Jun 18 2020, 9:15 AM

Change 606394 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] ci:master: prepare for contint switchover

https://gerrit.wikimedia.org/r/606394

MoritzMuehlenhoff added a comment.Jun 18 2020, 2:48 PM
In T224591#6234196, @hashar wrote:

Actually we do not need any name mapping. The uid and gid are the exact same ones on each servers






They are unique by chance, but not by design. When a local system user is created, the next available UID in the specified range is use, but it's not necessarily deterministic. If we reimage any of the contint* servers at a later point, the excution could be different and we end up with different UIDs again.
gerritbot added a comment.Jun 18 2020, 4:29 PM
Change 606286 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/puppet@production] jenkins: replace system user/group with systemd-sysuser



https://gerrit.wikimedia.org/r/606286
Dzahn added a comment.Jun 18 2020, 4:30 PM


In T224591#6235038, @MoritzMuehlenhoff wrote:


They are unique by chance, but not by design.





Or it's likely we already changed it in the past to avoid this issue. I remember doing this on a bunch of occasions where i first just edited the UID and then used find - exec to chown all the files.  I agree though it's a bit mysterious why we had that issue during the 1001->2001 switch recently.
Dzahn added a comment.EditedJun 18 2020, 4:37 PM
I just noticed the permissions of files in /var/lib/jenkins on contint1001 are _not_ like on contint2001 again, while i definitely remember fixing them to be the same on both sides.



Also f.e. there is a file called "secret.key.not-so-secret" there that does not exist on 2001, even though we ran rsync with --delete to have exact mirrors.



Why is that?
gerritbot added a comment.Jun 18 2020, 7:42 PM
Change 606394 merged by Dzahn:

[operations/puppet@production] ci:master: prepare for contint switchover



https://gerrit.wikimedia.org/r/606394
gerritbot added a comment.Jun 22 2020, 2:46 PM
Change 595525 abandoned by Hashar:

contint: fix git cloning of docroot for integration.wm.org



Reason:

The way we deploy integration/docroot and maintain its deployment is broken. Instead we will migrate the deployment to use scap (T256005) and have Apache DocumentRoot to point to the deployed path under /srv/deployment.  That will also addresses the issue Daniel raised to me: the git workspace is polluted by files published by CI.



https://gerrit.wikimedia.org/r/595525
gerritbot added a comment.Jun 24 2020, 11:57 PM
Change 607645 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/puppet@production] admins: add system user for jenkins, reserve UID 903



https://gerrit.wikimedia.org/r/607645
gerritbot added a comment.Jun 25 2020, 4:11 PM
Change 607645 abandoned by Dzahn:

admins: add system user for jenkins, reserve UID 903



Reason:

needs to be merged into https://gerrit.wikimedia.org/r/c/operations/puppet/ /606286



https://gerrit.wikimedia.org/r/607645
hashar mentioned this in T256396: Create a runbook for switching CI master.Jun 25 2020, 5:01 PM
gerritbot added a comment.Jun 25 2020, 6:08 PM
Change 607853 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/puppet@production] zuul: replace user/group with systemd-sysuser and reserved UID



https://gerrit.wikimedia.org/r/607853
Dzahn closed this task as Resolved.Jun 25 2020, 7:32 PM
Decided with Hashar we are calling this resolved because both hosts are on buster.



We are treating the switch-back and rsync issues in a separate ticket.
hashar added a comment.Jun 25 2020, 7:33 PM
Eventually I wanted to switch back to contint1001 as part of the migration, notably due to the added network latency between contint2001 in codfw and Gerrit in eqiad. But that is a non issue in the end.



There are  bunch of follow up actions for issues that surface during the migration. Some are listed as T224591#6124875



Then, both hosts are now using Buster which was the aim of this task. As such, in accordance with @Dzahn lets mark this solved and file other tasks for the rest ;)
Dzahn mentioned this in T256422: switch contint prod server back from contint2001 to contint1001.Jun 25 2020, 8:18 PM
gerritbot added a comment.Aug 26 2020, 9:01 PM
Change 606286 abandoned by Dzahn:

[operations/puppet@production] jenkins: replace system user/group with systemd-sysuser



Reason:



https://gerrit.wikimedia.org/r/606286
gerritbot added a comment.Aug 26 2020, 9:01 PM
Change 607853 abandoned by Dzahn:

[operations/puppet@production] zuul: replace user/group with systemd-sysuser and reserved UID



Reason:



https://gerrit.wikimedia.org/r/607853
hashar mentioned this in T283891: build and import blubber package for buster and bullseye (which supports v4).Jun 14 2021, 9:40 AM
hashar mentioned this in T313360: Setup rsync for phab data on disk.Jul 20 2022, 5:28 PM
hashar mentioned this in T324659: contint2002 service implementation tracking.Jul 5 2023, 3:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL