Page MenuHomePhabricator

Upgrade beta cluster puppet master from Precise to Trusty
Closed, ResolvedPublic

Description

From Releng checkin on 2015-07-21, we should upgrade the puppet master on Beta-Cluster-Infrastructure to Trusty and a standalone instance. The current one has performances issues and collocated with the salt master.

Notes:

Puppet master: time to migrate to Trusty!
Save cherry picked patches
Delete instance, recreate one with same name (integration-puppetmaster), being bigger and running Trusty
Run puppet on all instances via salt to request new cert
Wait for puppet autosigner to kick in
Profit \O/

Event Timeline

hashar created this task.Jul 23 2015, 8:07 AM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a subscriber: hashar.
Restricted Application added subscribers: Luke081515, Aklapper. · View Herald TranscriptJul 23 2015, 8:07 AM

Disabled puppet agent on all beta cluster instances.

Changed puppet master in wikitech hiera https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment-prep&diff=171624&oldid=171465

Migrating all instances using:

sudo -s
find /var/lib/puppet -type d -name ssl -exec rm -fR {} \;
find /var/lib/puppet -type d -name ssl -exec rm -fR {} \;
puppet agent --enable
puppet agent -tv
sleep 12
puppet agent -tv

I have a lame autosigner on deployment-puppetmaster:

while true; do date; puppet ca list|awk '{print $1}'|xargs -n1 puppet ca sign ; sleep 10; done;
hashar closed this task as Resolved.Jul 23 2015, 10:40 AM
hashar claimed this task.

All instances have been switched:

root@deployment-salt:~# salt '*' cmd.run 'grep "server =" /etc/puppet/puppet.conf'
deployment-fluorine.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-sca02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-cache-text03.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-urldownloader.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-parsoid05.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-videoscaler01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-cache-upload02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-redis01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-elastic05.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-mx.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-pdf02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-cxserver03.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-mediawiki01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-sca01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-logstash2.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-parsoidcache02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-redis02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-cache-bits01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-apertium01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-cache-text02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-cache-mobile03.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-puppetmaster.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-zookeeper01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-upload.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-restbase02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
mira.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-memc03.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-kafka02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-restbase01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-mediawiki03.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-stream.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-mediawiki02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-elastic07.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-sentry2.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-memc04.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-elastic06.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-db2.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-zotero01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-pdf01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-jobrunner01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-mathoid.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-bastion.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-elastic08.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-memc02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-test.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-db1.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-eventlogging02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-salt.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
root@deployment-salt:~#
hashar reopened this task as Open.Jul 23 2015, 10:43 AM

Forgot to migrate the labs/private.git local hacks :-(

Applied on labs/private:

0001-LOCALHACK-Keys-for-mwdeploy-user.patch
0002-Hack-touch-releases-id_rsa.upload.patch
hashar closed this task as Resolved.Jul 23 2015, 11:01 AM

Cleaned puppetmaster self config from deployment-salt https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Deployment-salt.deployment-prep.eqiad.wmflabs&diff=171629&oldid=168972

Removed /var/lib/git

dpkg --purge puppetmaster puppetmaster-common