Page MenuHomePhabricator
Create Task
Maniphest T106649

Upgrade beta cluster puppet master from Precise to Trusty
Closed, ResolvedPublic
Actions

Description

From Releng checkin on 2015-07-21, we should upgrade the puppet master on Beta-Cluster-Infrastructure to Trusty and a standalone instance. The current one has performances issues and collocated with the salt master.

Notes:

Puppet master: time to migrate to Trusty!
Save cherry picked patches
Delete instance, recreate one with same name (integration-puppetmaster), being bigger and running Trusty
Run puppet on all instances via salt to request new cert
Wait for puppet autosigner to kick in
Profit \O/

Related Objects
Search...

Event Timeline

hashar created this task.Jul 23 2015, 8:07 AM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a project: Beta-Cluster-Infrastructure.
hashar subscribed.
Restricted Application added subscribers: Luke081515, Aklapper. · View Herald TranscriptJul 23 2015, 8:07 AM
hashar added a comment.Jul 23 2015, 8:24 AM

Created deployment-puppetmaster m1.medium

https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-puppetmaster.deployment-prep.eqiad.wmflabs

hashar added a comment.Jul 23 2015, 8:42 AM

Disabled puppet agent on all beta cluster instances.

Changed puppet master in wikitech hiera https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment-prep&diff=171624&oldid=171465

hashar added a comment.EditedJul 23 2015, 8:52 AM

Reapplying the cherry-picked changes with latest patchsets and in this order:

https://gerrit.wikimedia.org/r/#q,I109c732b8b0b1a2a7c53c565f3a927103cb78923,n,z
git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/94/217794/2 && git cherry-pick FETCH_HEAD

https://gerrit.wikimedia.org/r/#q,Ie063022b7b4bfc7fa4e1dac2fd15f32bf3a02d3d,n,z
git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/18/210618/8 && git cherry-pick FETCH_HEAD

https://gerrit.wikimedia.org/r/#q,I44e9881d67345205e5ec97e0aab0f58ecb9ac751,n,z
git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/01/223301/1 && git cherry-pick FETCH_HEAD

https://gerrit.wikimedia.org/r/#q,Ib10deb5b4e42d440c5deff0897e714174f3e38fe,n,z
git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/16/158016/14 && git cherry-pick FETCH_HEAD

https://gerrit.wikimedia.org/r/#q,I5c0a1fd283cc545ab073a514e933a87d5fe23996,n,z
git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/29/224829/2 && git cherry-pick FETCH_HEAD

https://gerrit.wikimedia.org/r/#q,Ib206f3820e5aa11ff6d26e777609b5692f74dd4f,n,z
git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/91/212291/5 && git cherry-pick FETCH_HEAD

hashar added a comment.Jul 23 2015, 9:38 AM

Migrating all instances using:

sudo -s
find /var/lib/puppet -type d -name ssl -exec rm -fR {} \;
find /var/lib/puppet -type d -name ssl -exec rm -fR {} \;
puppet agent --enable
puppet agent -tv
sleep 12
puppet agent -tv

I have a lame autosigner on deployment-puppetmaster:

while true; do date; puppet ca list|awk '{print $1}'|xargs -n1 puppet ca sign ; sleep 10; done;
hashar closed this task as Resolved.Jul 23 2015, 10:40 AM
hashar claimed this task.

All instances have been switched:

root@deployment-salt:~# salt '*' cmd.run 'grep "server =" /etc/puppet/puppet.conf'
deployment-fluorine.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-sca02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-cache-text03.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-urldownloader.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-parsoid05.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-videoscaler01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-cache-upload02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-redis01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-elastic05.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-mx.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-pdf02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-cxserver03.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-mediawiki01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-sca01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-logstash2.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-parsoidcache02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-redis02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-cache-bits01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-apertium01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-cache-text02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-cache-mobile03.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-puppetmaster.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-zookeeper01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-upload.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-restbase02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
mira.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-memc03.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-kafka02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-restbase01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-mediawiki03.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-stream.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-mediawiki02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-elastic07.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-sentry2.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-memc04.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-elastic06.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-db2.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-zotero01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-pdf01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-jobrunner01.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-mathoid.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-bastion.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-elastic08.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-memc02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-test.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-db1.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-eventlogging02.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
deployment-salt.deployment-prep.eqiad.wmflabs:
    server = deployment-puppetmaster.deployment-prep.eqiad.wmflabs
root@deployment-salt:~#
hashar reopened this task as Open.Jul 23 2015, 10:43 AM

Forgot to migrate the labs/private.git local hacks :-(

hashar added a comment.Jul 23 2015, 10:51 AM

Applied on labs/private:

0001-LOCALHACK-Keys-for-mwdeploy-user.patch
0002-Hack-touch-releases-id_rsa.upload.patch
hashar closed subtask T106663: puppet fails on deployment-bastion attempts to access puppet:///private/ssh/tin/mwdeploy_rsa as Resolved.Jul 23 2015, 10:54 AM
hashar closed this task as Resolved.Jul 23 2015, 11:01 AM

Cleaned puppetmaster self config from deployment-salt https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Deployment-salt.deployment-prep.eqiad.wmflabs&diff=171629&oldid=168972

Removed /var/lib/git

dpkg --purge puppetmaster puppetmaster-common

Eevans closed subtask T106657: puppet broken on deployment-restbase01 and deployment-restbase02 as Resolved.Jul 23 2015, 3:20 PM
hashar closed subtask T106655: puppet run broken on deployment-fluorine as Resolved.Jul 27 2015, 7:32 PM
BBlack closed subtask T106662: deployment-parsoidcache02 fails varnish VCL compilation as Resolved.Jul 28 2015, 3:11 AM
hashar closed subtask T106658: Can not ssh to beta cluster instance deployment-apertium01 as Resolved.Jul 28 2015, 9:49 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits