Page MenuHomePhabricator

Migrate remaining self-hosted puppet masters to Puppet 5 / facter 3
Open, MediumPublic

Description

@Krenair migrated the main puppet masters for Cloud VPS to Puppet 5 (thanks!) and these self-hosted puppet masters are still using Puppet 4 and will break when Puppet 5-specific code gets merged to puppet.git (or when code gets merged which is buggy with < 5).

They need to be replaced with an equivalent VM running Buster (which has Puppet 5 by default) or removed if obsolete. If no puppetdb is used, they can also be upgraded to Puppet 5 by using component/puppet5 and component/facter3.

  • af-puppetmaster02.automation-framework.eqiad.wmflabs - jessie! handled in T236582: "automation-framework" Cloud VPS project jessie deprecation
  • clouddb-services-puppetmaster-01.clouddb-services.eqiad.wmflabs - (upgraded puppet/facter)
  • cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs - @Krenair working on it - shut off, ready to terminate on 11th March
  • cloudstore-puppetmaster-01.cloudstore.eqiad.wmflabs - (rebuilt on buster)
  • debmonitor-pm.sso.eqiad.wmflabs - @MoritzMuehlenhoff will handle it
  • deployment-dumps-puppetmaster02.deployment-prep.eqiad.wmflabs - @Krenair working on it - shut off, ready to terminate on 14th March
  • filippo-log-stretch01.logging.eqiad.wmflabs (deleted)
  • icingaduty-puppetmaster-1.icingaduty.eqiad.wmflabs - (removed)
  • integration-puppetmaster01.integration.eqiad.wmflabs - jessie! - T236576: "integration" Cloud VPS project jessie deprecation - @Krenair working on it - shut off, ready to terminate on 12th March
  • jbond-stretch-pm.puppet.eqiad.wmflabs - deliberately left on for now to test changes affecting old puppetmasters
  • jeh-puppetmaster.testlabs.eqiad.wmflabs (rebuilt on buster)
  • keith-puppetmaster.puppet.eqiad.wmflabs - (removed)
  • keith-puppetmaster1.puppet.eqiad.wmflabs - (removed)
  • maps-puppetmaster.maps.eqiad.wmflabs - does @TheDJ know anything about this?
  • openstack-puppetmaster-01.openstack.eqiad.wmflabs - (removed)
  • paws-puppetmaster-01.paws.eqiad.wmflabs - (upgraded puppet/facter)
  • puppet-lta.lta-tracker.eqiad.wmflabs - @Zppix
  • puppet-paladox.git.eqiad.wmflabs - @Paladox?
  • puppet-phabricator.phabricator.eqiad.wmflabs - @Paladox?
  • shinken-puppetmaster-01.shinken.eqiad.wmflabs - (rebuilt on buster)
  • toolsbeta-puppetmaster-02.toolsbeta.eqiad.wmflabs - cloud-services-team - @Krenair working on it - shut off, ready to terminate on 26th March

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 2 2020, 10:51 AM
herron triaged this task as Medium priority.Jan 3 2020, 7:43 PM
Paladox added a subscriber: Paladox.Jan 3 2020, 7:45 PM
Krenair added a comment.EditedSat, Feb 29, 10:46 PM

Worth noting there's a small army of random puppetmasters laying around running puppet 4:

  • jeh-puppetmaster.testlabs.eqiad.wmflabs
  • integration-puppetmaster01.integration.eqiad.wmflabs - jessie!
  • puppet-paladox.git.eqiad.wmflabs
  • icingaduty-puppetmaster-1.icingaduty.eqiad.wmflabs
  • af-puppetmaster02.automation-framework.eqiad.wmflabs - jessie!
  • keith-puppetmaster1.puppet.eqiad.wmflabs
  • cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs - edit: working on it
  • filippo-log-stretch01.logging.eqiad.wmflabs
  • cloudstore-puppetmaster-01.cloudstore.eqiad.wmflabs
  • puppet-lta.lta-tracker.eqiad.wmflabs
  • deployment-dumps-puppetmaster02.deployment-prep.eqiad.wmflabs - this will be either me or Ariel
  • puppet-phabricator.phabricator.eqiad.wmflabs
  • shinken-puppetmaster-01.shinken.eqiad.wmflabs
  • maps-puppetmaster.maps.eqiad.wmflabs
  • jbond-stretch-pm.puppet.eqiad.wmflabs
  • debmonitor-pm.sso.eqiad.wmflabs
  • toolsbeta-puppetmaster-02.toolsbeta.eqiad.wmflabs
  • openstack-puppetmaster-01.openstack.eqiad.wmflabs
  • paws-puppetmaster-01.paws.eqiad.wmflabs
  • keith-puppetmaster.puppet.eqiad.wmflabs
  • clouddb-services-puppetmaster-01.clouddb-services.eqiad.wmflabs

Maybe I was too quick to merge this and instead this is Migrate all Cloud VPS puppetmasters to Puppet 5 / facter 3, with the other two tasks (and others created for this list) being subtasks?

Maybe I was too quick to merge this and instead this is Migrate all Cloud VPS puppetmasters to Puppet 5 / facter 3, with the other two tasks (and others created for this list) being subtasks?

Or we move that to a separate task? From my PoV it's mostly on whoever creates a self-hosted puppet masters to catch up with large scale changes to the "upstream" puppet masters, so creating a tracking task and CCing people to the task should be enough?

Krenair added a comment.EditedMon, Mar 2, 1:22 PM

Sure, let's make this the tracking task? Or do you think we should have a separate task to track custom puppetmasters etc.?

MoritzMuehlenhoff reopened this task as Open.Tue, Mar 3, 10:41 AM
MoritzMuehlenhoff updated the task description. (Show Details)

Sure, let's make this the tracking task? Or do you think we should have a separate task to track custom puppetmasters etc.?

Ack, I turned that into a tracking task now

I think we can let people hand it as they prefer it workflow-wise, for some more complex migrations it might be useful to rather create a subtask, but some are also very likely easy wins (obsolete VMs that people simly forgot to remove etc.) which can be resolved without a sub task.

MoritzMuehlenhoff renamed this task from Migrate Cloud VPS to Puppet 5 / facter 3 to Migrate remaining self-hosted puppet masters to Puppet 5 / facter 3.Tue, Mar 3, 10:53 AM
jbond added a comment.Tue, Mar 3, 11:18 AM

in relation to jbond-stretch-pm.puppet.eqiad.wmflabs, this is avalible so i can continue to test changes work on puppet version 4. once everything elses has been migrated/updated this can be deleted without issue

fgiunchedi updated the task description. (Show Details)Tue, Mar 3, 11:21 AM
fgiunchedi updated the task description. (Show Details)
Krenair updated the task description. (Show Details)Tue, Mar 3, 9:02 PM
Krenair updated the task description. (Show Details)Tue, Mar 3, 9:50 PM
Krenair updated the task description. (Show Details)Tue, Mar 3, 9:52 PM
Krenair updated the task description. (Show Details)Tue, Mar 3, 10:01 PM
Krenair updated the task description. (Show Details)Tue, Mar 3, 10:06 PM
Krenair updated the task description. (Show Details)
Krenair added a subscriber: herron.
Krenair updated the task description. (Show Details)Tue, Mar 3, 10:08 PM
Krenair updated the task description. (Show Details)Tue, Mar 3, 10:12 PM
Krenair added subscribers: Zppix, TheDJ.

Mentioned in SAL (#wikimedia-cloud) [2020-03-04T22:33:31Z] <Krenair> Shutoff cloudinfra-internal-puppetmaster01, replaced with -02 per T241719

Krenair updated the task description. (Show Details)Wed, Mar 4, 10:34 PM

@ArielGlenn Hey, do you still need a separate puppetmaster (deployment-dumps-puppetmaster02) for deployment-snapshot01, distinct from the usual deployment-prep puppetmaster (now deployment-puppetmaster04)? If so I'll replace it with a deployment-dumps-puppetmaster03 running buster, but if not I'd like to get rid of it. From what I can tell it just has much fewer cherry-picks?

Krenair updated the task description. (Show Details)Wed, Mar 4, 10:46 PM
Krenair updated the task description. (Show Details)Thu, Mar 5, 12:40 AM

Hey @Krenair, I don't have to have it right now, but I might need it again in the future. Basically I used it when we were doing transitioning from hhvm to php or during testing serious refactoring of the mediawiki puppet module. I don't imagine having to go through something like that again for some time... but who knows. Famous last words and all that.

JHedden updated the task description. (Show Details)Thu, Mar 5, 3:37 PM
JHedden updated the task description. (Show Details)
Krenair added a comment.EditedThu, Mar 5, 9:57 PM

Hey @Krenair, I don't have to have it right now, but I might need it again in the future. Basically I used it when we were doing transitioning from hhvm to php or during testing serious refactoring of the mediawiki puppet module. I don't imagine having to go through something like that again for some time... but who knows. Famous last words and all that.

Yeah, I can see that being a use case for a separate puppetmaster. I think in this case I'll probably move deployment-snapshot to the normal deployment-prep puppetmaster and get rid of deployment-dumps-puppetmaster02. Then if and when a new such need emerges we can make a buster (or $current_latest_OS) puppetmaster for it.

Krenair updated the task description. (Show Details)Thu, Mar 5, 10:02 PM
Krenair updated the task description. (Show Details)Thu, Mar 5, 11:14 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-06T19:24:43Z] <jeh> create new puppetmaster cloudstore-puppetmaster-02 T241719

Mentioned in SAL (#wikimedia-cloud) [2020-03-06T20:39:11Z] <jeh> delete old puppetmaster cloudstore-puppetmaster-01 T241719

JHedden updated the task description. (Show Details)Fri, Mar 6, 8:40 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-06T21:09:56Z] <jeh> create new puppetmaster shinken-puppetmaster-02 T241719

Mentioned in SAL (#wikimedia-cloud) [2020-03-06T22:15:28Z] <jeh> migrate existing VMs to new shinken-puppetmaster-02 (local commits restored from shinken-puppetmaster-01) T241719

Mentioned in SAL (#wikimedia-cloud) [2020-03-06T22:17:46Z] <jeh> delete shinken-puppetmaster-01 T241719

JHedden updated the task description. (Show Details)Fri, Mar 6, 10:19 PM
Krenair updated the task description. (Show Details)Sat, Mar 7, 12:36 AM

@aborrero: Hi, do you still need openstack-puppetmaster-01? It's not got any cherry-picks in operations/puppet or labs/private, it's not got anything in volatile, it's not autosigning but all certs that have been signed on it (other than its own) are for instances that no longer appear to be in DNS.

Krenair updated the task description. (Show Details)Sun, Mar 8, 5:00 PM
Krenair updated the task description. (Show Details)Sun, Mar 8, 5:44 PM
Krenair updated the task description. (Show Details)Thu, Mar 12, 11:11 PM
Krenair updated the task description. (Show Details)
jbond updated the task description. (Show Details)Fri, Mar 13, 11:27 AM

Mentioned in SAL (#wikimedia-releng) [2020-03-17T19:49:34Z] <James_F> Deleted deployment-dumps-puppetmaster02 for T241719

Mentioned in SAL (#wikimedia-cloud) [2020-03-19T23:18:12Z] <Krenair> Shut down toolsbeta-puppet(db-01|master-02) - T241719

Krenair updated the task description. (Show Details)Thu, Mar 19, 11:18 PM
Paladox updated the task description. (Show Details)Fri, Mar 20, 12:47 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-20T13:43:38Z] <jeh> upgrade puppetmaster to v5 T241719

JHedden updated the task description. (Show Details)Fri, Mar 20, 1:49 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-20T14:03:36Z] <jeh> upgrade paws-puppetmaster-01 to v5 T241719

JHedden updated the task description. (Show Details)Fri, Mar 20, 2:05 PM
JHedden updated the task description. (Show Details)Fri, Mar 20, 2:12 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-20T14:58:33Z] <jeh> delete cloudvps project and VMs T241719

JHedden updated the task description. (Show Details)Fri, Mar 20, 2:58 PM
Krenair updated the task description. (Show Details)Thu, Mar 26, 6:30 PM