Change Details

Parent for tasks related to puppet 4 upgrade. add details Testing strategy -- [x] Set up octocatalog-diff host with access to puppet masters T177843 [x] tweak auth.conf on puppetmasters to allow compiling of catalogs from the catalog diff host T177843 [x] depool puppetmaster2001 [x] re-point puppet.codfw.wmnet CNAME at puppetmaster1001.codfw.wmnet (removing puppetmaster2001 from service) https://gerrit.wikimedia.org/r/#/c/385393/ [x] upgrade puppetmaster2001 to puppet 4.8.2 via jessie-backports package [x] validate puppetmaster2001 [x] compile catalogs for all nodes and compare v3 and v4 results using octocatalog-diff [x] fix issues, repeat until no issues observed [x] T179023 [x] T179033 [x] T179076 [x] T179077 [x] T179084 [x] T179086 [x] T179161 [x] T179162 [x] T179163 [x] T179164 [x] T179165 [x] T179166 [x] T179167 [x] T179168 [x] T179170 [x] T179171 [x] T179172 [x] T179174 [x] T179175 [x] T179176 [x] T179177 [x] T179178 [x] T179180 [x] T179290 [x] T179291 [x] T179382 [x] T179396 [x] T179400 [x] T179408 [x] fix the hole in auth.conf we introduced before [x] confirm puppet 3.8 agents play nice with puppet 4.8 master [x] upgrade puppetmaster2002 to puppet 4.8.2 via jessie-backports package Upgrade procedure -- [x] Point a small but representative sampling of prod hosts at codfw (which is now upgraded to puppet 4) [x] change the hiera "puppetmaster" variable to puppetmaster2001.codfw.wmnet [x] Move back dns for codfw, ulsfo to use puppetmaster2001, check puppet logs on all servers in those datacenters. [x] Disable puppet fleet-wide with cumin [x] upgrade puppet on all the puppetmasters in eqiad to puppet 4.8.2 via jessie-backports package [x] perform dry runs [x] perform actual runs on selected hosts [x] Reenable puppet everywhere via cumin [] Upgrade puppet to 4.8.2 on all jessie/stretch hosts. Trusty will be tackled later. Cleanup -- [] Remove puppetcompiler1001 ganeti VM [] Revert puppetmaster auth.conf change https://gerrit.wikimedia.org/r/#/c/384993/ [] Revert puppetdb ferm change https://gerrit.wikimedia.org/r/#/c/384762/ Work in progress. Open to discussion and suggestions

Parent for tasks related to puppet 4 upgrade. add details Testing strategy -- [x] Set up octocatalog-diff host with access to puppet masters T177843 [x] tweak auth.conf on puppetmasters to allow compiling of catalogs from the catalog diff host T177843 [x] depool puppetmaster2001 [x] re-point puppet.codfw.wmnet CNAME at puppetmaster1001.codfw.wmnet (removing puppetmaster2001 from service) https://gerrit.wikimedia.org/r/#/c/385393/ [x] upgrade puppetmaster2001 to puppet 4.8.2 via jessie-backports package [x] validate puppetmaster2001 [x] compile catalogs for all nodes and compare v3 and v4 results using octocatalog-diff [x] fix issues, repeat until no issues observed [x] T179023 [x] T179033 [x] T179076 [x] T179077 [x] T179084 [x] T179086 [x] T179161 [x] T179162 [x] T179163 [x] T179164 [x] T179165 [x] T179166 [x] T179167 [x] T179168 [x] T179170 [x] T179171 [x] T179172 [x] T179174 [x] T179175 [x] T179176 [x] T179177 [x] T179178 [x] T179180 [x] T179290 [x] T179291 [x] T179382 [x] T179396 [x] T179400 [x] T179408 [x] fix the hole in auth.conf we introduced before [x] confirm puppet 3.8 agents play nice with puppet 4.8 master [x] upgrade puppetmaster2002 to puppet 4.8.2 via jessie-backports package Upgrade procedure -- [x] Point a small but representative sampling of prod hosts at codfw (which is now upgraded to puppet 4) [x] change the hiera "puppetmaster" variable to puppetmaster2001.codfw.wmnet [x] Move back dns for codfw, ulsfo to use puppetmaster2001, check puppet logs on all servers in those datacenters. [x] Disable puppet fleet-wide with cumin [x] upgrade puppet on all the puppetmasters in eqiad to puppet 4.8.2 via jessie-backports package [x] perform dry runs [x] perform actual runs on selected hosts [x] Reenable puppet everywhere via cumin [] Upgrade puppet to 4.8.2 on all jessie/stretch hosts. Trusty will be tackled later. Cleanup -- [] Remove puppetcompiler1001 ganeti VM [x] Revert puppetmaster auth.conf change https://gerrit.wikimedia.org/r/#/c/384993/ [] Revert puppetdb ferm change https://gerrit.wikimedia.org/r/#/c/384762/ Work in progress. Open to discussion and suggestions

Parent for tasks related to puppet 4 upgrade. add details Testing strategy -- [x] Set up octocatalog-diff host with access to puppet masters T177843 [x] tweak auth.conf on puppetmasters to allow compiling of catalogs from the catalog diff host T177843 [x] depool puppetmaster2001 [x] re-point puppet.codfw.wmnet CNAME at puppetmaster1001.codfw.wmnet (removing puppetmaster2001 from service) https://gerrit.wikimedia.org/r/#/c/385393/ [x] upgrade puppetmaster2001 to puppet 4.8.2 via jessie-backports package [x] validate puppetmaster2001 [x] compile catalogs for all nodes and compare v3 and v4 results using octocatalog-diff [x] fix issues, repeat until no issues observed [x] T179023 [x] T179033 [x] T179076 [x] T179077 [x] T179084 [x] T179086 [x] T179161 [x] T179162 [x] T179163 [x] T179164 [x] T179165 [x] T179166 [x] T179167 [x] T179168 [x] T179170 [x] T179171 [x] T179172 [x] T179174 [x] T179175 [x] T179176 [x] T179177 [x] T179178 [x] T179180 [x] T179290 [x] T179291 [x] T179382 [x] T179396 [x] T179400 [x] T179408 [x] fix the hole in auth.conf we introduced before [x] confirm puppet 3.8 agents play nice with puppet 4.8 master [x] upgrade puppetmaster2002 to puppet 4.8.2 via jessie-backports package Upgrade procedure -- [x] Point a small but representative sampling of prod hosts at codfw (which is now upgraded to puppet 4) [x] change the hiera "puppetmaster" variable to puppetmaster2001.codfw.wmnet [x] Move back dns for codfw, ulsfo to use puppetmaster2001, check puppet logs on all servers in those datacenters. [x] Disable puppet fleet-wide with cumin [x] upgrade puppet on all the puppetmasters in eqiad to puppet 4.8.2 via jessie-backports package [x] perform dry runs [x] perform actual runs on selected hosts [x] Reenable puppet everywhere via cumin [] Upgrade puppet to 4.8.2 on all jessie/stretch hosts. Trusty will be tackled later. Cleanup -- [] Remove puppetcompiler1001 ganeti VM [x] Revert puppetmaster auth.conf change https://gerrit.wikimedia.org/r/#/c/384993/ [] Revert puppetdb ferm change https://gerrit.wikimedia.org/r/#/c/384762/ Work in progress. Open to discussion and suggestions