Page MenuHomePhabricator

Work required to prepare for puppet 7
Closed, DuplicatePublic

Description

This task is in place so we can track the tasks required to prepare are systems for a move to puppet version 6. It is not envisaged that we will move to puppet6 untill at least bullseye is releases however we should try to get our code base to a level where the transition should be simple

Blockers To an upgrade

puppet forge core type

when puppet version 6 was released a bunch of core resource types where removed from the puppet code base and spun of as external forge modules. We should be aware of any resource we are using which fall into this catagory and either migrate to a different module/type or add the core modules to our code base when we upgrade

  • migrate all cron types to systemd::timer::job
  • if still using nagios migrate to puppetlabs/nagios_core as nagios types are no longer core puppet
  • Check if which other native types need to be migrated

Benefits to explore (some of theses benefits have be realized by updating to the latest 5.5 branch)

Details

SubjectRepoBranchLines +/-
operations/puppetproduction+7 K -0
operations/puppetproduction+104 -100
operations/puppetproduction+75 -91
operations/puppetproduction+0 -6
operations/puppetproduction+23 -5
operations/puppetproduction+0 -8
operations/puppetproduction+11 -3
operations/puppetproduction+1 -10
operations/puppetproduction+5 -1
operations/puppetproduction+1 -1
operations/puppetproduction+18 -7
operations/puppetproduction+107 -83
operations/puppetproduction+14 -13
operations/puppetproduction+15 -96
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+0 -4
operations/puppetproduction+19 -13
operations/puppetproduction+15 -3
operations/puppetproduction+7 -6
operations/puppetproduction+9 -0
operations/puppetproduction+26 -24
operations/puppetproduction+0 -4
operations/puppetproduction+0 -3
operations/puppetproduction+0 -3
operations/puppetproduction+0 -3
operations/puppetproduction+1 -4
operations/puppetproduction+0 -18
operations/puppetproduction+15 -13
operations/puppetproduction+25 -29
operations/puppetproduction+17 -11
operations/puppetproduction+14 -7
operations/puppetproduction+14 -7
operations/puppetproduction+15 -7
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
Duplicatejbond
Resolvedjbond
Resolvedjbond
DeclinedNone
Resolvedjbond
Resolvedjbond
ResolvedDzahn
ResolvedSLyngshede-WMF
ResolvedLadsgroup
ResolvedZabe
ResolvedNone
OpenNone
Resolvedelukey
Resolvedelukey
Resolvedfaidon
OpenNone
Resolvedfaidon
Resolvedherron
Resolvedherron
ResolvedAndrew
Resolvedfgiunchedi
DeclinedNone
OpenNone
Resolvedjcrespo
ResolvedNone
Resolvedelukey
ResolvedNone
ResolvedDzahn
Resolved ema
ResolvedMoritzMuehlenhoff
ResolvedPRODUCTION ERRORCatrope
ResolvedNone
Resolvedelukey
DuplicateNone
ResolvedNone
ResolvedNone
ResolvedDzahn
Resolvedfaidon
DuplicateNone
Resolvedfgiunchedi
DeclinedNone
DeclinedNone
DeclinedNone
ResolvedNone
Resolvedelukey
OpenNone
ResolvedNone
Resolvedfgiunchedi
ResolvedJoe
ResolvedBBlack
Resolvedfgiunchedi
DuplicateNone
Resolvedelukey
Declinedfaidon
ResolvedMoritzMuehlenhoff
Resolvedfgiunchedi
Resolvedcolewhite
ResolvedPRODUCTION ERRORjcrespo
Resolved Gilles
Resolved Gilles
Resolvedfgiunchedi
ResolvedNone
DuplicateNone
Resolved chasemp
Resolvedjijiki
ResolvedJoe
ResolvedAndrew
Resolvedjcrespo
Resolved mmodell
ResolvedNone
Declinedaaron
ResolvedMarostegui
Resolvedjbond
Resolved GTirloni
Resolvedelukey
ResolvedMoritzMuehlenhoff
Resolvedjbond
Resolvedfgiunchedi
ResolvedVolans
ResolvedArielGlenn
Resolvedaaron
ResolvedJMeybohm
Resolved dpifke
DuplicateNone
Resolvedcolewhite
Resolved chasemp
OpenNone
ResolvedAndrew
Resolvedfgiunchedi
ResolvedBTullis
ResolvedJelto
Resolved Vgutierrez
OpenNone
Opennettrom_WMF
OpenNone
OpenNone
OpenNone
OpenNone
ResolvedDzahn
Resolvedjbond
DuplicateNone
ResolvedBUG REPORTjhathaway
Resolvedjbond
Openjhathaway

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 636082 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mirrors: replace cron jobs with systemd timers

https://gerrit.wikimedia.org/r/636082

Change 636104 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] puppetmaster: replace cron to remove old reports with systemd timer

https://gerrit.wikimedia.org/r/636104

Change 636105 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: replace update cron jobs with systemd timers

https://gerrit.wikimedia.org/r/636105

Change 636087 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] dumps: rm profile::dumps::distribution::datasets::cleanup_miscdatasets

https://gerrit.wikimedia.org/r/636087

Change 633857 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] gerrit: replace cron jobs with systemd timers (WIP)

https://gerrit.wikimedia.org/r/633857

Change 636628 merged by Jbond:
[operations/puppet@production] systemd::timer::job: switch monitoring_enabled default to false

https://gerrit.wikimedia.org/r/636628

Change 633857 merged by Dzahn:
[operations/puppet@production] gerrit: replace clear_gerrit_logs cron job with systemd timer

https://gerrit.wikimedia.org/r/633857

Change 637037 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mailman: replace cron with systemd timer

https://gerrit.wikimedia.org/r/637037

Change 637038 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] OTRS: replace cron with systemd timer

https://gerrit.wikimedia.org/r/637038

Change 636087 merged by Bstorm:
[operations/puppet@production] dumps: rm profile::dumps::distribution::datasets::cleanup_miscdatasets

https://gerrit.wikimedia.org/r/636087

Change 636405 abandoned by Jbond:
[operations/puppet@production] cumin: remove cron type

Reason:

https://gerrit.wikimedia.org/r/636405

Change 636411 merged by Jbond:
[operations/puppet@production] remote-backup-mariadb: remove cron type

https://gerrit.wikimedia.org/r/636411

Change 636409 merged by Jbond:
[operations/puppet@production] prometheus_intel_microcode: remove cron type

https://gerrit.wikimedia.org/r/636409

Change 636402 merged by Jbond:
[operations/puppet@production] smart: remove cron type

https://gerrit.wikimedia.org/r/636402

Change 635517 merged by Jbond:
[operations/puppet@production] service-auto-restart: clean up cron

https://gerrit.wikimedia.org/r/635517

Change 636105 merged by Dzahn:
[operations/puppet@production] planet: replace update cron jobs with systemd timers

https://gerrit.wikimedia.org/r/636105

Mentioned in SAL (#wikimedia-operations) [2020-11-16T20:40:06Z] <mutante> planet1002/planet2002 - delete entire crontab of user planet, drop update cronjobs after switching to systemd timers with gerrit:636105 (T265138)

Change 636102 merged by Dzahn:
[operations/puppet@production] cumin: replace check-aliases-cron with a systemd timer

https://gerrit.wikimedia.org/r/636102

Mentioned in SAL (#wikimedia-operations) [2020-11-16T23:28:44Z] <mutante> cumin1001 - sudo systemctl start cumin-check-aliases (to confirm switching cron to timer worked) T265138

Change 641315 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aptrepo: replace cron with systemd timer

https://gerrit.wikimedia.org/r/641315

Change 637038 merged by Dzahn:
[operations/puppet@production] OTRS: replace cron with systemd timer

https://gerrit.wikimedia.org/r/637038

Mentioned in SAL (#wikimedia-operations) [2020-11-17T22:07:15Z] <mutante> otrs1001 - removing otrs-cache-cleanup cron from otrs's crontab - adding same command as systemd timer. gerrit:637038 T265138

Change 641579 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] planet: let systemd timer for each language run at random minute

https://gerrit.wikimedia.org/r/641579

Change 641579 merged by Dzahn:
[operations/puppet@production] planet: let systemd timer for each language run at random minute

https://gerrit.wikimedia.org/r/641579

Change 636082 merged by Jbond:
[operations/puppet@production] mirrors: replace cron jobs with systemd timers

https://gerrit.wikimedia.org/r/636082

Change 636644 merged by Jcrespo:
[operations/puppet@production] mariadb: Cleanup old cron deletions after some time after deploy

https://gerrit.wikimedia.org/r/636644

Change 644905 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mirrors: use "1h" instead of "hourly" for interval

https://gerrit.wikimedia.org/r/644905

Change 644905 merged by Dzahn:
[operations/puppet@production] mirrors: use "1h" instead of "hourly" for interval

https://gerrit.wikimedia.org/r/644905

Change 644920 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mirrors: fix shebang line in update-ubuntu-mirror script

https://gerrit.wikimedia.org/r/644920

Change 644920 merged by Dzahn:
[operations/puppet@production] mirrors: fix shebang line in update-ubuntu-mirror script

https://gerrit.wikimedia.org/r/644920

Change 637037 merged by Dzahn:
[operations/puppet@production] mailman: replace cron with systemd timer and move to profile

https://gerrit.wikimedia.org/r/637037

Change 645455 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikistats: replace all cron jobs with systemd timers

https://gerrit.wikimedia.org/r/645455

Change 645455 merged by Dzahn:
[operations/puppet@production] wikistats: replace all cron jobs with systemd timers

https://gerrit.wikimedia.org/r/645455

Mentioned in SAL (#wikimedia-cloud) [2020-12-07T22:25:18Z] <mutante> - replacing all cron jobs with systemd timers - gerrit:645455 - T265138

Change 636104 merged by Dzahn:
[operations/puppet@production] puppetmaster: replace cron to remove old reports with systemd timer

https://gerrit.wikimedia.org/r/636104

Change 648325 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] puppetmaster: run "remove old reports" job as root

https://gerrit.wikimedia.org/r/648325

Change 648325 merged by Dzahn:
[operations/puppet@production] puppetmaster: run "remove old reports" job as root

https://gerrit.wikimedia.org/r/648325

Change 648327 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] puppetmaster: remove code to remove crons, replaced by timer

https://gerrit.wikimedia.org/r/648327

Change 648328 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] puppetmaster: ensure previously used cron is properly removed

https://gerrit.wikimedia.org/r/648328

Change 648328 merged by Dzahn:
[operations/puppet@production] puppetmaster: ensure previously used cron is properly removed

https://gerrit.wikimedia.org/r/648328

Change 648327 merged by Dzahn:
[operations/puppet@production] puppetmaster: remove code to remove crons, replaced by timer

https://gerrit.wikimedia.org/r/648327

Change 641315 merged by Dzahn:
[operations/puppet@production] aptrepo: replace cron with systemd timer

https://gerrit.wikimedia.org/r/641315

Change 651644 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aptrepo: remove absented cron code, replaced by timer

https://gerrit.wikimedia.org/r/651644

Change 651644 merged by Dzahn:
[operations/puppet@production] aptrepo: remove absented cron code, replaced by timer

https://gerrit.wikimedia.org/r/651644

Change 655172 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] deployment::rsync: replace cron with systemd timer

https://gerrit.wikimedia.org/r/655172

@jbond: Hey, you wrote this as a checkbox

  • migrate all cron types to systemd::timer::job (the cron type is no longer a native puppet type in puppet version6)

I'm trying to migrate some and have been looking in release notes saying something like this. I couldn't find anything implying that in puppet docs. release notes for all of 6.x releases have no mention of cron and puppet 5.x ones have only one mention which is about fixing a bug in solaris.

The puppet six docs also mentions cron as a core type: https://puppet.com/docs/puppet/6.20/cheatsheet_core_types.html#cron

I really can't find anything that says it's deprecated. It's better to do it for sure but I'm not sure if it should be a blocker for the upgrade. Unless I'm missing something very obvious in which I'm sorry :(

@jbond: Hey, you wrote this as a checkbox

  • migrate all cron types to systemd::timer::job (the cron type is no longer a native puppet type in puppet version6)

I really can't find anything that says it's deprecated. It's better to do it for sure but I'm not sure if it should be a blocker for the upgrade. Unless I'm missing something very obvious in which I'm sorry :(

@Ladsgroup thanks for the work your are doing here it is greatly appreciated :). in relation to this point the important link was in the point directly below the one you highlighted

specificly from the link

In Puppet 6.0, we removed some of Puppet’s built-in types and moved them into individual modules.
Supported type modules in puppet-agent

The following types are included in supported modules on the Forge. However, they are also included in the puppet-agent package, so you do not have to install them separately. See each module’s README for detailed information about that type.

Cron, among others, is included in this list. Its also wotth noting that when the mention the puppet-agent package in this context they are referring to the packages provided by puppetlabs not the stock debian ones. When considering the debian ones its unclear weather they will beincluded by default or not. my gut feeling is that they will be provided as seperate packages e.g. puppet-module-puppetlabs-cron but its hard to say conclusively at this point. however it is worth noting that, when we move to puppet 6 , we can if needed install the puppetlabs-cron module (which is assumedly this one) so this is not a blocker. however in general it seems healthy to move towards using sytemd::timers

FYI i have tried to clarify things in the task description

Thanks. I don't know much about puppet so I'm probably wrong but it's just it's mentioned in list of core types: https://puppet.com/docs/puppet/6.21/cheatsheet_core_types.html#cron but you're probably right, we might need a separate package for it. Can we test it somewhere?

@Ladsgroup This could well be to do with how puppetlabs defines core type however it has definitely been removed from the puppet git repo you can see that in this commit which also references PUP-8836 (although there is not much information on the later). That said i have created a documentation bug to seek clarification.

@Ladsgroup This could well be to do with how puppetlabs defines core type however it has definitely been removed from the puppet git repo you can see that in this commit which also references PUP-8836 (although there is not much information on the later). That said i have created a documentation bug to seek clarification.

Still no update on the ticked however it looks like debian will package resources like cron_core as separate packages

All hiera() lookups removed across the repo.

All hiera() lookups removed across the repo.

Amazing thanks for all the effort 💃🏻 🎉

Change 655172 merged by Dzahn:
[operations/puppet@production] deployment::rsync: replace cron with systemd timer

https://gerrit.wikimedia.org/r/655172

Change 666756 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] deployment::rsync: remove absented cron code

https://gerrit.wikimedia.org/r/666756

Change 666756 merged by Dzahn:
[operations/puppet@production] deployment::rsync: remove absented cron code

https://gerrit.wikimedia.org/r/666756

@Ladsgroup This could well be to do with how puppetlabs defines core type however it has definitely been removed from the puppet git repo you can see that in this commit which also references PUP-8836 (although there is not much information on the later). That said i have created a documentation bug to seek clarification.

Still no update on the ticked however it looks like debian will package resources like cron_core as separate packages

just a quick update in puppet has renamed the types. now we have:

  • built in types which are all the types included in the puppet source tree.
  • core types which are types that puppet labs packages with the puppet-agent and labeled puppetlabs-core-$foo on puppet forge
  • third party modules

will move to puppet6 untill at least bullseye

Its worth noting that bullseye currently has puppet 5.5.19 (with sid on 5.5.21) its not clear if bullseye will ship with puppet6 and im unclear how we would progress if they don't. one options could be to hire external expertise to do the debian work on our behalf

It looks like Bullseye ended up releasing with puppet 5.5.22 (package), with sid currently on the same version; experimental has 6.16.0.

jbond renamed this task from OKR: Work required to prepare for puppet 6 to Work required to prepare for puppet 6.Nov 3 2021, 11:38 AM
jbond added a parent task: T294906: Puppet Improvements.

Change 763311 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] ini(), php_ini(): convert to modern Ruby function API

https://gerrit.wikimedia.org/r/763311

Change 763311 merged by JHathaway:

[operations/puppet@production] ini(), php_ini(): convert to modern Ruby function API

https://gerrit.wikimedia.org/r/763311

Change 763611 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] Add nagios_core & mailalias_core modules

https://gerrit.wikimedia.org/r/763611

jbond renamed this task from Work required to prepare for puppet 6 to Work required to prepare for puppet 7.Jul 3 2023, 3:01 PM

Change 763611 abandoned by JHathaway:

[operations/puppet@production] Add nagios_core & mailalias_core modules

Reason:

done as part of the puppet 7 work

https://gerrit.wikimedia.org/r/763611