Page MenuHomePhabricator
Create Task
Maniphest T254480

Shell/Python/other scripts should not be generated by ERB files; dynamic parts should be a simple ERB config file
Open, MediumPublic
Actions

Description

We have many scripts that are defined as ERB templates.

We often make simple errors when editing them, as there's no tooling for dealing with such a format -- and in fact it's quite hard to write tooling for such, given the format of Puppet catalogs (which elides which template file produced a given generated file).

Instead, let's forbid scripts being generated from templates. (And also set up shellcheck for .sh files.)

Audit

The following one liner helps auditing said candidates: for file in $(find . -not -path "./.bundle/*" -name "*.erb"); do if head -1 "$file" | grep -q '^#!\/'; then wc -l "$file"; fi; done | sort -n

So for example in puppet as of 2f2ae5d303df2 we have:

3 ./modules/community_civicrm/templates/civicrm/community_civicrm_path.sh.erb
3 ./modules/dumps/templates/rsync/run_rsync.erb
3 ./modules/mariadb/templates/mysql-section.sh.erb
4 ./modules/openstack/templates/util/envscript.sh.erb
5 ./modules/gitlab/templates/gitlab-backup-config.sh.erb
7 ./modules/conftool/templates/conftool-merge.erb
7 ./modules/fastnetmon/templates/fastnetmon_notify.sh.erb
8 ./modules/conftool/templates/safe-depool.erb
8 ./modules/conftool/templates/safe-pool.erb
8 ./modules/profile/templates/ncredir/ncredirlog.sh.erb
8 ./modules/profile/templates/puppetserver/git/private/hooks/post-receive.erb
8 ./modules/profile/templates/quarry/trove/dbdump.sh.erb
8 ./modules/profile/templates/quarry/trove/dbdumpcleanup.sh.erb
8 ./modules/profile/templates/trafficserver/atslog.sh.erb
9 ./modules/profile/templates/prometheus/node-directory-size.erb
9 ./modules/service/templates/check-service.erb
9 ./modules/vagrant/templates/mwvagrant.erb
9 ./modules/vagrant/templates/start-mwvagrant.sh.erb
10 ./modules/role/templates/mariadb/backups/dumps-otrs.sh.erb
11 ./modules/base/templates/initramfs_sleep.erb
11 ./modules/package_builder/templates/D04php.erb
11 ./modules/snapshot/templates/set_dump_dirs.sh.erb
12 ./modules/docker/templates/images/build-base-images.erb
12 ./modules/phabricator/templates/vcs/phabricator-ssh-hook.sh.erb
12 ./modules/profile/templates/mediawiki/restart-php-fpm-all.sh.erb
12 ./modules/vagrant/templates/labs-vagrant.erb
13 ./modules/conftool/templates/safe-restart.erb
13 ./modules/elasticsearch/templates/set-cross-cluster-seeds.sh.erb
13 ./modules/monitoring/templates/check_dir-not-bad-owner.erb
13 ./modules/ncmonitor/templates/ncmonitor-update-psl.sh.erb
13 ./modules/profile/templates/presto/presto_client_ssl_kerberos.erb
13 ./modules/service/templates/node/tail-log.erb
13 ./modules/snapshot/templates/dumps/nfs_testing/test_outputdir_paths.sh.erb
14 ./modules/package_builder/templates/D02backports.erb
14 ./modules/package_builder/templates/D04component.erb
14 ./modules/package_builder/templates/D05localsources.erb
14 ./modules/profile/templates/analytics/refinery/job/refinery-drop-mediawiki-xmldumps-pages_meta_history.sh.erb
14 ./modules/profile/templates/kerberos/replicate_krb_database.erb
15 ./modules/service/templates/node/apply-config.sh.erb
16 ./modules/opensearch/templates/set-cross-cluster-seeds.sh.erb
16 ./modules/package_builder/templates/D01security.erb
16 ./modules/profile/templates/analytics/refinery/job/refinery-sqoop-whole-mediawiki.sh.erb
16 ./modules/profile/templates/kubernetes/kube-conf.sh.erb
17 ./modules/profile/templates/analytics/airflow/airflow.sh.erb
19 ./modules/conftool/templates/initialize.sh.erb
20 ./modules/profile/templates/analytics/airflow/airflow-clean-log-dirs.erb
20 ./modules/profile/templates/analytics/refinery/job/java_job.sh.erb
20 ./modules/profile/templates/analytics/refinery/job/refinery-import-mediawiki-dumps.sh.erb
20 ./modules/profile/templates/icinga/inactive.motd.erb
21 ./modules/profile/templates/analytics/refinery/job/refinery-sqoop-mediawiki-production-history.sh.erb
21 ./modules/profile/templates/analytics/refinery/job/refinery-sqoop-mediawiki-production-not-history.sh.erb
22 ./modules/package_builder/templates/D01apt.wikimedia.org.erb
22 ./modules/profile/templates/analytics/refinery/job/refinery-sqoop-mediawiki-not-history.sh.erb
22 ./modules/profile/templates/analytics/refinery/job/spark_job.sh.erb
23 ./modules/airflow/templates/airflow.sh.erb
23 ./modules/openstack/templates/initscripts/nova-fullstack.erb
23 ./modules/profile/templates/analytics/refinery/job/refinery-import-wikibase-dumps.sh.erb
23 ./modules/profile/templates/analytics/refinery/job/refinery-sqoop-mediawiki-history.sh.erb
23 ./modules/profile/templates/analytics/refinery/job/refinery-sqoop-mediawiki-private.sh.erb
23 ./modules/profile/templates/analytics/refinery/job/refinery-sqoop-mediawiki-production-daily.sh.erb
23 ./modules/profile/templates/debmonitor/server/run_django_command.sh.erb
23 ./modules/profile/templates/mediawiki/maintenance/inactive.motd.erb
23 ./modules/role/templates/deployment/inactive.motd.erb
24 ./modules/profile/templates/kerberos/kadminserver/inactive.motd.erb
24 ./modules/statistics/templates/published-sync.sh.erb
26 ./modules/profile/templates/analytics/refinery/job/refinery-sqoop-wikifunctions-production.sh.erb
26 ./modules/role/templates/releases/rsync_source_warning.motd.erb
28 ./modules/profile/templates/dns/auth/check_state.erb
28 ./modules/rsync/templates/quickdatacopy-ssl-wrapper.erb
32 ./modules/aptrepo/templates/ztp-juniper.sh.erb
32 ./modules/profile/templates/puppetserver/gitprivate/postcommit.erb
33 ./modules/monitoring/templates/check_git-needs-merge.erb
33 ./modules/profile/templates/hadoop/net-topology.py.erb
36 ./modules/puppetmaster/templates/git-master-postcommit.erb
39 ./modules/git/templates/replicated_local_postcommit.sh.erb
39 ./modules/profile/templates/dns/auth/check_authdns_update_run.erb
40 ./modules/profile/templates/analytics/refinery/job/refinery-salt-rotate.erb
41 ./modules/acme_chief/templates/create_acme_le_account.py.erb
42 ./modules/osm/templates/import_waterlines.erb
48 ./modules/openstack/templates/bootstrap/glance/glance_seed.sh.erb
49 ./modules/postgresql/templates/prometheus/postgresql_replication_lag.sh.erb
50 ./modules/phabricator/templates/phab_deploy_finalize.sh.erb
60 ./modules/httpbb/templates/deploy_apache_change.sh.erb
61 ./modules/backup/templates/mysql-predump.erb
65 ./modules/profile/templates/toolforge/harbor/prepare.erb
69 ./modules/bacula/templates/bpipe-mysql-db.erb
69 ./modules/haproxy/templates/check_haproxy.erb
73 ./modules/profile/templates/statistics/explorer/ml/model_upload.sh.erb
95 ./modules/spamassassin/templates/sa-update-cron.erb
110 ./modules/bigtop/templates/spark/spark-env.sh.erb
138 ./modules/labstore/templates/tc-setup.sh.erb
148 ./modules/beta/templates/wmf-beta-autoupdate.py.erb
157 ./modules/service/templates/deployment_script.sh.erb
179 ./modules/profile/templates/hadoop/spark3/spark3-env.sh.erb
224 ./modules/openstack/templates/bobcat/barbican/barbican-api.erb
224 ./modules/openstack/templates/caracal/barbican/barbican-api.erb
261 ./modules/openstack/templates/caracal/keystone/keystone-admin-service.erb
261 ./modules/openstack/templates/caracal/keystone/keystone-public-service.erb
262 ./modules/openstack/templates/bobcat/heat/heat-api-cfn.erb
262 ./modules/openstack/templates/bobcat/heat/heat-api.erb
262 ./modules/openstack/templates/caracal/heat/heat-api-cfn.erb
262 ./modules/openstack/templates/caracal/heat/heat-api.erb
265 ./modules/openstack/templates/bobcat/keystone/keystone-admin-service.erb
265 ./modules/openstack/templates/bobcat/keystone/keystone-public-service.erb
266 ./modules/openstack/templates/bobcat/magnum/magnum-api.erb
266 ./modules/openstack/templates/caracal/magnum/magnum-api.erb
270 ./modules/openstack/templates/bobcat/placement/placement-api.erb
270 ./modules/openstack/templates/caracal/placement/placement-api.erb

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
P:debmonitor::server remove unused templateoperations/puppetproduction+0 -23
C:bigtop::hadoop switch to new topology script.operations/puppetproduction+12 -7
C:bigtop::hadoop move net-topology.py to files.operations/puppetproduction+73 -0
C:bigtop::hadoop Fix script pathoperations/puppetproduction+1 -1
C:bigtop::hadoop ensure net-topology script is installed.operations/puppetproduction+6 -5
C:beta:autoupdate Move wmf-beta-autoupdate to files.operations/puppetproduction+222 -0
P:hive::client move beeline script to files.operations/puppetproduction+76 -19
alytics::refinery::job: build simple script inline in puppetoperations/puppetproduction+10 -3
phabricator: don't create chk_phuser shell script from erboperations/puppetproduction+18 -20
rsync: fix quickdatacopy sync scriptoperations/puppetproduction+3 -2
trafficserver::instance: move single line scripts inlineoperations/puppetproduction+1 -6
varnish::logging: move default definitions inlineoperations/puppetproduction+1 -4
CI: add some shell scripts to test the new shellcheck CI checkoperations/puppetproduction+0 -2
puppet-merge: split dynamic values out of puppet-merge scriptoperations/puppetproduction+36 -19
CI: add CI to check shell scriptsoperations/puppetproduction+43 -2
dumps: fix shellcheck issuesoperations/puppetproduction+55 -50
Example: build script in line in puppetoperations/puppetproduction+7 -13
rsync: move oneline script inlineoperations/puppetproduction+13 -5
aptrepo: build single line shell script inlineoperations/puppetproduction+5 -4
profile::icinga: move single line scripts in lineoperations/puppetproduction+12 -11
phabricator: move template to file as no dynamic valuesoperations/puppetproduction+1 -1
phabricator: move template to file as no dynamic valuesoperations/puppetproduction+1 -1
puppetmaster::gitclone: build single line script inlineoperations/puppetproduction+5 -3
icinga: convert sync_icinga_state.sh.erb to file with configoperations/puppetproduction+16 -4
httpbb: convert an .erb.sh script to inline contentoperations/puppetproduction+3 -5
jjb: Upgrade puppet jobs to new shellcheck versionintegration/configmaster+2 -2
docker: use more recent shellcheckintegration/configmaster+10 -0
puppet-merge: fix shellcheck issuesoperations/puppetproduction+37 -37
phabricator: convert 2 scripts created from erb to files with config filesoperations/puppetproduction+61 -39
prometheus: fix shellcheck issues in prometheus-local-crontabs.shoperations/puppetproduction+1 -1
osm: fix shellcheck issuesoperations/puppetproduction+6 -3
jjb: update puppet.git container for shellcheckintegration/configmaster+2 -2
operations-puppet: add shellcheck to docker imageintegration/configmaster+8 -1
statistics: fix shelcheck issues in hardsyncoperations/puppetproduction+22 -17
confluent: fix shellcheck issuesoperations/puppetproduction+24 -23
misc: fix minor shellcheck issues in a few scriptsoperations/puppetproduction+3 -2
mailmain: fix shellcheck issues remove_from_private.shoperations/puppetproduction+14 -14
planet: fix shellcheck issues in check_httpsoperations/puppetproduction+7 -7
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ArielGlenn added a comment.Jun 5 2020, 8:22 PM

I agree. Example: the first attempt at ./modules/dumps/templates/web/fetches/analytics/job/rsync_script.sh.erb was here https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594773/2/modules/dumps/manifests/web/fetches/analytics/job.pp so you can see why erb was the natural approach in the end. Another file in there (modules/snapshot/templates/set_dump_dirs.sh.erb) literally just sets some vars to be picked up by the 'real' shell scripts that use it; I favour allowing scripts like these and strongly encouraging folks with long scripts as erb files to refactor them accordingly.

jbond added a comment.Jun 5 2020, 8:38 PM
In T254480#6197931, @Volans wrote:

@CDanis yeah, sorry I noticed that after replying.
Totally agree with this approach, would be nice to have the CI check but we have a bunch of few liners that would become just more complex and probably not gaining a lot, see for example the first part of this list:

for scripts that are just one line i think it reasonable to just build the content in puppet, this is a bit of a cheat but for simple one liners i think its a fine compromise. I took a quick look through the files other then the oneliners, some of them don't actually need to be templates as there is no erb and a few others look like they are already config files so we could possibly just remove the shebang . however there are of course some that may be a bit awkward to fit into this model. Perhaps we should just whitelist them. It would at least mean going forward we don't have more violations added.

Ill have a closer look next week and see if how much we can reduce that list with quick wins.

gerritbot added a comment.Jun 5 2020, 9:02 PM

Change 602771 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] Example: build script in line in puppet

https://gerrit.wikimedia.org/r/602771

jbond added a comment.EditedJun 5 2020, 9:05 PM
In T254480#6197961, @ArielGlenn wrote:

I agree. Example: the first attempt at ./modules/dumps/templates/web/fetches/analytics/job/rsync_script.sh.erb was here https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594773/2/modules/dumps/manifests/web/fetches/analytics/job.pp so you can see why erb was the natural approach in the end. Another file in there (modules/snapshot/templates/set_dump_dirs.sh.erb) literally just sets some vars to be picked up by the 'real' shell scripts that use it; I favour allowing scripts like these and strongly encouraging folks with long scripts as erb files to refactor them accordingly.

sorry i missed this before i responded. I took a look at this script and Created a CR to show how it may look building the script in puppet. like i said its a bit of a cheat but tbh for simple things like this i prefer to see them inline in the puppet manifest instead of having to open a new file. that said i suspect many may disagree

ArielGlenn added a comment.Jun 5 2020, 9:17 PM

@jbond I added the author of the one patch as a reviewer for their thoughts, since likely analytics folks will wind up maintaining that particular bit.

gerritbot added a comment.Jun 6 2020, 1:36 AM

Change 602647 merged by Ryan Kemper:
[operations/puppet@production] osm: fix shellcheck issues

https://gerrit.wikimedia.org/r/602647

gerritbot added a comment.Jun 8 2020, 6:45 AM

Change 603175 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: use more recent shellcheck

https://gerrit.wikimedia.org/r/603175

hashar added a comment.Jun 8 2020, 6:52 AM

The CI container is build using Buster which comes with shellcheck 0.5.0 from May 2018. I have proposed a change to borrow the package from buster-backports and get 0.7.1 instead. https://gerrit.wikimedia.org/r/#/c/integration/config/+/603175

Notably because it adds new checks and it also has two interesting options which we might want to use:

  • --severity for filtering by minimum severity. So that potentially we can make the task fail on error but just report on warnings/info etc.
  • --wiki-link-count for showing wiki links . Grants us detailed information
gerritbot added a comment.Jun 8 2020, 8:32 AM

Change 603364 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: convert 2 scripts created from erb to files with config files

https://gerrit.wikimedia.org/r/603364

jbond added a comment.Jun 8 2020, 9:45 AM
In T254480#6198075, @ArielGlenn wrote:

@jbond I added the author of the one patch as a reviewer for their thoughts, since likely analytics folks will wind up maintaining that particular bit.

great thanks

gerritbot added a comment.Jun 8 2020, 10:12 AM

Change 602649 merged by Jbond:
[operations/puppet@production] prometheus: fix shellcheck issues in prometheus-local-crontabs.sh

https://gerrit.wikimedia.org/r/602649

gerritbot added a comment.Jun 8 2020, 12:01 PM

Change 603364 merged by Dzahn:
[operations/puppet@production] phabricator: convert 2 scripts created from erb to files with config files

https://gerrit.wikimedia.org/r/603364

gerritbot added a comment.Jun 8 2020, 2:01 PM

Change 602732 merged by Jbond:
[operations/puppet@production] puppet-merge: split dynamic values out of puppet-merge script

https://gerrit.wikimedia.org/r/602732

gerritbot added a comment.Jun 8 2020, 2:01 PM

Change 602738 merged by Jbond:
[operations/puppet@production] puppet-merge: fix shellcheck issues

https://gerrit.wikimedia.org/r/602738

gerritbot added a comment.Jun 8 2020, 2:37 PM

Change 603490 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] httpbb: convert an .erb.sh script to inline content

https://gerrit.wikimedia.org/r/603490

gerritbot added a comment.Jun 8 2020, 2:46 PM

Change 603492 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: convert sync_icinga_state.sh.erb to file with config

https://gerrit.wikimedia.org/r/603492

gerritbot added a comment.Jun 8 2020, 5:50 PM

Change 603175 merged by jenkins-bot:
[integration/config@master] docker: use more recent shellcheck

https://gerrit.wikimedia.org/r/603175

Stashbot added a comment.Jun 8 2020, 5:55 PM

Mentioned in SAL (#wikimedia-releng) [2020-06-08T17:55:39Z] <James_F> Docker: Publishing operations-puppet image, now with upgraded shellcheck T254480

gerritbot added a comment.Jun 8 2020, 6:14 PM

Change 603563 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] jjb: Upgrade puppet jobs to new shellcheck version

https://gerrit.wikimedia.org/r/603563

gerritbot added a comment.Jun 8 2020, 6:20 PM

Change 603563 merged by jenkins-bot:
[integration/config@master] jjb: Upgrade puppet jobs to new shellcheck version

https://gerrit.wikimedia.org/r/603563

gerritbot added a comment.Jun 9 2020, 7:16 AM

Change 603490 merged by Dzahn:
[operations/puppet@production] httpbb: convert an .erb.sh script to inline content

https://gerrit.wikimedia.org/r/603490

gerritbot added a comment.Jun 11 2020, 1:30 PM

Change 603492 merged by Dzahn:
[operations/puppet@production] icinga: convert sync_icinga_state.sh.erb to file with config

https://gerrit.wikimedia.org/r/603492

gerritbot added a comment.Jun 12 2020, 3:31 PM

Change 605261 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] alytics::refinery::job: build simple script inline in puppet

https://gerrit.wikimedia.org/r/605261

gerritbot added a comment.Jun 12 2020, 3:39 PM

Change 605262 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster::gitclone: build single line script inline

https://gerrit.wikimedia.org/r/605262

gerritbot added a comment.Jun 12 2020, 3:47 PM

Change 605262 merged by Jbond:
[operations/puppet@production] puppetmaster::gitclone: build single line script inline

https://gerrit.wikimedia.org/r/605262

gerritbot added a comment.Jun 12 2020, 4:01 PM

Change 605267 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] aptrepo: build single line shell script inline

https://gerrit.wikimedia.org/r/605267

gerritbot added a comment.Jun 12 2020, 4:12 PM

Change 605271 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] profile::icinga: move single line scripts in line

https://gerrit.wikimedia.org/r/605271

gerritbot added a comment.Jun 12 2020, 4:20 PM

Change 605272 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] varnish::logging: move default definitions inline

https://gerrit.wikimedia.org/r/605272

gerritbot added a comment.Jun 12 2020, 4:24 PM

Change 605274 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] phabricator: move template to file as no dynamic values

https://gerrit.wikimedia.org/r/605274

gerritbot added a comment.Jun 12 2020, 4:26 PM

Change 605274 merged by Jbond:
[operations/puppet@production] phabricator: move template to file as no dynamic values

https://gerrit.wikimedia.org/r/605274

gerritbot added a comment.Jun 12 2020, 4:34 PM

Change 605275 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] rsync: move oneline script inline

https://gerrit.wikimedia.org/r/605275

gerritbot added a comment.Jun 12 2020, 4:37 PM

Change 605276 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] phabricator: move template to file as no dynamic values

https://gerrit.wikimedia.org/r/605276

gerritbot added a comment.Jun 12 2020, 4:41 PM

Change 605279 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] trafficserver::instance: move single line scripts inline

https://gerrit.wikimedia.org/r/605279

gerritbot added a comment.Jun 12 2020, 4:41 PM

Change 605276 merged by Jbond:
[operations/puppet@production] phabricator: move template to file as no dynamic values

https://gerrit.wikimedia.org/r/605276

gerritbot added a comment.Jun 15 2020, 9:00 AM

Change 605271 merged by Jbond:
[operations/puppet@production] profile::icinga: move single line scripts in line

https://gerrit.wikimedia.org/r/605271

gerritbot added a comment.Jun 15 2020, 9:01 AM

Change 605267 merged by Jbond:
[operations/puppet@production] aptrepo: build single line shell script inline

https://gerrit.wikimedia.org/r/605267

gerritbot added a comment.Jun 16 2020, 8:40 AM

Change 605275 merged by Jbond:
[operations/puppet@production] rsync: move oneline script inline

https://gerrit.wikimedia.org/r/605275

gerritbot added a comment.Jun 16 2020, 8:48 AM

Change 602771 merged by Jbond:
[operations/puppet@production] Example: build script in line in puppet

https://gerrit.wikimedia.org/r/602771

gerritbot added a comment.Jun 26 2020, 10:35 AM

Change 602645 merged by ArielGlenn:
[operations/puppet@production] dumps: fix shellcheck issues

https://gerrit.wikimedia.org/r/602645

gerritbot added a comment.Jun 29 2020, 9:10 AM

Change 602693 merged by Jbond:
[operations/puppet@production] CI: add CI to check shell scripts

https://gerrit.wikimedia.org/r/c/operations/puppet/ /602693

gerritbot added a comment.Jul 7 2020, 10:29 AM

Change 602694 abandoned by Jbond:
[operations/puppet@production] CI: add some shell scripts to test the new shellcheck CI check

Reason:

https://gerrit.wikimedia.org/r/602694

gerritbot added a comment.Jul 21 2020, 2:22 PM

Change 605272 merged by Jbond:
[operations/puppet@production] varnish::logging: move default definitions inline

https://gerrit.wikimedia.org/r/605272

gerritbot added a comment.Jul 21 2020, 2:23 PM

Change 605279 merged by Jbond:
[operations/puppet@production] trafficserver::instance: move single line scripts inline

https://gerrit.wikimedia.org/r/605279

gerritbot added a comment.Jul 22 2020, 8:31 AM

Change 615413 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] rsync: fix quickdatacopy sync script

https://gerrit.wikimedia.org/r/615413

gerritbot added a comment.Jul 22 2020, 8:44 AM

Change 615413 merged by Filippo Giunchedi:
[operations/puppet@production] rsync: fix quickdatacopy sync script

https://gerrit.wikimedia.org/r/615413

hashar mentioned this in T259586: Add CI to mediawiki/tools/train-dev.Aug 4 2020, 7:25 AM
Dzahn mentioned this in T200984: Stop introducing new code expanded from erb templates.Oct 3 2020, 12:07 AM
gerritbot added a comment.Oct 3 2020, 1:15 AM

Change 630257 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: don't create chk_phuser shell script from erb (WIP)

https://gerrit.wikimedia.org/r/630257

gerritbot added a comment.Oct 3 2020, 1:42 AM

Change 630257 merged by Dzahn:
[operations/puppet@production] phabricator: don't create chk_phuser shell script from erb

https://gerrit.wikimedia.org/r/630257

gerritbot added a comment.Nov 20 2020, 1:18 PM

Change 605261 merged by Jbond:
[operations/puppet@production] alytics::refinery::job: build simple script inline in puppet

https://gerrit.wikimedia.org/r/605261

• crusnov subscribed.Feb 5 2021, 4:44 PM
• crusnov added a comment.Feb 5 2021, 4:46 PM

I'm scoping part of the broader python3 porting to get rid of .py.erb files also since they break all the tools I'm using to semi-automate parts of the process. :)

jbond merged a task: T277892: Convert .py.erb files to files with configurations.Mar 22 2021, 11:30 AM
jbond updated the task description. (Show Details)
jbond added subscribers: RhinosF1, Ottomata, elukey, MoritzMuehlenhoff.
• crusnov added a parent task: T247364: Forward port Python2 files to Python3 in Puppet Repository.Mar 31 2021, 3:09 PM
thcipriani edited projects, added Release-Engineering-Team (thcipriani-workboard-fiddling); removed Release-Engineering-Team (CI & Testing services).Apr 20 2021, 1:03 AM
thcipriani moved this task from thcipriani-workboard-fiddling to Seen (ARCHIVE) on the Release-Engineering-Team board.Apr 20 2021, 1:05 AM
thcipriani edited projects, added Release-Engineering-Team; removed Release-Engineering-Team (thcipriani-workboard-fiddling).
thcipriani edited projects, added Release-Engineering-Team (Seen); removed Release-Engineering-Team.Apr 20 2021, 3:23 PM
jbond moved this task from Unsorted 💣 to Friday tasks on the User-jbond board.May 27 2021, 4:50 PM
hashar removed projects: Release-Engineering-Team (Seen), Continuous-Integration-Config, Patch-For-Review.Jun 4 2021, 10:13 AM

Dropping releng/CI, doesn't seem we have anything to do to complete the resolution of this task. It seems decoupling code and configuration can be achieved entirely in the Puppet repo without further tweaking in CI config/jobs.

Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 8:59 PM
jbond added a parent task: T294906: Puppet Improvements.Nov 3 2021, 11:50 AM
Volans mentioned this in T211750: Introduce Python code formatters usage.Apr 14 2022, 10:40 PM
hashar unsubscribed.Apr 20 2022, 10:22 AM
gerritbot added a comment.Jun 12 2023, 8:23 AM

Change 929156 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:hive::client move beeline script to files.

https://gerrit.wikimedia.org/r/929156

gerritbot added a project: Patch-For-Review.Jun 12 2023, 8:23 AM
gerritbot added a comment.Jun 13 2023, 9:20 AM

Change 929643 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] C:bigtop::hadoop move net-topology.py to files.

https://gerrit.wikimedia.org/r/929643

gerritbot added a comment.Jun 20 2023, 7:00 AM

Change 931499 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] C:beta:autoupdate Move wmf-beta-autoupdate to files.

https://gerrit.wikimedia.org/r/931499

SLyngshede-WMF subscribed.Jun 20 2023, 7:34 AM

I've added patches for the remaining three Python scripts, one of which may just need to be deleted. All have been updated to have a configuration file, and to Python 3.

SLyngshede-WMF claimed this task.Jun 20 2023, 7:34 AM
SLyngshede-WMF updated the task description. (Show Details)Jul 4 2023, 10:23 AM
gerritbot added a comment.Jul 4 2023, 10:24 AM

Change 929156 merged by Slyngshede:

[operations/puppet@production] P:hive::client move beeline script to files.

https://gerrit.wikimedia.org/r/929156

gerritbot added a comment.Jul 4 2023, 10:24 AM

Change 931499 abandoned by Slyngshede:

[operations/puppet@production] C:beta:autoupdate Move wmf-beta-autoupdate to files.

Reason:

https://gerrit.wikimedia.org/r/931499

Dzahn unsubscribed.Jul 5 2023, 5:40 PM
jbond edited projects, added Puppet CI; removed User-jbond, Puppet.Jul 17 2023, 3:55 PM
gerritbot added a comment.Aug 24 2023, 7:51 AM

Change 929643 merged by Slyngshede:

[operations/puppet@production] C:bigtop::hadoop move net-topology.py to files.

https://gerrit.wikimedia.org/r/929643

SLyngshede-WMF added a comment.Aug 24 2023, 8:00 AM

The new version of the script have been deployed, but not yet configured in Hadoop.

gerritbot added a comment.Aug 24 2023, 8:17 AM

Change 952112 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] C:bigtop::hadoop ensure net-topology script is installed.

https://gerrit.wikimedia.org/r/952112

gerritbot added a comment.Aug 24 2023, 8:26 AM

Change 952112 merged by Slyngshede:

[operations/puppet@production] C:bigtop::hadoop ensure net-topology script is installed.

https://gerrit.wikimedia.org/r/952112

gerritbot added a comment.Aug 24 2023, 9:03 AM

Change 952125 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] C:bigtop::hadoop Fix script path

https://gerrit.wikimedia.org/r/952125

gerritbot added a comment.Aug 24 2023, 9:07 AM

Change 952125 merged by Slyngshede:

[operations/puppet@production] C:bigtop::hadoop Fix script path

https://gerrit.wikimedia.org/r/952125

gerritbot added a comment.Jan 9 2024, 9:47 AM

Change 954911 had a related patch set uploaded (by Stevemunene; author: Slyngshede):

[operations/puppet@production] C:bigtop::hadoop switch to new topology script.

https://gerrit.wikimedia.org/r/954911

gerritbot added a comment.Jan 9 2024, 11:40 AM

Change 954911 merged by Stevemunene:

[operations/puppet@production] C:bigtop::hadoop switch to new topology script.

https://gerrit.wikimedia.org/r/954911

Stashbot added a comment.Jan 9 2024, 12:57 PM

Mentioned in SAL (#wikimedia-analytics) [2024-01-09T12:57:42Z] <stevemunene> roll restart analytics hadoop masters to pickup new net_topology script and new JRE T254480

SLyngshede-WMF closed this task as Resolved.Jan 23 2024, 2:33 PM
fgiunchedi reopened this task as Open.Mar 11 2025, 11:37 AM

Reopening as we are not done here (and I wish we were!)

SLyngshede-WMF removed SLyngshede-WMF as the assignee of this task.Mar 11 2025, 11:38 AM
fgiunchedi updated the task description. (Show Details)Mar 11 2025, 11:50 AM
gerritbot added a comment.Mar 12 2025, 10:02 AM

Change #1126937 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:debmonitor::server remove unused template

https://gerrit.wikimedia.org/r/1126937

gerritbot added a comment.Mar 12 2025, 12:41 PM

Change #1126937 merged by Slyngshede:

[operations/puppet@production] P:debmonitor::server remove unused template

https://gerrit.wikimedia.org/r/1126937

BCornwall subscribed.Aug 26 2025, 5:29 PM

If this is an "official" policy, should https://wikitech.wikimedia.org/wiki/Puppet/Coding_and_style_guidelines be updated to reflect this?

BCornwall unsubscribed.Aug 26 2025, 5:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits