puppetserver-deploy-code emits scary looking error messages during a `git rebase` operation
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	bd808
	Jun 25 2025, 8:45 PM

Description

We run sudo /usr/local/bin/puppetserver-deploy-code via post-checkout, post-commit, and post-merge git hooks in the /srv/git/operations/puppet directory. This script takes care of atomically updating the puppetserver's view of ops/puppet.git when we update the local checkout. The script includes a check to decide if work should be done or not, and emits an ERROR: ... log message when it decides not not continue:

if [[ "$current_branch" != 'production' ]]; then
        printf 'ERROR: Current branch in /srv/git/operations/puppet is "%s", should be "production"\n' "$current_branch" 1>&2
        printf 'ERROR: Exiting rather than deploying something surprising\n' 1>&2
        exit 1
fi

This log message looks sort of scary when the git operation being performed is a rebase of the local cherry-picks on top of the latest upstream HEAD:

gitpuppet@deployment-puppetserver-1:/srv/git/operations/puppet$ git rebase --interactive origin/production
ERROR: Current branch in /srv/git/operations/puppet is "", should be "production"
ERROR: Exiting rather than deploying something surprising
ERROR: Current branch in /srv/git/operations/puppet is "", should be "production"
ERROR: Exiting rather than deploying something surprising
ERROR: Current branch in /srv/git/operations/puppet is "", should be "production"
ERROR: Exiting rather than deploying something surprising
ERROR: Current branch in /srv/git/operations/puppet is "", should be "production"
ERROR: Exiting rather than deploying something surprising
ERROR: Current branch in /srv/git/operations/puppet is "", should be "production"
ERROR: Exiting rather than deploying something surprising
ERROR: Current branch in /srv/git/operations/puppet is "", should be "production"
Successfully rebased and updated refs/heads/production.

Changing the log line label from ERROR: to INFO: might make things look less scary while still letting folks know what is happening. Alternately we could try to figure out within the script itself if the call is happening during a rebase. Or we could leave things as is knowing that this explanation of what is happening is now available to those who search Phabricator for the error message.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	puppetserver: Generalize git-rebase fix to work for labs/private	operations/puppet	production	+14 -10
	puppetserver: check for rebase in puppetserver-deploy-code	operations/puppet	production	+12 -4

Customize query in gerrit

Event Timeline

bd808 created this task.Jun 25 2025, 8:45 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 25 2025, 8:45 PM

bd808 moved this task from Epics / Tracking to Backlog on the Beta-Cluster-Infrastructure board.Jun 25 2025, 8:46 PM

thcipriani awarded a token.Jun 25 2025, 8:49 PM

Change #1163883 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/puppet@production] puppetserver: check for rebase in puppetserver-deploy-code

https://gerrit.wikimedia.org/r/1163883

gerritbot added a project: Patch-For-Review.Jun 25 2025, 9:13 PM

Mentioned in SAL (#wikimedia-releng) [2025-06-25T21:16:46Z] <bd808> Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1163883/1 to deployment-puppetserver-1 (T397877)

bd808@deployment-puppetserver-1:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(2a0dc7fe3d) git-sync-upstream - puppetserver: check for rebase in puppetserver-deploy-code'
Notice: /Stage[main]/Puppetserver/File[/usr/local/bin/puppetserver-deploy-code]/content:
--- /usr/local/bin/puppetserver-deploy-code     2024-08-13 13:37:12.537377757 +0000
+++ /tmp/puppet-file20250625-1722938-fawube     2025-06-25 21:17:33.604680379 +0000
@@ -36,9 +36,16 @@
                exit 1
        fi
        if [[ "$current_branch" != 'production' ]]; then
-               printf 'ERROR: Current branch in /srv/git/operations/puppet is "%s", should be "production"\n' "$current_branch" 1>&2
-               printf 'ERROR: Exiting rather than deploying something surprising\n' 1>&2
-               exit 1
+        GIT_DIR=/srv/git/operations/puppet/.git
+               if [[ -d $GIT_DIR/rebase-apply || -d $GIT_DIR/rebase-merge ]]; then
+                       # T397877
+                       printf 'INFO: skipping puppetserver update because a rebase is in progress\n'
+                       exit 2
+               else
+                       printf 'ERROR: Current branch in /srv/git/operations/puppet is "%s", should be "production"\n' "$current_branch" 1>&2
+                       printf 'ERROR: Exiting rather than deploying something surprising\n' 1>&2
+                       exit 1
+               fi
        fi
        envdir="${codedir}/environments"
        g10k_envdir="${codedir}/environments_staging"

Notice: /Stage[main]/Puppetserver/File[/usr/local/bin/puppetserver-deploy-code]/content: content changed '{sha256}74fc67ed6adb7eadcfa70bdbd82ba0819d6a161f0c0ad71d575bdd9dcf0d7367' to '{sha256}cf582ac7b96efb3b643b39bf0b68c1551bc094acbdb85120fbab8d04dd72c854'
Notice: Applied catalog in 10.56 seconds
bd808@deployment-puppetserver-1:~$ sudo su - gitpuppet
$ exec bash
gitpuppet@deployment-puppetserver-1:~$ cd /srv/git/operations/puppet/
gitpuppet@deployment-puppetserver-1:/srv/git/operations/puppet$ git rebase --interactive origin/production
INFO: skipping puppetserver update because a rebase is in progress
INFO: skipping puppetserver update because a rebase is in progress
INFO: skipping puppetserver update because a rebase is in progress
INFO: skipping puppetserver update because a rebase is in progress
INFO: skipping puppetserver update because a rebase is in progress
INFO: skipping puppetserver update because a rebase is in progress
INFO: skipping puppetserver update because a rebase is in progress
INFO: skipping puppetserver update because a rebase is in progress
Successfully rebased and updated refs/heads/production.
gitpuppet@deployment-puppetserver-1:/srv/git/operations/puppet$ sudo /usr/local/bin/puppetserver-deploy-code
INFO: Puppet code deployed
INFO: Puppet code cache evicted
gitpuppet@deployment-puppetserver-1:/srv/git/operations/puppet$

bd808 changed the task status from Open to In Progress.Jun 25 2025, 9:26 PM

bd808 claimed this task.

bd808 triaged this task as Medium priority.

Restricted Application added a project: User-bd808. · View Herald TranscriptJun 25 2025, 9:26 PM

Change #1163883 merged by Andrew Bogott:

[operations/puppet@production] puppetserver: check for rebase in puppetserver-deploy-code

https://gerrit.wikimedia.org/r/1163883

Maintenance_bot removed a project: Patch-For-Review.Aug 21 2025, 6:31 PM

bd808 closed this task as Resolved.Aug 21 2025, 9:12 PM

Is something preventing this fix from applying to labs/private?

profile::puppetserver::git::repos:
  "operations/puppet":
    branch: "production"
    hooks:
      …
      'post-merge': 'puppet:///modules/profile/puppetserver/git/operations/hooks/deploy-code.sh'
  "labs/private":
    branch: "master"
    hooks:
      …
      'post-merge': 'puppet:///modules/profile/puppetserver/git/operations/hooks/deploy-code.sh'
    link: "/etc/puppet/private"

It seems this script is shared between the operations/puppet and labs/private clone on deployment-puppetserver in Beta Cluster. But.. git-rebase is still very slow for labs/private with puppet deployments between each step.

INFO: Puppet code deployed
INFO: Puppet code cache evicted

Change #1198373 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/puppet@production] puppetserver: Generalize git-rebase fix to work for labs/private

https://gerrit.wikimedia.org/r/1198373

gerritbot added a project: Patch-For-Review.Oct 23 2025, 4:53 PM

In T397877#11303762, @Krinkle wrote:

It seems this script is shared between the operations/puppet and labs/private clone on deployment-puppetserver in Beta Cluster. But.. git-rebase is still very slow for labs/private with puppet deployments between each step.

Nothing I changed in this task would have made the script faster or slower. I only changed the logging logic to detect when a rebase is in progress in /srv/git/operations/puppet.

Change #1198373 merged by Andrew Bogott: