Page MenuHomePhabricator

PipelineBot fails to push update to deployment-charts
Closed, ResolvedPublic

Description

After merging a patch into our repo (services/wikifeeds), we're suddenly seeing an error when PipelineBot tries to push an update to deployment-charts:

remote: ERROR: commit 5b3e4f2: email address releng@lists.wikimedia.org is not registered in your account, and you lack 'forge committer' permission.

Here is the console log:
https://integration.wikimedia.org/ci/job/wikifeeds-pipeline-publish/148/console

Details

Related Changes in Gerrit:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
hashar added subscribers: thcipriani, hashar.

I have updated all Jenkins plugins earlier today which updated Pipeline* , git and git-client plugins. Something must have regressed there (the previous build of that job from Sep 15th worked just fine).

The relevant bit are:

Running in /srv/jenkins-agent/workspace/wikifeeds-pipeline-publish/deployment-charts

+ ./update_version/update_version.py -s wikifeeds -v 2026-09-17-150402-production

+ git add -A
+ git config user.email tcipriani+pipelinebot@wikimedia.org
+ git config user.name PipelineBot
+ git commit -m 'wikifeeds: pipeline bot promote' -m 'Promote wikifeeds to version 2026-09-17-150402-production' -m 'Job: wikifeeds-pipeline-publish Build: 148'
[ba3fpfcl 5b3e4f2] wikifeeds: pipeline bot promote
 1 file changed, 1 insertion(+), 1 deletion(-)
[Pipeline] withCredentials
Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[Pipeline] {
[Pipeline] sh
+ set +x
+ git push origin HEAD:refs/for/master%topic=pipeline-promote,r=1651
remote: 
remote: Processing changes: refs: 1        
remote: Processing changes: refs: 1, done            
remote: ERROR: commit 5b3e4f2: email address releng@lists.wikimedia.org is not registered in your account, and you lack 'forge committer' permission.        
remote: The following addresses are currently registered:        
remote:    tcipriani+****@wikimedia.org        
remote: To register an email address, visit:        
remote: https://gerrit.wikimedia.org/r/settings#EmailAddresses        
remote: 
remote: 
To https://gerrit.wikimedia.org/r/operations/deployment-charts.git
 ! [remote rejected] HEAD -> refs/for/master%topic=pipeline-promote,r=1651 (commit 5b3e4f2: invalid committer)

Thus it runs git config to set the email to tcipriani+pipelinebot@wikimedia.org and then does a commit that will be sent.

The job ran on:

  • integration-agent-docker-1089.integration.eqiad1.wikimedia.cloud and I captured the workspace as /srv/jenkins/workspace/wikifeeds-pipeline-publish-BROKEN
  • contint1003.wikimedia.org with workspace captured at /srv/jenkins-agent/workspace/wikifeeds-pipeline-publish-BROKEN/deployment-charts

Looking on the later I see the commit was crafted with a different name and email which comes from Jenkins global config:

$ git -C /srv/jenkins-agent/workspace/wikifeeds-pipeline-publish-BROKEN/deployment-charts show --format=fuller
commit 5b3e4f2200c2bb10b4b019f8a223f860840b06c6 (HEAD -> ba3fpfcl)
Author:     Wikimedia CI <releng@lists.wikimedia.org>
AuthorDate: Thu Sep 17 15:05:10 2026 +0000
...

The config is correct:

$ git -C /srv/jenkins-agent/workspace/wikifeeds-pipeline-publish-BROKEN/deployment-charts config list --show-origin|grep user
file:/var/lib/jenkins-agent/.gitconfig	user.email=jenkins-agent@contint1003.wikimedia.org
file:/var/lib/jenkins-agent/.gitconfig	user.name=Wikimedia Jenkins Bot
file:.git/config	user.email=tcipriani+pipelinebot@wikimedia.org
file:.git/config	user.name=PipelineBot

So something is missing/overriding the config and that is a regression.

Jenkins does set:

GIT_COMMITTER_EMAIL="releng@lists.wikimedia.org"
GIT_COMMITTER_NAME="Wikimedia CI"

And surely if I set them from the command line, the environment variables do override the local config:

$ GIT_COMMITTER_EMAIL="releng@lists.wikimedia.org" \
   GIT_COMMITTER_NAME="Wikimedia CI" \
     git commit -m 'HELLO with env variable' --allow-empty
[ba3fpfcl c44bb4f] HELLO with env variable
 Author: PipelineBot <tcipriani+pipelinebot@wikimedia.org>
$ $ git show --no-patch --format=fuller
commit c44bb4f5bad6c305db152940519f9a6a2e92f65f (HEAD -> ba3fpfcl)
Author:     PipelineBot <tcipriani+pipelinebot@wikimedia.org>
AuthorDate: Thu Sep 17 17:04:24 2026 +0000

            vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Commit:     Wikimedia CI <releng@lists.wikimedia.org>
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

CommitDate: Thu Sep 17 17:04:24 2026 +0000

    HELLO with env variable

So my guess is previously the git commands were executed by the git / git-client plugin without passing the environment variable. That got changed at some point and the result is the per repository git config we set get overridden.

Change #1342764 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/pipelinelib@master] Set git user name/email explicitly

https://gerrit.wikimedia.org/r/1342764

Change #1342764 merged by jenkins-bot:

[integration/pipelinelib@master] Set git user name/email explicitly

https://gerrit.wikimedia.org/r/1342764

@Dbrant I've fixed up and merged @hashar's patch. That should resolve the issue.

Mentioned in SAL (#wikimedia-releng) [2026-09-17T21:27:54Z] <hashar> Re running postmerge for https://gerrit.wikimedia.org/r/c/mediawiki/services/wikifeeds/+/1342700 for T438379 # zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/services/wikifeeds --change 1342700,1

hashar claimed this task.

I have added this change back to postmerge using zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/services/wikifeeds --change 1342700,1.

The promoting change is https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1342812

Success!

Thank you @dduvall for the CI fix, having added test to my change and for the review / approval!