Page MenuHomePhabricator
Create Task
Maniphest T331098

jenkins.job=mediawiki-i18n-check-docker fails: couldnot lock config file .gitconfig: File exists
Closed, ResolvedPublicBUG REPORT
Actions

Description

Since ~ 2 weeks the jenkins.job=mediawiki-i18n-check-docker fails sometimes without showing a real error.

Example: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/893960 with https://integration.wikimedia.org/ci/job/mediawiki-i18n-check-docker/123994/console

A recheck fixes the issue then.

+ git fetch --quiet --update-head-ok --depth 2 git://contint1002.wikimedia.org/mediawiki/extensions/DiscussionTools +refs/zuul/REL1_39/Z3f9e336152c94858861b98ac8d8d3491:refs/zuul/REL1_39/Z3f9e336152c94858861b98ac8d8d3491
+ '[' -z REL1_39 ']'
+ git checkout -B REL1_39 FETCH_HEAD
Switched to a new branch 'REL1_39'
+ set +x
+ git submodule --quiet update --init --recursive
[mediawiki-i18n-check-docker] $ /bin/bash /tmp/jenkins17197787001907672904.sh
+ cd src
++ pwd
+ git config --global --add safe.directory /srv/jenkins/workspace/mediawiki-i18n-check-docker/src
error: could not lock config file /mnt/home/jenkins-deploy/.gitconfig: File exists

Which is due to T329266 adding the safe.directory setting.

Details

SubjectRepoBranchLines +/-
jjb: enhance mediawiki-i18n-check jobintegration/configmaster+31 -19
jjb: fix mediawiki-i18n-check-docker empty errorintegration/configmaster+5 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Raymond created this task.Mar 3 2023, 9:41 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 3 2023, 9:41 AM
Nikerabbit added a subscriber: hashar.Apr 4 2023, 8:10 AM

Recent example: https://integration.wikimedia.org/ci/job/mediawiki-i18n-check-docker/127709/console

09:15:02 [mediawiki-i18n-check-docker] $ /bin/bash /tmp/jenkins17197787001907672904.sh
09:15:02 + cd src
09:15:02 ++ pwd
09:15:02 + git config --global --add safe.directory /srv/jenkins/workspace/mediawiki-i18n-check-docker/src
09:15:02 error: could not lock config file /mnt/home/jenkins-deploy/.gitconfig: File exists
09:15:02 Build step 'Execute shell' marked build as failure

Fully manual review of big changes is time consuming.

Raymond added a comment.Apr 10 2023, 12:38 PM

Today ~ 200 repros failed without showing a real error :-(

Amire80 awarded a token.Apr 10 2023, 12:41 PM
abi_ added a comment.Apr 10 2023, 1:51 PM

Example: https://integration.wikimedia.org/ci/job/mediawiki-i18n-check-docker/128796/console

11:45:00 + git fetch --quiet --update-head-ok --depth 2 git://contint1002.wikimedia.org/mediawiki/extensions/DiscussionTools +refs/zuul/REL1_39/Z3f9e336152c94858861b98ac8d8d3491:refs/zuul/REL1_39/Z3f9e336152c94858861b98ac8d8d3491
11:45:02 + '[' -z REL1_39 ']'
11:45:02 + git checkout -B REL1_39 FETCH_HEAD
11:45:02 Switched to a new branch 'REL1_39'
11:45:02 + set +x
11:45:02 + git submodule --quiet update --init --recursive
11:45:02 [mediawiki-i18n-check-docker] $ /bin/bash /tmp/jenkins17197787001907672904.sh
11:45:02 + cd src
11:45:02 ++ pwd
11:45:02 + git config --global --add safe.directory /srv/jenkins/workspace/mediawiki-i18n-check-docker/src
11:45:02 error: could not lock config file /mnt/home/jenkins-deploy/.gitconfig: File exists
11:45:02 Build step 'Execute shell' marked build as failure
11:45:02 [PostBuildScript] - [INFO] Executing post build scripts.
11:45:03 [mediawiki-i18n-check-docker] $ /bin/bash -xe /tmp/jenkins9911740476435647155.sh

And an example of a different error: https://integration.wikimedia.org/ci/job/mediawiki-i18n-check-docker/128796/console; I'm not sure what's causing the error here:

13:26:15 Building remotely on integration-agent-docker-1023 (pipelinelib Docker blubber) in workspace /srv/jenkins/workspace/mediawiki-i18n-check-docker@4
13:26:16 [mediawiki-i18n-check-docker@4] $ /bin/bash -xe /tmp/jenkins6763737696491111990.sh
13:26:16 + set -eux
13:26:16 + mkdir -m 2777 -p src
13:26:16 [mediawiki-i18n-check-docker@4] $ /bin/bash /tmp/jenkins8326925099231278844.sh
13:26:16 + set -o pipefail
13:26:16 + exec docker run --entrypoint=/usr/bin/find --user=nobody --volume /srv/jenkins/workspace/mediawiki-i18n-check-docker@4:/workspace --security-opt seccomp=unconfined --init --rm --label jenkins.job=mediawiki-i18n-check-docker --label jenkins.build=128796 --env-file /dev/fd/63 docker-registry.wikimedia.org/buster:latest /workspace/src -mindepth 1 -delete
13:26:16 ++ /usr/bin/env
13:26:16 ++ egrep -v '^(HOME|SHELL|PATH|LOGNAME|MAIL)='
13:26:17 [mediawiki-i18n-check-docker@4] $ /bin/bash /tmp/jenkins9222105025128184359.sh
13:26:17 + set -o pipefail
13:26:17 ++ pwd
13:26:17 ++ pwd
13:26:17 + exec docker run --volume /srv/jenkins/workspace/mediawiki-i18n-check-docker@4/src:/src --volume /srv/jenkins/workspace/mediawiki-i18n-check-docker@4/cache:/cache --volume /srv/git:/srv/git:ro --security-opt seccomp=unconfined --init --rm --label jenkins.job=mediawiki-i18n-check-docker --label jenkins.build=128796 --env-file /dev/fd/63 docker-registry.wikimedia.org/releng/ci-src-setup-simple:0.4.2-s1
13:26:17 ++ /usr/bin/env
13:26:17 ++ egrep -v '^(HOME|SHELL|PATH|LOGNAME|MAIL)='
13:26:18 + git init
13:26:19 Initialized empty Git repository in /src/.git/
13:26:19 + git fetch --quiet --update-head-ok --depth 2 git://contint2001.wikimedia.org/mediawiki/extensions/CommentStreams +refs/zuul/master/Zac8e2d574e744cb6a91aa9031608fc82:refs/zuul/master/Zac8e2d574e744cb6a91aa9031608fc82
13:26:19 + '[' -z master ']'
13:26:19 + git checkout -B master FETCH_HEAD
13:26:19 Reset branch 'master'
13:26:19 + set +x
13:26:19 + git submodule --quiet update --init --recursive
13:26:19 [mediawiki-i18n-check-docker@4] $ /bin/bash /tmp/jenkins13449682070583816511.sh
13:26:19 + cd src
13:26:19 ++ pwd
13:26:19 + git config --global --add safe.directory /srv/jenkins/workspace/mediawiki-i18n-check-docker@4/src
13:26:19 ++ mktemp
13:26:19 + additions=/tmp/tmp.u9XQKja3le
13:26:19 + git show FETCH_HEAD -U0
13:26:19 + grep '^+'
13:26:19 Build step 'Execute shell' marked build as failure
13:26:19 [PostBuildScript] - [INFO] Executing post build scripts.
13:26:19 [mediawiki-i18n-check-docker@4] $ /bin/bash -xe /tmp/jenkins4904502183614127051.sh
13:26:19 + set -euxo pipefail
13:26:19 + docker ps -q --filter label=jenkins.job=mediawiki-i18n-check-docker --filter label=jenkins.build=128796
13:26:19 + xargs --no-run-if-empty docker stop
13:26:19 [PostBuildScript] - [INFO] Executing post build scripts.
Raymond added a comment.Apr 10 2023, 4:33 PM
In T331098#8768104, @Raymond wrote:

Today ~ 200 repros failed without showing a real error :-(

The main reason for the huge number is, that I had a network error on my side and restarted the export some minutes later. But I did not check if there are patches to review.

But that do not explain, why all (?) 2nd patches per repo fail for mediawiki-i18n-check-docker.

Nikerabbit added a comment.Oct 2 2023, 10:45 AM

Still happening:

08:47:36 + git config --global --add safe.directory /srv/jenkins/workspace/mediawiki-i18n-check-docker@2/src
08:47:36 error: could not lock config file /mnt/home/jenkins-deploy/.gitconfig: File exists
08:47:36 Build step 'Execute shell' marked build as failure

https://integration.wikimedia.org/ci/job/mediawiki-i18n-check-docker/154955/console

Nikerabbit added a comment.Oct 24 2023, 11:10 AM

https://integration.wikimedia.org/ci/job/mediawiki-i18n-check-docker/157299/console

Nikerabbit added a project: Release-Engineering-Team.Oct 24 2023, 11:29 AM

I'd like to request assistance with this task.

Reviewing translation updates manually is tedious and often boring work. We'd like to keep the number of patches needing manual review as low as possible.

We'd appreciate any help towards making the i18n-checker job more reliable.

Nikerabbit added a project: Language-Team (Language-2023-October-December).Oct 24 2023, 11:30 AM
Nikerabbit moved this task from Quarter Backlog to Blocked on the Language-Team (Language-2023-October-December) board.
thcipriani subscribed.Oct 27 2023, 4:25 PM
In T331098#9214975, @Nikerabbit wrote:

Still happening:

08:47:36 + git config --global --add safe.directory /srv/jenkins/workspace/mediawiki-i18n-check-docker@2/src
08:47:36 error: could not lock config file /mnt/home/jenkins-deploy/.gitconfig: File exists
08:47:36 Build step 'Execute shell' marked build as failure

https://integration.wikimedia.org/ci/job/mediawiki-i18n-check-docker/154955/console

Oh. A lot of these are running in parallel, yes? This seems like a parallelism problem operating on the .gitconfig file; i.e., multiple jobs trying to edit at the same time.

This looks like fallout from T329266. Since this job isn't running within a container each job is trying to modify the .gitconfig file at the same time (which fails due to the first job creating a .gitconfig.lock)

Nikerabbit added a comment.Oct 27 2023, 4:33 PM

Yeah we push a lot of patches to Gerrit in a short time frame.

thcipriani edited projects, added Release-Engineering-Team (Quid Pro Crow 🦃); removed Release-Engineering-Team.Nov 8 2023, 4:26 PM
Pginer-WMF edited projects, added Language-Team (Language-2024-January-March); removed Language-Team (Language-2023-October-December).Jan 8 2024, 10:35 AM
Pginer-WMF moved this task from Quarter Backlog to Blocked on the Language-Team (Language-2024-January-March) board.
Nikerabbit moved this task from Backlog to Synchronization on the affects-translatewiki.net board.Feb 7 2024, 8:53 AM
thcipriani edited projects, added Release-Engineering-Team (Now this 🫠); removed Release-Engineering-Team (Quid Pro Crow 🦃).Feb 14 2024, 4:10 PM
thcipriani assigned this task to hashar.Feb 14 2024, 5:53 PM
hashar renamed this task from jenkins.job=mediawiki-i18n-check-docker fails without a real error to jenkins.job=mediawiki-i18n-check-docker fails: couldnot lock config file .gitconfig: File exists.Feb 20 2024, 9:35 AM
hashar updated the task description. (Show Details)
hashar updated the task description. (Show Details)
gerritbot added a comment.Feb 20 2024, 11:02 AM

Change 1005049 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: enhance mediawiki-i18n-check job

https://gerrit.wikimedia.org/r/1005049

gerritbot added a project: Patch-For-Review.Feb 20 2024, 11:02 AM
hashar added a parent task: T329266: Debian security update for git silently broke mediawiki-i18n-check-docker.Feb 20 2024, 11:04 AM
hashar added a comment.Feb 22 2024, 2:48 PM

Another issue is the check failing WITHOUT any output such as:

13:26:19 + git show FETCH_HEAD -U0
13:26:19 + grep '^+'
13:26:19 Build step 'Execute shell' marked build as failure

The reason is that when the proposed change is behind the target branch, such as in:

* (master)
| * (change 1234,99) Localisation update
|/
*  whatever common parent

What Zuul is doing behind the hood, is that it merge the change with the branch and mark it with a reference (that shows in the Jenkins builds parameters as the ZUUL_REF parameter). So you get:

* (refs/zuul/master/Z987654321) Merge commit 'change (1234,99)' into HEAD
| \
| * (change 1234,99) Localisation update
* | (master)
|/
*  whatever common parent

And CI does fetch that refs/zuul/master/Z987654321. The script does a git show FETCH_HEAD which for a merge commit defaults to not showing anything at all. There is no diff, nothing matches ^+, grep fails and the build fails with a mysterious empty output.

That exactly matches:

In T331098#8768813, @Raymond wrote:

But that do not explain, why all (?) 2nd patches per repo fail for mediawiki-i18n-check-docker.

Cause I guess the 2nd patch causes Zuul to do a merge commit. The fix is to use git show -m --first-parent which cause it to show the difference for a merge commit following the first parent (master). I have battle tested that via something name git-changed-in-head and which I have ported in Quibble:

+        # Some explanations for the git command below:
+        # HEAD^ will not exist for an initial commit, we thus need `git show`
+        # --name-only: strip patch payload, only report the file being altered
+        # --diff-filter=ACM: only care about files Added, Copied or Modified
+        # --find-renames=100%: renamed files that had a slight change would be
+        #                      considered modified and thus included.
+        # -m: show differences for merge commits ...
+        # --first-parent: ... but only follow the first parent commit
+        # --format=format: : strip out the commit summary
+        cmd = [
+            'git', 'show', 'HEAD',
+            '--name-only',
+            '--diff-filter=ACM',
+            '--find-renames=100%',
+            '-m',
+            '--first-parent',
+            '--format=format:',
+        ]
gerritbot added a comment.Feb 22 2024, 2:58 PM

Change 1005774 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: fix mediawiki-i18n-check-docker empty error

https://gerrit.wikimedia.org/r/1005774

hashar added a comment.Feb 22 2024, 3:06 PM

The test case I use was a failing build of the checker. That was for mediawiki/extensions/GlobalBlocking change 1005424. The job failed with an empty error:

++ mktemp
+ additions=/tmp/tmp.RB2gqyOAHT
+ git show FETCH_HEAD -U0
+ grep '^+'
Build step 'Execute shell' marked build as failure

Which is how I found out the empty error issue.

After deploying the JJB patches, the job passes now:

set -euo pipefail; git show -m --first-parent --find-renames=100% FETCH_HEAD -U0 | grep '\''^+'\'' | tee /log/additions.txt'
+++ b/i18n/is.json
+	"globalblocking-block-ipinvalid": "IP-staðfangið ($1) sem þú ritaðir er ógilt.\nVinsamlegast athugaðu að þú getur ekki ritað notandanafn!",
gerritbot added a comment.Feb 22 2024, 3:10 PM

Change 1005774 merged by jenkins-bot:

[integration/config@master] jjb: fix mediawiki-i18n-check-docker empty error

https://gerrit.wikimedia.org/r/1005774

gerritbot added a comment.Feb 22 2024, 3:10 PM

Change 1005049 merged by jenkins-bot:

[integration/config@master] jjb: enhance mediawiki-i18n-check job

https://gerrit.wikimedia.org/r/1005049

hashar added a comment.Feb 22 2024, 3:12 PM

Summary

Since ~ 2 weeks the jenkins.job=mediawiki-i18n-check-docker fails sometimes without showing a real error.

I am confident that was due to some of the l10n change being tested by CI as merge commits of the change + the tip of the branch. With git show not showing the diff of merges, that did caused the empty error. Using git show -m --first-parent fixed it.

error: could not lock config file /mnt/home/jenkins-deploy/.gitconfig: File exists

That was due to a race condition between builds using git config to set safe.directory=$(pwd). The fix I have made is to move the logic to execute inside a container as it should.

I believe the issues are fixed, and at least that worked for the test case I had (rebuild the last failing build in CI). Things to check is whether the job still catch issues ;)

hashar awarded a token.Feb 22 2024, 3:13 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 22 2024, 3:31 PM
Nikerabbit moved this task from Blocked to Check after deployment on the Language-Team (Language-2024-January-March) board.Feb 27 2024, 11:02 AM

Thanks so much @hashar. Let's give a few days more to see if there are any unexpected issues and then close.

One thing I noticed is that I have to scroll a bit more now that all the additions are visible in the log. Not a problem for me though.

Nikerabbit closed this task as Resolved.Feb 29 2024, 2:42 PM
Nikerabbit moved this task from Check after deployment to Done on the Language-Team (Language-2024-January-March) board.
dancy moved this task from Backlog to Done on the Release-Engineering-Team (Now this 🫠) board.Mar 1 2024, 4:24 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL