Page MenuHomePhabricator
Create Task
Maniphest T307495

Train-dev: mirror-repos.sh does not fail if a git remote update fails
Closed, ResolvedPublicBUG REPORT
Actions

Description

What happens?:
When running a build of the train-dev environment, error was encountered while preparing the git repository mirror, but the build did not fail:

Preparing Gerrit repository mirror.  This will take a while
Cloning https://gerrit.wikimedia.org/r//mediawiki/core.git into mediawiki/core.git
Cloning https://gerrit.wikimedia.org/r//mediawiki/tools/release.git into mediawiki/tools/release.git
Cloning https://gerrit.wikimedia.org/r//mediawiki/tools/scap.git into mediawiki/tools/scap.git
Cloning https://gerrit.wikimedia.org/r//mediawiki/extensions.git into mediawiki/extensions.git
Cloning https://gerrit.wikimedia.org/r//mediawiki/services/function-schemata.git into mediawiki/services/function-schemata.git
Cloning https://gerrit.wikimedia.org/r//mediawiki/skins.git into mediawiki/skins.git
Cloning https://gerrit.wikimedia.org/r//mediawiki/vendor.git into mediawiki/vendor.git
Cloning https://gerrit.wikimedia.org/r//operations/mediawiki-config.git into operations/mediawiki-config.git
Cloning https://gerrit.wikimedia.org/r//operations/mediawiki-config/fonts.git into operations/mediawiki-config/fonts.git
Cloning https://gerrit.wikimedia.org/r//operations/deployment-charts.git into operations/deployment-charts.git
Cloning https://gerrit.wikimedia.org/r//VisualEditor/VisualEditor.git into VisualEditor/VisualEditor.git
Cloning https://gerrit.wikimedia.org/r//wikibase/javascript-api.git into wikibase/javascript-api.git
Cloning https://gerrit.wikimedia.org/r//wikibase/termbox.git into wikibase/termbox.git
Cloning https://gerrit.wikimedia.org/r//data-values/value-view.git into data-values/value-view.git
Cloning https://gerrit.wikimedia.org/r//wikimedia/portals/deploy.git into wikimedia/portals/deploy.git
Cloning https://phabricator.wikimedia.org/source//wikibase-data-model.git into wikibase-data-model.git
Cloning https://phabricator.wikimedia.org/source//wikibase-serialization.git into wikibase-serialization.git
Cloning https://phabricator.wikimedia.org/source//wikibase-termbox.git into wikibase-termbox.git
Cloning https://phabricator.wikimedia.org/source//datavalues-javascript.git into datavalues-javascript.git
Processing all extensions
Cloning https://gerrit.wikimedia.org/r//mediawiki/extensions/3D.git into mediawiki/extensions/3D.git
Cloning https://gerrit.wikimedia.org/r//mediawiki/extensions/AControlImageLink.git into mediawiki/extensions/AControlImageLink.git
Cloning https://gerrit.wikimedia.org/r//mediawiki/extensions/AJAXPoll.git into mediawiki/extensions/AJAXPoll.git
.
.
.
Cloning https://gerrit.wikimedia.org/r//mediawiki/extensions/SafeDelete.git into mediawiki/extensions/SafeDelete.git
Cloning https://gerrit.wikimedia.org/r//mediawiki/extensions/SandboxLink.git into mediawiki/extensions/SandboxLink.git
git remote update failed!
stdout:Fetching origin
stderr:POST git-upload-pack (128 bytes)
POST git-upload-pack (490 bytes)
error: RPC failed; curl 56 LibreSSL SSL_read: SSL_ERROR_SYSCALL, errno 60
error: 34828 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
error: Could not fetch origin
Cloning https://gerrit.wikimedia.org/r//mediawiki/extensions/Sarcasm.git into mediawiki/extensions/Sarcasm.git
Cloning https://gerrit.wikimedia.org/r//mediawiki/extensions/SaveSpinner.git into mediawiki/extensions/SaveSpinner.git
.
.
.
Cloning https://gerrit.wikimedia.org/r//mediawiki/extensions/WikiLovesMonuments.git into mediawiki/extensions/WikiLovesMonuments.git
Cloning https://gerrit.wikimedia.org/r//mediawiki/extensions/WikiSEO.git into mediawiki/extensions/WikiSEO.git
git remote update failed!
stdout:Fetching origin
stderr:POST git-upload-pack (128 bytes)
error: RPC failed; curl 56 LibreSSL SSL_read: SSL_ERROR_SYSCALL, errno 60
fatal: expected flush after ref listing
error: Could not fetch origin
Cloning https://gerrit.wikimedia.org/r//mediawiki/extensions/WikiShare.git into mediawiki/extensions/WikiShare.git

The mirror-repos.sh script is running some commands in parallel to speed up the build.

What should have happened instead?:

  • The build should fail, or it should re-attempt the failed update
    • mirror-repos.sh should ensure all subprocesses have succeeded in order to continue

Details

ProjectBranchLines +/-Subject
mediawiki/tools/train-devmaster+13 -10mirror-repos.sh: fix subprocess error handling
Customize query in gerrit

Event Timeline

jeena created this task.May 3 2022, 7:09 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 3 2022, 7:09 PM
jeena updated the task description. (Show Details)May 3 2022, 7:10 PM
dancy renamed this task from Train-dev: mirror-repos does not fail if a git remote update fails to Train-dev: mirror-repos.sh does not fail if a git remote update fails.May 3 2022, 7:32 PM
dancy updated the task description. (Show Details)
dancy added a subscriber: hashar.
dancy added a comment.May 3 2022, 7:51 PM

https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/train-dev/+/refs/heads/master/mirror-repos.sh#85
attempts to collecting the exit status of subprocesses but it only works for Bash 4+.

Bash v4 docs:

wait [-n] [n ...]
         Wait for each specified child process and return its termination status.  Each n may be a process ID or a job specification; if a job spec is given, all processes in that job's pipeline are waited for.  If n is not given, all currently active child processes are waited for, and the return status is zero.  If the -n option is supplied, wait waits for any job to terminate and returns its exit status.  If n specifies a non-existent process or job, the return status is 127.  Otherwise, the return status is the exit status of the last process or job waited for.

Bash v3 (the max supplied by default Mac OS) does not offer the -n flag. We'll need a better workaround than just calling plain wait. A possible workaround is to track the pids of the spawned subprocesses.

There is still a lurking bug at https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/train-dev/+/refs/heads/master/mirror-repos.sh#158 where plain 'wait' (not 'WAIT') is called. This will ignore the exit status from the remaining subprocesess.

gerritbot added a comment.May 4 2022, 9:47 PM

Change 789281 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[mediawiki/tools/train-dev@master] mirror-repos.sh: fix subprocess error handling

https://gerrit.wikimedia.org/r/789281

gerritbot added a project: Patch-For-Review.May 4 2022, 9:47 PM
gerritbot added a comment.May 6 2022, 2:08 PM

Change 789281 merged by jenkins-bot:

[mediawiki/tools/train-dev@master] mirror-repos.sh: fix subprocess error handling

https://gerrit.wikimedia.org/r/789281

Maintenance_bot removed a project: Patch-For-Review.May 6 2022, 2:31 PM
dancy closed this task as Resolved.May 6 2022, 3:12 PM
dancy claimed this task.

Fixed.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL