Page MenuHomePhabricator

Abstract out git commands for CI Docker containers
Closed, ResolvedPublic

Description

Lot of the CI images entry points share the same common code:

git init
git fetch --depth 2 --quiet "${ZUUL_URL}/${ZUUL_PROJECT}" "$ZUUL_REF"
git checkout --quiet FETCH_HEAD
git submodule --quiet update --init --recursive

ci-src-setup-simple even has a feature flag to skip processing submodules when GIT_NO_SUBMODULES is set:

set +x
if [ -z "${GIT_NO_SUBMODULES:-}" ]; then
    set -x
    git submodule --quiet update --init --recursive
else
    echo "\$GIT_NO_SUBMODULES set, skipping submodules"
fi

We also need to git protocol version 2 for the git versions not defaulting to it for T103990 (protocol.version=2)

And, I could not find the task, we need a git config to have submodules processed in parallel (submodule.fetchJobs which defaults to 1)

The aim of this task is to have a single shell script snippet in integration/config which addresses the above point and get included in the CI base images (ci-jessie, ci-stretch, ci-buster). Then get the entrypoints to use that.

Event Timeline

hashar created this task.Jun 26 2020, 10:06 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 26 2020, 10:06 AM

Change 608013 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: abstract common git operations

https://gerrit.wikimedia.org/r/608013

Change 608013 merged by jenkins-bot:
[integration/config@master] dockerfiles: abstract common git operations

https://gerrit.wikimedia.org/r/608013

hashar closed this task as Resolved.Jun 26 2020, 4:18 PM
hashar claimed this task.

I have rebuild the containers:

docker-registry.discovery.wmnet/releng/ci-common0.1260f2332830c428B
docker-registry.discovery.wmnet/releng/ci-src-setup-simple0.3.02b2d3502ac31174MB
docker-registry.discovery.wmnet/releng/ci-jessie0.5.0956458e7b428159MB
docker-registry.discovery.wmnet/releng/ci-stretch0.2.001e04dff218b144MB
docker-registry.discovery.wmnet/releng/ci-buster0.2.0d72904fed544174MB

Change 608069 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: use ci-src-setup-simple:0.3.0

https://gerrit.wikimedia.org/r/608069

hashar reopened this task as Open.Jun 26 2020, 4:21 PM

Bah still have to update jobs and rebuild all containers ...

Mentioned in SAL (#wikimedia-releng) [2020-06-29T12:48:43Z] <hashar> Updating Jenkins jobs for releng/ci-src-setup-simple:0.3.0 # https://gerrit.wikimedia.org/r/c/integration/config/+/608069 | T256462

Change 608069 merged by jenkins-bot:
[integration/config@master] jjb: use ci-src-setup-simple:0.3.0

https://gerrit.wikimedia.org/r/c/integration/config/ /608069

hashar triaged this task as Medium priority.Jul 27 2020, 1:32 PM

Change 622374 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: refresh images to use ci-src-setup.sh

https://gerrit.wikimedia.org/r/622374

Change 622374 merged by jenkins-bot:
[integration/config@master] dockerfiles: refresh images to use ci-src-setup.sh

https://gerrit.wikimedia.org/r/622374

Change 622417 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: update jobs to use /utils/ci-src-setup.sh

https://gerrit.wikimedia.org/r/622417

Change 622417 abandoned by Hashar:
[integration/config@master] jjb: update jobs to use /utils/ci-src-setup.sh

Reason:
faulty change :-(

https://gerrit.wikimedia.org/r/622417

Change 623762 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: update images to latest versions

https://gerrit.wikimedia.org/r/623762

Change 623762 merged by jenkins-bot:
[integration/config@master] jjb: update images to latest versions

https://gerrit.wikimedia.org/r/623762

hashar closed this task as Resolved.Sep 2 2020, 4:34 PM

I think I can now call it done. The script is in the scratch image docker-registry.wikimedia.org/releng/ci-common in utils/ci-src-setup.sh and is copied in each of our base images (ci-jessie, ci-stretch, ci-buster).