Page MenuHomePhabricator

Switch Toolforge installation of "composer" to use the Debian package
Closed, ResolvedPublic

Description

At the time we set up the integration/composer git repo because there was no Debian package and we wanted to use a relatively recent version. The version of composer in buster+ is recent enough (https://tracker.debian.org/pkg/composer) and I think switching to the packaged version would help us get rid of one more oddity. This is blocked on Toolforge fully moving to buster AIUI.

This will also take care of T249949: Stop using integration/composer and then archive the repo.

Related Objects

StatusSubtypeAssignedTask
Resolvedhashar
Resolved taavi
Resolvedaborrero
Resolvedaborrero
Resolvedaborrero
Resolvedaborrero
Resolvedaborrero
Resolved taavi
Resolvedaborrero
Resolved taavi
DuplicateNone
Resolved taavi
DeclinedNone
Resolvedaborrero
DeclinedNone
Resolvedaborrero
Resolved taavi
Resolved taavi
Resolved nskaggs
Declined taavi

Event Timeline

Change 723760 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P::toolforge: Use composer package on buster

https://gerrit.wikimedia.org/r/723760

taavi changed the task status from Open to In Progress.Sep 26 2021, 4:05 PM
taavi claimed this task.
taavi subscribed.

This was already done when building Bullseye based containers (php74). The above patch also does it on the grid for Buster and newer.

Note that buster's composer is broken when run in containers, but it's fine for usage on bastions/grid.

Change 723760 merged by Legoktm:

[operations/puppet@production] P::toolforge: Use composer package on buster

https://gerrit.wikimedia.org/r/723760

Is there anything left to do here with regards to unblocking the parent? Is that now done?

No, we're still using the integration/composer repository:

user@dev ~/g/o/d/toollabs-images> rg composer
php74-sssd/base/Dockerfile.template
6:# Unzip improves composer's handling of certain zip features (unix permissions for example)
9:        "composer",

php5-sssd/base/Dockerfile.template
25:# T172358 - install composer simlarly to ::toollabs::composer
26:{{ macros.integration_composer() }}

macros.jinja2
59: # Install composer from integration/composer.git.
61:{% macro integration_composer() %}
63:    "https://gerrit.wikimedia.org/r/p/integration/composer.git",
64:    "/srv/composer",
67:    && ln -s /srv/composer/vendor/bin/composer /usr/local/bin/composer

php72-sssd/base/Dockerfile.template
37:# T172358 - install composer simlarly to ::toollabs::composer
38:{{ macros.integration_composer() }}

php73-sssd/base/Dockerfile.template
31:# T172358 - install composer simlarly to ::toollabs::composer
32:{{ macros.integration_composer() }}

I think we should switch the older images to just download a phar with a pinned sha checksum.

Does anyone know where those toollabs-images live and how to update the process running those images?

If it helps, for the CI image it is done at https://gerrit.wikimedia.org/g/integration/config/+/refs/heads/master/dockerfiles/composer-scratch/

Dockerfile.template
FROM {{ "ci-buster" | image_tag }} AS build

USER root

RUN mkdir /srv/composer

COPY composer.phar.sha256sum /srv/composer/composer.phar.sha256sum

RUN {{ "curl" | apt_install }}

RUN cd /srv/composer \
    && curl --silent --fail --output composer.phar https://getcomposer.org/download/2.3.3/composer.phar \
    && sha256sum -c composer.phar.sha256sum

RUN chmod +x /srv/composer/composer.phar \
    && mv /srv/composer/composer.phar /usr/bin/composer

# Various helpful scripts, copied to /srv/composer; child containers are expected to copy them if needed.

# Wrapper script for composer that uses `COMPOSER_GITHUB_OAUTHTOKEN` if set.
COPY run-composer.sh /srv/composer/run-composer.sh


FROM scratch

COPY --from=build /srv/composer /srv/composer
COPY --from=build /usr/bin/composer /usr/bin/composer
composer.phar.sha256sum
d6931ec2b38b41bd0ad62f9d157908e6688bac091bbf0bd6a619c1067b922402 composer.phar

The operations/docker-images/toollabs-images repository has a Jinja2 macro named integration_composer:

macros.jinja2
{##
 # Install composer from integration/composer.git.
 #}
{% macro integration_composer() %}
{{ git_clone(
    "https://gerrit.wikimedia.org/r/p/integration/composer.git",
    "/srv/composer",
    continue=True,
) }}
    && ln -s /srv/composer/vendor/bin/composer /usr/local/bin/composer
{% endmacro %}

It is used by three php sssd images:

git grep integration_composer
php5-sssd/base/Dockerfile.template:{{ macros.integration_composer() }}
php72-sssd/base/Dockerfile.template:{{ macros.integration_composer() }}
php73-sssd/base/Dockerfile.template:{{ macros.integration_composer() }}

Which are based respectively on Debian Jessie, Stretch and Buster. I don't know whether those images are still of any use though.

Which are based respectively on Debian Jessie, Stretch and Buster. I don't know whether those images are still of any use though.

All three are still actively used by Toolforge tools per https://k8s-status.toolforge.org/images/:

  • php5: 171 running
  • php72: 88 running
  • php73: 391 running (most used image in all of the Toolforge k8s cluster)
git grep integration_composer
php5-sssd/base/Dockerfile.template:{{ macros.integration_composer() }}
php72-sssd/base/Dockerfile.template:{{ macros.integration_composer() }}
php73-sssd/base/Dockerfile.template:{{ macros.integration_composer() }}

Which are based respectively on Debian Jessie, Stretch and Buster. I don't know whether those images are still of any use though.

Jessie and Stretch got removed by 7fc63314f51db5e0a92b69949fdb3e29e02e0448.

Debian Buster provides composer 1.8.4.

I guess the remaining image can be switched to download composer.phar. From the releng/composer-scratch image we have:

dockerfiles/composer-scratch/composer.phar.sha256sum
f1b94fee11a5bd6a1aae5d77c8da269df27c705fcc806ebf4c8c2e6fa8645c20 composer.phar
Dockerfile.template
FROM {{ "ci-buster" | image_tag }} AS build

USER root

RUN mkdir /srv/composer

COPY composer.phar.sha256sum /srv/composer/composer.phar.sha256sum

RUN {{ "curl" | apt_install }}

RUN cd /srv/composer \
    && curl --silent --fail --output composer.phar https://getcomposer.org/download/2.5.1/composer.phar \
    && sha256sum -c composer.phar.sha256sum

RUN chmod +x /srv/composer/composer.phar \
    && mv /srv/composer/composer.phar /usr/bin/composer

# Various helpful scripts, copied to /srv/composer; child containers are expected to copy them if needed.

# Wrapper script for composer that uses `COMPOSER_GITHUB_OAUTHTOKEN` if set.
COPY run-composer.sh /srv/composer/run-composer.sh

FROM scratch

COPY --from=build /srv/composer /srv/composer
COPY --from=build /usr/bin/composer /usr/bin/composer

Change 991594 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/docker-images/toollabs-images@master] Stop building buster based images

https://gerrit.wikimedia.org/r/991594

Change 991594 merged by jenkins-bot:

[operations/docker-images/toollabs-images@master] Stop building buster based images

https://gerrit.wikimedia.org/r/991594