Page MenuHomePhabricator

Build Bullseye based Toolforge images
Closed, ResolvedPublic

Description

With a new Debian version it's time to build and deploy new runtime versions on Kubernetes :-)

Stalled until Bullseye comes out

Event Timeline

Majavah changed the task status from Open to Stalled.Jun 8 2021, 6:22 PM
Majavah created this task.
Majavah changed the task status from Stalled to Open.Aug 14 2021, 3:58 PM
Majavah claimed this task.

Change 713052 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P::toolforge: Prepare apt repos for bullseye

https://gerrit.wikimedia.org/r/713052

Change 713053 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/docker-images/toollabs-images@master] Add Bullseye based images

https://gerrit.wikimedia.org/r/713053

Change 713055 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/software/tools-webservice@master] kubernetes: Add Bullseye images

https://gerrit.wikimedia.org/r/713055

Change 713071 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/toolforge/jobs-framework-api@main] kubernetes: Add Bullseye images

https://gerrit.wikimedia.org/r/713071

Change 713052 merged by Andrew Bogott:

[operations/puppet@production] P::toolforge: Prepare apt repos for bullseye

https://gerrit.wikimedia.org/r/713052

Change 713053 merged by jenkins-bot:

[operations/docker-images/toollabs-images@master] Add Bullseye based images

https://gerrit.wikimedia.org/r/713053

Change 713055 merged by jenkins-bot:

[operations/software/tools-webservice@master] kubernetes: Add Bullseye images

https://gerrit.wikimedia.org/r/713055

Change 713079 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/software/tools-webservice@master] d/changelog: Prepare for 0.76 release

https://gerrit.wikimedia.org/r/713079

Change 713079 merged by jenkins-bot:

[operations/software/tools-webservice@master] d/changelog: Prepare for 0.76 release

https://gerrit.wikimedia.org/r/713079

Mentioned in SAL (#wikimedia-cloud) [2021-08-15T16:44:49Z] <majavah> tagged and building toollabs-webservice 0.76 with bullseye images defined T284590

Mentioned in SAL (#wikimedia-cloud) [2021-08-15T16:51:08Z] <majavah> starting build of initial bullseye based images - T284590

Mentioned in SAL (#wikimedia-cloud) [2021-08-15T17:21:58Z] <majavah> finished initial build of images: php74, jdk17, python39, ruby27 - T284590

Change 713071 merged by jenkins-bot:

[cloud/toolforge/jobs-framework-api@main] kubernetes: Add Bullseye images

https://gerrit.wikimedia.org/r/713071

Status update:

Following images are available for testing on both webservice and using the new Jobs framework:

  • jdk17
  • php74
  • python39
  • ruby27

Not yet:

  • node12 (will have to look why node10 is using a Stretch image with custom backport)
  • golang115 (@Bstorm had concerns about this + best practices with as-empty-as-possible images, needs discussion)

Most of them are untested. Please test them and report issues. Don't rely them on important tools yet. Thanks.

Mentioned in SAL (#wikimedia-cloud) [2021-08-15T17:54:06Z] <wm-bot> <lucaswerkmeister> deployed 9e864a3b9b (Python 3.9, no issues so far; CC T284590)

PagePile Visual Filter (one of my less important tools ^^) seems to work fine in Python 3.9. (Migration process: edit service.template, webservice stop, move the old venv out of the way, set up a new venv from a webservice shell, webservice start.)

Mentioned in SAL (#wikimedia-cloud) [2021-08-15T18:23:49Z] <wm-bot> <lucaswerkmeister> deployed 9235b38189 (Python 3.9, CC T284590)

I also migrated Ranker – I would also characterize that as “not important” but it gets used more than PagePile Visual Filter so if there are any issues (haven’t noticed any so far) I’ll hopefully hear about it soon.

Change 713267 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/docker-images/toollabs-images@master] Add node12-sssd

https://gerrit.wikimedia.org/r/713267

Change 713267 merged by jenkins-bot:

[operations/docker-images/toollabs-images@master] Add node12-sssd

https://gerrit.wikimedia.org/r/713267

Following images are available for testing on both webservice and using the new Jobs framework:

  • jdk17
  • php74
  • python39
  • ruby27

And now this also includes:

  • node12
  • bullseye-standalone

Inside toolforge-php74-sssd-web image when running composer I encountered

Warning from https://repo.packagist.org: Support for Composer 1 is deprecated and some packages will not be available. You should upgrade to Composer 2. See https://blog.packagist.com/deprecating-composer-1-support/

as well as

As there is no 'unzip' command installed zip files are being unpacked using the PHP zip extension.
This may cause invalid reports of corrupted archives. Besides, any UNIX permissions (e.g. executable) defined in the archives will be lost.
Installing 'unzip' may remediate them.

Both non-critical warning, but Composer 2 for one should be a welcome thing to have as it is faster.

Inside toolforge-php74-sssd-web image when running composer I encountered

Warning from https://repo.packagist.org: Support for Composer 1 is deprecated and some packages will not be available. You should upgrade to Composer 2. See https://blog.packagist.com/deprecating-composer-1-support/

as well as

As there is no 'unzip' command installed zip files are being unpacked using the PHP zip extension.
This may cause invalid reports of corrupted archives. Besides, any UNIX permissions (e.g. executable) defined in the archives will be lost.
Installing 'unzip' may remediate them.

Both non-critical warning, but Composer 2 for one should be a welcome thing to have as it is faster.

Can you file a separate task please? I can get those sorted.

Inside toolforge-php74-sssd-web image when running composer I encountered

Warning from https://repo.packagist.org: Support for Composer 1 is deprecated and some packages will not be available. You should upgrade to Composer 2. See https://blog.packagist.com/deprecating-composer-1-support/

as well as

As there is no 'unzip' command installed zip files are being unpacked using the PHP zip extension.
This may cause invalid reports of corrupted archives. Besides, any UNIX permissions (e.g. executable) defined in the archives will be lost.
Installing 'unzip' may remediate them.

Both non-critical warning, but Composer 2 for one should be a welcome thing to have as it is faster.

Can you file a separate task please? I can get those sorted.

I would note that from experience MediaWiki doesn’t fully support composer 2, which may be why toolforge uses v1

I would note that from experience MediaWiki doesn’t fully support composer 2, which may be why toolforge uses v1

Actually, it was a leftover from times when Debian packaged Composer versions were fully out of date and unusable. These days Debian packaged are fine, I just didn't change from our legacy git clone to the Debian packages when initially building the Bullseye images. Given composer in the php74 container doesn't appear to be in use yet (only php74 container I saw running on k8s did not use Composer), I went ahead and rebuilt the image with Composer 2 from debian repos.