Page MenuHomePhabricator
Create Task
Maniphest T284590

Build Bullseye based Toolforge images
Closed, ResolvedPublic
Actions

Description

With a new Debian version it's time to build and deploy new runtime versions on Kubernetes :-)

Stalled until Bullseye comes out

Details

SubjectRepoBranchLines +/-
Add node12-sssdoperations/docker-images/toollabs-imagesmaster+33 -0
kubernetes: Add Bullseye imagescloud/toolforge/jobs-framework-apimain+14 -0
d/changelog: Prepare for 0.76 releaseoperations/software/tools-webservicemaster+6 -0
kubernetes: Add Bullseye imagesoperations/software/tools-webservicemaster+40 -0
Add Bullseye based imagesoperations/docker-images/toollabs-imagesmaster+350 -2
P::toolforge: Prepare apt repos for bullseyeoperations/puppetproduction+8 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

taavi changed the task status from Open to Stalled.Jun 8 2021, 6:22 PM
taavi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 8 2021, 6:22 PM
taavi added subtasks: T284586: webservice: Use python 3, T281596: Publish wikimedia-bullseye base docker image.Jun 8 2021, 6:23 PM
taavi added parent tasks: T264713: Add Python 3.8/3.9 to Toolforge Kubernetes/Job Grid, T266210: Add PHP 7.4 for Kubernetes, T243159: Request to enable node version 12.14.1 in toolforge to deploy VideoCutTool.
taavi moved this task from Unsorted to Kubernetes on the Toolforge (Software install/update) board.
bd808 subscribed.Jun 8 2021, 6:57 PM

There is a base image to start from at https://docker-registry.wikimedia.org/bullseye/tags/

taavi closed subtask T284586: webservice: Use python 3 as Resolved.Jul 2 2021, 11:54 AM
• nskaggs triaged this task as High priority.Aug 10 2021, 10:12 PM
MacFan4000 subscribed.Aug 14 2021, 3:19 PM
taavi changed the task status from Stalled to Open.Aug 14 2021, 3:58 PM
taavi claimed this task.
Restricted Application added a project: User-Majavah. · View Herald TranscriptAug 14 2021, 3:58 PM
gerritbot added a comment.Aug 14 2021, 3:58 PM

Change 713052 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P::toolforge: Prepare apt repos for bullseye

https://gerrit.wikimedia.org/r/713052

gerritbot added a project: Patch-For-Review.Aug 14 2021, 3:58 PM
gerritbot added a comment.Aug 14 2021, 4:04 PM

Change 713053 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/docker-images/toollabs-images@master] Add Bullseye based images

https://gerrit.wikimedia.org/r/713053

gerritbot added a comment.Aug 14 2021, 4:20 PM

Change 713055 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/software/tools-webservice@master] kubernetes: Add Bullseye images

https://gerrit.wikimedia.org/r/713055

gerritbot added a comment.Aug 15 2021, 7:58 AM

Change 713071 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/toolforge/jobs-framework-api@main] kubernetes: Add Bullseye images

https://gerrit.wikimedia.org/r/713071

gerritbot added a comment.Aug 15 2021, 2:33 PM

Change 713052 merged by Andrew Bogott:

[operations/puppet@production] P::toolforge: Prepare apt repos for bullseye

https://gerrit.wikimedia.org/r/713052

gerritbot added a comment.Aug 15 2021, 4:27 PM

Change 713053 merged by jenkins-bot:

[operations/docker-images/toollabs-images@master] Add Bullseye based images

https://gerrit.wikimedia.org/r/713053

gerritbot added a comment.Aug 15 2021, 4:28 PM

Change 713055 merged by jenkins-bot:

[operations/software/tools-webservice@master] kubernetes: Add Bullseye images

https://gerrit.wikimedia.org/r/713055

gerritbot added a comment.Aug 15 2021, 4:30 PM

Change 713079 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/software/tools-webservice@master] d/changelog: Prepare for 0.76 release

https://gerrit.wikimedia.org/r/713079

gerritbot added a comment.Aug 15 2021, 4:32 PM

Change 713079 merged by jenkins-bot:

[operations/software/tools-webservice@master] d/changelog: Prepare for 0.76 release

https://gerrit.wikimedia.org/r/713079

Stashbot added a comment.Aug 15 2021, 4:44 PM

Mentioned in SAL (#wikimedia-cloud) [2021-08-15T16:44:49Z] <majavah> tagged and building toollabs-webservice 0.76 with bullseye images defined T284590

Stashbot added a comment.Aug 15 2021, 4:51 PM

Mentioned in SAL (#wikimedia-cloud) [2021-08-15T16:51:08Z] <majavah> starting build of initial bullseye based images - T284590

taavi mentioned this in rODITc43dec9eb2cc: Add Bullseye based images.Aug 15 2021, 4:52 PM
Stashbot added a comment.Aug 15 2021, 5:22 PM

Mentioned in SAL (#wikimedia-cloud) [2021-08-15T17:21:58Z] <majavah> finished initial build of images: php74, jdk17, python39, ruby27 - T284590

gerritbot added a comment.Aug 15 2021, 5:27 PM

Change 713071 merged by jenkins-bot:

[cloud/toolforge/jobs-framework-api@main] kubernetes: Add Bullseye images

https://gerrit.wikimedia.org/r/713071

taavi added subscribers: • Bstorm, LucasWerkmeister.Aug 15 2021, 5:32 PM

Status update:

Following images are available for testing on both webservice and using the new Jobs framework:

  • jdk17
  • php74
  • python39
  • ruby27

Not yet:

  • node12 (will have to look why node10 is using a Stretch image with custom backport)
  • golang115 (@Bstorm had concerns about this + best practices with as-empty-as-possible images, needs discussion)

Most of them are untested. Please test them and report issues. Don't rely them on important tools yet. Thanks.

Stashbot added a comment.Aug 15 2021, 5:54 PM

Mentioned in SAL (#wikimedia-cloud) [2021-08-15T17:54:06Z] <wm-bot> <lucaswerkmeister> deployed 9e864a3b9b (Python 3.9, no issues so far; CC T284590)

LucasWerkmeister added a comment.EditedAug 15 2021, 5:55 PM

PagePile Visual Filter (one of my less important tools ^^) seems to work fine in Python 3.9. (Migration process: edit service.template, webservice stop, move the old venv out of the way, set up a new venv from a webservice shell, webservice start.)

Maintenance_bot removed a project: Patch-For-Review.Aug 15 2021, 6:10 PM
Stashbot added a comment.Aug 15 2021, 6:23 PM

Mentioned in SAL (#wikimedia-cloud) [2021-08-15T18:23:49Z] <wm-bot> <lucaswerkmeister> deployed 9235b38189 (Python 3.9, CC T284590)

LucasWerkmeister added a comment.Aug 15 2021, 6:24 PM

I also migrated Ranker – I would also characterize that as “not important” but it gets used more than PagePile Visual Filter so if there are any issues (haven’t noticed any so far) I’ll hopefully hear about it soon.

taavi mentioned this in T266210: Add PHP 7.4 for Kubernetes.Aug 16 2021, 2:08 PM
taavi mentioned this in T264713: Add Python 3.8/3.9 to Toolforge Kubernetes/Job Grid.
gerritbot added a comment.Aug 16 2021, 2:15 PM

Change 713267 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/docker-images/toollabs-images@master] Add node12-sssd

https://gerrit.wikimedia.org/r/713267

gerritbot added a project: Patch-For-Review.Aug 16 2021, 2:15 PM
Stashbot mentioned this in T243159: Request to enable node version 12.14.1 in toolforge to deploy VideoCutTool.Aug 16 2021, 2:19 PM

Mentioned in SAL (#wikimedia-cloud) [2021-08-16T14:19:16Z] <majavah> building node12 images - T284590 T243159

gerritbot added a comment.Aug 16 2021, 2:19 PM

Change 713267 merged by jenkins-bot:

[operations/docker-images/toollabs-images@master] Add node12-sssd

https://gerrit.wikimedia.org/r/713267

taavi mentioned this in rODITf413da192bcf: Add node12-sssd.Aug 16 2021, 2:19 PM
taavi added a comment.Aug 16 2021, 2:28 PM
In T284590#7283732, @Majavah wrote:

Following images are available for testing on both webservice and using the new Jobs framework:

  • jdk17
  • php74
  • python39
  • ruby27

And now this also includes:

  • node12
  • bullseye-standalone
Maintenance_bot removed a project: Patch-For-Review.Aug 16 2021, 3:10 PM
taavi closed this task as Resolved.Aug 16 2021, 5:06 PM

Announced: https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/W6XG6KXRZTEZKX636XELNI3IWIR4MAGE/

Base subscribed.Aug 20 2021, 2:39 PM

Inside toolforge-php74-sssd-web image when running composer I encountered

Warning from https://repo.packagist.org: Support for Composer 1 is deprecated and some packages will not be available. You should upgrade to Composer 2. See https://blog.packagist.com/deprecating-composer-1-support/

as well as

As there is no 'unzip' command installed zip files are being unpacked using the PHP zip extension.
This may cause invalid reports of corrupted archives. Besides, any UNIX permissions (e.g. executable) defined in the archives will be lost.
Installing 'unzip' may remediate them.

Both non-critical warning, but Composer 2 for one should be a welcome thing to have as it is faster.

taavi added a comment.Aug 20 2021, 3:05 PM
In T284590#7297767, @Base wrote:

Inside toolforge-php74-sssd-web image when running composer I encountered

Warning from https://repo.packagist.org: Support for Composer 1 is deprecated and some packages will not be available. You should upgrade to Composer 2. See https://blog.packagist.com/deprecating-composer-1-support/

as well as

As there is no 'unzip' command installed zip files are being unpacked using the PHP zip extension.
This may cause invalid reports of corrupted archives. Besides, any UNIX permissions (e.g. executable) defined in the archives will be lost.
Installing 'unzip' may remediate them.

Both non-critical warning, but Composer 2 for one should be a welcome thing to have as it is faster.

Can you file a separate task please? I can get those sorted.

MacFan4000 added a comment.Aug 20 2021, 3:35 PM
In T284590#7297843, @Majavah wrote:
In T284590#7297767, @Base wrote:

Inside toolforge-php74-sssd-web image when running composer I encountered

Warning from https://repo.packagist.org: Support for Composer 1 is deprecated and some packages will not be available. You should upgrade to Composer 2. See https://blog.packagist.com/deprecating-composer-1-support/

as well as

As there is no 'unzip' command installed zip files are being unpacked using the PHP zip extension.
This may cause invalid reports of corrupted archives. Besides, any UNIX permissions (e.g. executable) defined in the archives will be lost.
Installing 'unzip' may remediate them.

Both non-critical warning, but Composer 2 for one should be a welcome thing to have as it is faster.

Can you file a separate task please? I can get those sorted.

I would note that from experience MediaWiki doesn’t fully support composer 2, which may be why toolforge uses v1

Obscurationism subscribed.Aug 23 2021, 9:58 AM
taavi added a comment.Aug 23 2021, 2:10 PM
In T284590#7297887, @MacFan4000 wrote:

I would note that from experience MediaWiki doesn’t fully support composer 2, which may be why toolforge uses v1

Actually, it was a leftover from times when Debian packaged Composer versions were fully out of date and unusable. These days Debian packaged are fine, I just didn't change from our legacy git clone to the Debian packages when initially building the Bullseye images. Given composer in the php74 container doesn't appear to be in use yet (only php74 container I saw running on k8s did not use Composer), I went ahead and rebuilt the image with Composer 2 from debian repos.

dima.batt reopened subtask T281596: Publish wikimedia-bullseye base docker image as In Progress.Dec 28 2021, 3:12 PM
taavi closed subtask T281596: Publish wikimedia-bullseye base docker image as Resolved.Dec 28 2021, 3:18 PM
taavi mentioned this in T335507: Build Bookworm based Toolforge Kubernetes images.Apr 27 2023, 1:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits