Page MenuHomePhabricator

Remove jessie + stretch containers from Toolforge
Open, MediumPublic

Description

Debian Jessie has reached "End of Life" upstream for some time. Toolforge removed Jessie hosts as well some time ago: T236565: "tools" Cloud VPS project jessie deprecation. Since containers exist in a different support space (running as non-privileged and mostly immutable elements of the system), they've been allowed to continue running for a while, but the runtimes in them are now so old that there is little-to-no maintenance of any of them which can cause serious problems for tools that still use them.

In some cases, this will require developers to test and or update their code on Debian stretch or (better yet) buster containers as soon as possible before we remove jessie from the image registry and then shut down containers still running it.

The jessie-based images are:

  • toolforge-python2-sssd-base/web
  • toolforge-php5-sssd-base/web
  • toolforge-node6-sssd-base/web
  • toolforge-python34-sssd-base/web
  • toolforge-ruby21-sssd-base/web
  • toolforge-jessie-sssd
  • wikimedia-jessie (also cached in our registry)

Process (dates to come):

  • Remove the definitions from toollabs-images
  • Announce the change to the community with some time for lead development, assistance and feedback - it should be no surprise as the webservice cli will inform you of the deprecation
  • Remove the images from the registry
  • Delete running jessie-based containers
  • Delete running stretch-based containers

Event Timeline

This should probably have a tally of how many containers are running with these images before pinning dates up so we know how big a problem it is.

From https://k8s-status.toolforge.org/images/ just now:

toolforge-php5-sssd-web213
toolforge-python34-sssd-web89
toolforge-python2-sssd-web24
toolforge-node6-sssd-web13
toolforge-python2-sssd-base3
toolforge-ruby21-sssd-web2
toolforge-node6-sssd-base1

The toolforge-php5-sssd-web image was the default until rOSTW7b338a06c479: kubernetes: Set php7.3 as the default type was deployed which I think was on 2020-05-07. We talked multiple times about "stop using PHP 5" campaigns, but never really did anything meaningful as a result.

nskaggs moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.
Andrew renamed this task from Remove jessie containers from Toolforge to Remove jessie + stretch containers from Toolforge.Aug 27 2021, 7:14 PM

I do have something of a question around stretch images. The Debian LTS team supports Stretch into 2022. As long as there are updates available, why remove container images that don't use puppet? Jessie has been dropped by LTS and must be removed.

I do have something of a question around stretch images. The Debian LTS team supports Stretch into 2022. As long as there are updates available, why remove container images that don't use puppet? Jessie has been dropped by LTS and must be removed.

Two reasons that come to my mind:

  • We don't have any automation around security upgrades, and so would need to manually look for + actually rebuild the containers when there is some issue. This of course applies to newer containers too, but having less containers to worry about is better in my mind.
  • It blocks (with the grid upgrade) using Python 3.6 and newer in webservice and related tooling

The build files for the container images have been removed in rODIT7fc63314f51d: Remove jessie and stretch image configuration & rODITf8ade4849651: Remove jessie-sssd & stretch-sssd image configuration. Now we need to draft a timeline for removing the containers from the registry and by extension the cluster and announce that timeline via cloud-announce and other appropriate channels.

The report available at https://k8s-status.toolforge.org/images/ can be used to get a reasonable estimate of the current usage of the affected images:

  • jessie
    • python2
    • php5
    • node6
    • python34
    • ruby21
  • stretch
    • python35
    • golang
    • jdk8
    • node10
    • php72