Page MenuHomePhabricator

Review and update LPL service base images – 2026Q3
Open, In Progress, HighPublic4 Estimated Story Points

Description

Update the base images used by LPL-maintained services to supported/current images.

This task covers one maintenance cycle. The base images should be reviewed periodically (roughly quarterly), but each review should be a separate completable task rather than keeping this task open indefinitely.

Scope

The current LPL service base images are:

ServiceCurrent baseActionStatus
Apertiumdocker-registry.wikimedia.org/trixie:20251026Update to a current Trixie imageUpdated to 20260927
CXServerdocker-registry.wikimedia.org/nodejs24-slimReview only; no update currently identified; Wait for next Node 26?-
Machine Translationdocker-registry.wikimedia.org/python3-build-bookwormUpdate to python3-build-trixieDeployed in staging for more testing
Recommendation APIdocker-registry.wikimedia.org/python3-build-bookwormUpdate to python3-build-trixie-

For Apertium, the update to the newer Trixie base image has already identified that mirrors.wikimedia.org is no longer available. The existing patch updates the build accordingly.

For the Python services, verify that their Python/system dependencies and container build steps are compatible with the Trixie-based image.

If an individual service requires substantial application changes rather than a straightforward base-image update, split that work into a separate task.

Acceptance criteria

  • All LPL service base images listed above have been reviewed.
  • Apertium uses a current supported Trixie base image and its container builds successfully.
  • Machine Translation uses python3-build-trixie and builds/tests successfully.
  • Recommendation API uses python3-build-trixie and builds/tests successfully.
  • CXServer has been reviewed and either:
    • requires no change, or
    • any required update is included/documented.
  • Updated service images are tested in staging before production deployment.
  • Updated images are deployed to production and the affected services operate normally afterwards.
  • Any service intentionally left unchanged has the reason recorded in this task.

Dependencies and prerequisites

No known implementation blockers.

Normal deployment access and review are required for the affected services.

Recommendation API follows its existing ml-services deployment process rather than the deployment process used by the other LPL services.

No documentation, configuration migration, data migration, or coordinated client-side update is currently expected.

QA plan

Risks and affected areas

The main risks are:

  • Debian package availability or package names changing between Bookworm and Trixie.
  • Python dependencies being incompatible with the Python/runtime environment provided by the new base image.
  • Container build/install scripts relying on repositories or infrastructure that are no longer available.
  • A service building successfully but failing to start or behave correctly after deployment.

The change should not alter application functionality; expected behaviour before and after the base-image update is the same.

Automated testing

For each changed service:

  • The container image must build successfully in CI.
  • Existing unit/integration tests must pass using the new base image.
  • Dependency installation must complete without relying on unsupported/obsolete repositories.

No new application-level automated tests are required solely for changing the base image.

Staging and manual verification

Deploy each changed image to its normal staging environment before production.

For each service:

  • Verify the deployment completes and instances/pods become healthy.
  • Exercise a representative API request or normal service operation.
  • Check service logs for startup errors, dependency/runtime errors, or other new warnings attributable to the base-image change.

Use the existing service-specific staging procedures, including:

  • Apertium: test a translation request through the staging CXServer/Apertium integration.
  • Machine Translation: test a translation through the machinetranslation staging service.
  • Recommendation API: test a recommendation request against the Recommendation API staging deployment.

There is no need to duplicate these checks in a separate local manual-testing phase; staging is the appropriate integration environment for this infrastructure change.

Production deployment and verification

After successful staging verification, deploy the updated image for each service using its normal production deployment procedure.

After each rollout:

  • Verify the new image version is running and instances/pods are healthy.
  • Perform a representative production smoke test.
  • Check recent service logs/monitoring for new errors.

If a deployment fails or introduces a regression, roll back to the previously deployed image version and investigate before retrying.

Recurrence

After completing this task, create the next base-image review task for approximately three months later. The next task should start by inventorying the base images used by LPL-maintained services and comparing them against currently supported Wikimedia base images.

Event Timeline

Can you list the images we maintain and which one of those require an update?

$ grep -H "base:" */.pipeline/blubber.yaml
apertium/.pipeline/blubber.yaml:base: docker-registry.wikimedia.org/trixie:20251026
apertium/.pipeline/blubber.yaml:    base: docker-registry.wikimedia.org/trixie:20251026
cxserver/.pipeline/blubber.yaml:base: docker-registry.wikimedia.org/nodejs24-slim
machinetranslation/.pipeline/blubber.yaml:base: docker-registry.wikimedia.org/python3-build-bookworm
recommendation-api/.pipeline/blubber.yaml:base: docker-registry.wikimedia.org/python3-build-bookworm

We can surely update apertium, machinetranslation, and recommendation-api here. I'll verify Python dependencies as well.

Change #1338889 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/apertium@master] Update base image to trixie:20260308

https://gerrit.wikimedia.org/r/1338889

For Apertium, we needed to rebuild the image as mirrors.w.o is no longer available. Fixed it with the 20260906 tag.

Nikerabbit renamed this task from Update base images for LPL services to Review and update LPL service base images – 2026Q3.Mon, Sep 14, 7:44 AM
Nikerabbit triaged this task as Medium priority.
Nikerabbit added a project: Essential-Work.
Nikerabbit updated the task description. (Show Details)
Nikerabbit set the point value for this task to 4.

Change #1338889 merged by jenkins-bot:

[mediawiki/services/apertium@master] Update base image to trixie:20260914

https://gerrit.wikimedia.org/r/1338889

Change #1341866 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update Apertium to 2026-09-15-084320-production

https://gerrit.wikimedia.org/r/1341866

For Apertium, we needed to rebuild the image as mirrors.w.o is no longer available. Fixed it with the 20260906 tag.

Updated image again.

Change #1341866 merged by jenkins-bot:

[operations/deployment-charts@master] Update Apertium to 2026-09-15-084320-production

https://gerrit.wikimedia.org/r/1341866

Mentioned in SAL (#wikimedia-operations) [2026-09-16T04:56:16Z] <kart_> Updated Apertium to 2026-09-15-084320-production (T437213)

Nikerabbit raised the priority of this task from Medium to High.Wed, Sep 16, 11:49 AM

Change #1342539 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/machinetranslation@master] Update base image to trixie

https://gerrit.wikimedia.org/r/1342539

Change #1342539 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] Update base image to trixie

https://gerrit.wikimedia.org/r/1342539

Change #1343502 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] machinetranslation: staging: Update to 2026-09-21-112314-production

https://gerrit.wikimedia.org/r/1343502

Nikerabbit changed the task status from Open to In Progress.Tue, Sep 22, 11:10 AM

Change #1343502 merged by jenkins-bot:

[operations/deployment-charts@master] machinetranslation: staging: Update to 2026-09-21-112314-production

https://gerrit.wikimedia.org/r/1343502

Change #1344129 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[research/recommendation-api@master] WIP: Update base image to Trixie

https://gerrit.wikimedia.org/r/1344129

Mentioned in SAL (#wikimedia-operations) [2026-09-24T06:41:50Z] <kart_> staging: Update machinetranslation/MinT to 2026-09-21-112314-production (T437213)

Change #1344659 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/apertium@master] WIP: Apertium: Add python3-lxml package and update image

https://gerrit.wikimedia.org/r/1344659

Change #1344659 merged by jenkins-bot:

[mediawiki/services/apertium@master] Apertium: Add python3-lxml package and update image

https://gerrit.wikimedia.org/r/1344659

Change #1346288 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update Apertium to 2026-09-30-051826-production

https://gerrit.wikimedia.org/r/1346288

Change #1346288 merged by jenkins-bot:

[operations/deployment-charts@master] Update Apertium to 2026-09-30-051826-production

https://gerrit.wikimedia.org/r/1346288

Mentioned in SAL (#wikimedia-operations) [2026-10-07T05:10:36Z] <kart_> Updated Apertium to 2026-09-30-051826-production (T437213)

Change #1344129 merged by jenkins-bot:

[research/recommendation-api@master] Update base image to Trixie

https://gerrit.wikimedia.org/r/1344129

Change #1352720 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update Recommendation API to 2026-10-07-145851-production

https://gerrit.wikimedia.org/r/1352720

Change #1352722 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] machinetranslation: Update to 2026-09-21-112314-production

https://gerrit.wikimedia.org/r/1352722