Page MenuHomePhabricator

Ignore Valid-Until for WMF container images
Closed, ResolvedPublic

Description

debuerreotype, the tool we use to create our container images build reproducible container images. Since Debian stable releases are still to some extent volatile (security updates, point releases) it doesn't fetch packages from the regular debian.org mirrors, but instead from snapshot.debian.org.

snapshot.d.o however configures a more strict Valid-Until setting compared to main Debian, quoting from the home page:

To access snapshots of suites using Valid-Until that are older than a dozen days, it is necessary to ignore the Valid-Until header within Release files, in order to prevent apt from disregarding snapshot entries ("Release file expired"). Use aptitude -o Acquire::Check-Valid-Until=false update or apt-get -o Acquire::Check-Valid-Until=false update for this purpose.

Valid-Until protects against outdated mirrors, which is valid for some use cases, but not needed here since we are in full control of the build and distribution of our internal images. And notably the current behaviour prevents us from using older container images if e.g. the weekly rebuild failed (as it happened in T437829, from which this task is spun off) and also from intentionally using older images where needed (as requested in https://phabricator.wikimedia.org/T437994#12345753).

@tstarling submitted an upstream pull request to configure the generated images to ignore V-U: https://github.com/debuerreotype/debuerreotype/issues/201

While this hasn't been merged upstream, it looks fine to me and I'll patch debuerreotype internally with that patch for now.

Details

Event Timeline

Change #1344257 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Only install debuerreotype on the server if base images are built

https://gerrit.wikimedia.org/r/1344257

Mentioned in SAL (#wikimedia-operations) [2026-09-23T14:39:29Z] <moritzm> upload debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia T438866

Mentioned in SAL (#wikimedia-operations) [2026-09-24T08:46:33Z] <moritzm> uploaded debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia T438866

I've updated build2004 to a package which includes Tim's patch. The next weekly rebuild will have Valid-Until disabled in the apt config and thus container images which can also be updated beyond the 12 day cutoff imposed by snapshot.debian.org

Valid-Until is now disabled for the base images (and production images built on top). For the latest trixie:20260927 image:

root@acb175dacd54:/# rgrep Valid-Until /etc/apt/
/etc/apt/apt.conf.d/50debuerreotype-valid-until.conf:Acquire::Check-Valid-Until "no";
root@acb175dacd54:/#

Also reproduced with faketime:

root@acb175dacd54:/# faketime '2026-12-01 12:00:00' apt-get update
Hit:1 http://security.debian.org/debian-security trixie-security InRelease
Hit:2 http://deb.debian.org/debian trixie InRelease                                             
Hit:3 http://deb.debian.org/debian trixie-updates InRelease                                     
Hit:4 http://deb.debian.org/debian trixie-backports InRelease                                   
Hit:5 http://snapshot.debian.org/archive/debian/20260927T040000Z trixie InRelease               
Hit:6 http://snapshot.debian.org/archive/debian/20260927T040000Z trixie-updates InRelease
Hit:7 http://snapshot.debian.org/archive/debian-security/20260927T040000Z trixie-security InRelease
Get:8 http://apt.wikimedia.org/wikimedia trixie-wikimedia InRelease [105 kB]
Fetched 105 kB in 1s (186 kB/s)    
Reading package lists... Done
root@acb175dacd54:/#

Change #1345857 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] etherpad: update image to newest version

https://gerrit.wikimedia.org/r/1345857

Change #1345857 merged by jenkins-bot:

[operations/deployment-charts@master] etherpad: update image to newest version

https://gerrit.wikimedia.org/r/1345857

Change #1344257 merged by Muehlenhoff:

[operations/puppet@production] Only install debuerreotype on the server if base images are built

https://gerrit.wikimedia.org/r/1344257