Page MenuHomePhabricator
Create Task
Maniphest T437829

scap backport failed due to expired debian snapshot
Closed, ResolvedPublic
Actions

Description

scap backport failed due to an expired debian snapshot:

[0251][tstarling@deploy1003:~]$ scap backport 1340801 1340802 
...
03:00:24 [mediawiki-publish-85-next] E: Release file for http://snapshot.debian.org/archive/debian/20260906T040200Z/dists/bookworm-updates/InRelease is expired (invalid since 1d 0h 47min 38s). Updates for this repository will not be applied.
03:00:24 [mediawiki-publish-85-next] E: Release file for http://snapshot.debian.org/archive/debian-security/20260906T040200Z/dists/bookworm-security/InRelease is expired (invalid since 23h 35min 57s). Updates for this repository will not be applied.
03:00:24 [mediawiki-publish-83-next] E: Release file for http://snapshot.debian.org/archive/debian/20260906T040200Z/dists/bookworm-updates/InRelease is expired (invalid since 1d 0h 47min 38s). Updates for this repository will not be applied.
03:00:24 [mediawiki-publish-83-next] E: Release file for http://snapshot.debian.org/archive/debian-security/20260906T040200Z/dists/bookworm-security/InRelease is expired (invalid since 23h 35min 57s). Updates for this repository will not be applied.
...
03:00:25 [mediawiki-publish-85-next] The command '/bin/sh -c apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y php${PHP_VERSION}-xhprof' returned a non-zero code: 100

snapshot.debian.org says

To access snapshots of suites using Valid-Until that are older than a dozen days, it is necessary to ignore the Valid-Until header within Release files, in order to prevent apt from disregarding snapshot entries ("Release file expired"). Use aptitude -o Acquire::Check-Valid-Until=false update or apt-get -o Acquire::Check-Valid-Until=false update for this purpose."

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
build-bare-slim: Avoid snapshot.debian.org by using --non-debian (WIP)Muehlenhoffoperations/puppetproduction+1 -1
build-production-images: remove buster from base imagesGiuseppe Lavagettooperations/puppetproduction+0 -1
Customize query in gerrit

Related Objects

Event Timeline

• tstarling created this task.Mon, Sep 14, 3:38 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Sep 14, 3:38 AM
• tstarling renamed this task from scap backport due to expired debian snapshot to scap backport failed due to expired debian snapshot.Mon, Sep 14, 3:38 AM
• tstarling added a comment.Mon, Sep 14, 3:43 AM

Full terminal output: P96435

• tstarling added a comment.Mon, Sep 14, 4:38 AM

The base image is broken. Locally I can do

$ docker run -it docker-registry.wikimedia.org/bookworm:latest
root@9efc76ae3ac4:/# cat /etc/apt/sources.list.d/debian.sources 
Types: deb
URIs: http://snapshot.debian.org/archive/debian/20260913T040200Z
Suites: bookworm bookworm-updates
Components: main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb
URIs: http://snapshot.debian.org/archive/debian-security/20260913T040200Z
Suites: bookworm-security
Components: main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

That's the expired snapshot.

I think I've found where this comes from in debuerreotype, and I found wt:Kubernetes/Images#Base_images which explains how to rebuild the base images.

Stashbot added a comment.Mon, Sep 14, 4:52 AM

Mentioned in SAL (#wikimedia-operations) [2026-09-14T04:52:58Z] <TimStarling> on build2001 rebuilding base images T437829

• tstarling added a comment.Mon, Sep 14, 5:10 AM

Base image build gave warnings:

warning: no apparent 'bookworm/updates/main' for 'amd64' on any of the following; skipping
  - http://snapshot.debian.org/archive/debian-security/20260914T045700Z
warning: no apparent 'bookworm-updates/main' for 'amd64' on any of the following; skipping
  - http://snapshot.debian.org/archive/debian/20260914T045700Z
W: Failed to fetch http://snapshot.debian.org/archive/debian/20260914T045700Z/dists/bookworm/InRelease  Could not connect to snapshot.debian.org:80 (2a04:4e42:1b::644), connection timed out Could not connect to snapshot.debian.org:80 (146.75.106.132), connection timed out
W: Some index files failed to download. They have been ignored, or old ones used instead.

And I don't see the snapshot repo in /etc/apt/sources.list.d in the new image.

Stashbot added a comment.Mon, Sep 14, 5:39 AM

Mentioned in SAL (#wikimedia-operations) [2026-09-14T05:39:57Z] <_joe_> force-running build-base-images on build2004 for T437829

Stashbot added a comment.Mon, Sep 14, 5:51 AM

Mentioned in SAL (#wikimedia-operations) [2026-09-14T05:51:27Z] <_joe_> pulled bookworm:latest from build2004 to build2001 T437829

gerritbot added a comment.Mon, Sep 14, 6:05 AM

Change #1340943 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] build-production-images: remove buster from base images

https://gerrit.wikimedia.org/r/1340943

gerritbot added a project: Patch-For-Review.Mon, Sep 14, 6:05 AM
gerritbot added a comment.Mon, Sep 14, 6:09 AM

Change #1340943 merged by Giuseppe Lavagetto:

[operations/puppet@production] build-production-images: remove buster from base images

https://gerrit.wikimedia.org/r/1340943

Maintenance_bot removed a project: Patch-For-Review.Mon, Sep 14, 6:30 AM
hashar mentioned this in T437836: "apt update" failing in CI, on both bookworm and trixie, and both gerrit and gitlab.Mon, Sep 14, 9:52 AM
hashar mentioned this in T437069: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830.Mon, Sep 14, 9:54 AM
hashar subscribed.

There was a similar issue (Release file expired) that surfaced on Friday 4th and filed as T437069

• tstarling added a comment.Mon, Sep 14, 10:07 AM

I think it's mostly a bug in debuerreotype. The purpose of using snapshot.debian.org is for reproducible builds. The build isn't reproducible after the snapshot expires unless you configure apt to ignore Valid-Until. It doesn't seem to be intentional. I'm happy to file it upstream.

MoritzMuehlenhoff subscribed.Mon, Sep 14, 10:32 AM
In T437829#12316871, @tstarling wrote:

I think it's mostly a bug in debuerreotype. The purpose of using snapshot.debian.org is for reproducible builds. The build isn't reproducible after the snapshot expires unless you configure apt to ignore Valid-Until.

Agreed.

It doesn't seem to be intentional. I'm happy to file it upstream.

Sounds good!

gerritbot added a comment.Mon, Sep 14, 10:54 AM

Change #1341110 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] build-bare-slim: Avoid snapshot.debian.org by using --non-debian (WIP)

https://gerrit.wikimedia.org/r/1341110

gerritbot added a project: Patch-For-Review.Mon, Sep 14, 10:54 AM
MoritzMuehlenhoff added a comment.Mon, Sep 14, 10:56 AM
In T437829#12316871, @tstarling wrote:

I think it's mostly a bug in debuerreotype. The purpose of using snapshot.debian.org is for reproducible builds. The build isn't reproducible after the snapshot expires unless you configure apt to ignore Valid-Until. It doesn't seem to be intentional. I'm happy to file it upstream.

Looking at https://github.com/debuerreotype/debuerreotype/issues/75#issuecomment-557637414 using the (a little confusingly named) --non-debian option might already do what we need: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1341110

Tgr mentioned this in T437824: wikifeeds-pipeline-test fails due to expired Debian snapshot repositories.Mon, Sep 14, 1:37 PM
• taavi triaged this task as Unbreak Now! priority.Mon, Sep 14, 1:39 PM
• taavi subscribed.

Not being able to deploy any MW patches sounds like an UBN, so boldly setting priority.

Ladsgroup subscribed.Mon, Sep 14, 1:47 PM
MoritzMuehlenhoff added a comment.Mon, Sep 14, 2:59 PM

The weekly rebuild from yesterday failed as "buster" was still listed, but had been garbage-collected from the registry as part of the S3 migration work. Giuseppe merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1340943 earlier and I've kicked off a fresh rebuild of the production images, it's running on build2004 (production-images-weekly-rebuild.service).

Raymond_Ndibe subscribed.Mon, Sep 14, 3:36 PM

This is also an issue for us at toolforge. we started getting similar failure messages while trying to build trixie based images.
See https://phabricator.wikimedia.org/T437069#12318562 for more context

Joe subscribed.Mon, Sep 14, 4:00 PM
In T437829#12317806, @taavi wrote:

Not being able to deploy any MW patches sounds like an UBN, so boldly setting priority.

That has been resolved this morning.

Aklapper mentioned this in T430839: 1.47.0-wmf.20 deployment blockers.Mon, Sep 14, 4:14 PM
aputhin subscribed.Mon, Sep 14, 4:17 PM
EBernhardson mentioned this in T435824: Create opensearch image for 3.8.Mon, Sep 14, 4:29 PM
dancy closed this task as Resolved.Mon, Sep 14, 6:34 PM
dancy claimed this task.
• tstarling added a comment.Mon, Sep 14, 9:00 PM
In T437829#12317118, @MoritzMuehlenhoff wrote:

Looking at https://github.com/debuerreotype/debuerreotype/issues/75#issuecomment-557637414 using the (a little confusingly named) --non-debian option might already do what we need: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1341110

It just turns off reproducible builds. If I did a PR which gave us non-expiring reproducible builds, would that be useful?

• tstarling added a comment.Tue, Sep 15, 1:48 AM

I filed an issue and a PR https://github.com/debuerreotype/debuerreotype/issues/201

atsuko mentioned this in T437981: Airflow 3: remove Amazon provider warnings from CLI startup.Tue, Sep 15, 9:23 AM
atsuko mentioned this in T437994: CI for Airflow 3.Tue, Sep 15, 9:47 AM
Nemoralis merged a task: T437824: wikifeeds-pipeline-test fails due to expired Debian snapshot repositories.Thu, Sep 17, 7:56 PM
Nemoralis added subscribers: Nemoralis, Tgr.
MoritzMuehlenhoff mentioned this in T438866: Ignore Valid-Until for WMF container images.Tue, Sep 22, 3:09 PM
MoritzMuehlenhoff added a comment.Wed, Sep 23, 7:35 AM
In T437829#12320689, @tstarling wrote:

I filed an issue and a PR https://github.com/debuerreotype/debuerreotype/issues/201

Now tracked separately from this task under https://phabricator.wikimedia.org/T438866

gerritbot added a comment.Thu, Sep 24, 9:02 AM

Change #1341110 abandoned by Muehlenhoff:

[operations/puppet@production] build-bare-slim: Avoid snapshot.debian.org by using --non-debian (WIP)

Reason:

Alternate approach was chosen at https://phabricator.wikimedia.org/T438866

https://gerrit.wikimedia.org/r/1341110

Maintenance_bot removed a project: Patch-For-Review.Thu, Sep 24, 9:31 AM
Nemoralis unsubscribed.Thu, Sep 24, 11:14 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits