scap backport fails to build a image for k8s deployment
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Urbanecm_WMF
	Aug 17 2023, 1:20 PM

Description

Today, while deploying a patch to production, I saw the following error:

  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131
Reading package lists...
W: GPG error: http://mirrors.wikimedia.org/debian buster-backports InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131
E: The repository 'http://mirrors.wikimedia.org/debian buster-backports InRelease' is not signed.

full k8s build transcript is at P50592.

I finished the deployment, as scap said non-k8s deployment will proceed normally, but this has very likely caused a code difference between k8s and non-k8s deployed versions.

Details

	Subject	Repo	Branch	Lines +/-
	Re-update artificially images to overcome a docker-pkg bug	operations/docker-images/production-images	master	+18 -0

Customize query in gerrit

Related Objects

Mentioned In: T344478: Fix how we keep docker-pkg based images up to date
Mentioned Here: T343978: Allow more flexibility in ResourceQuota and LimitRanger config
T338952: mwaddlink fails to build because of a missing public key

Event Timeline

Urbanecm_WMF created this task.Aug 17 2023, 1:20 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 17 2023, 1:20 PM

Seems like a critical bug.

Tagging Infrastructure-Foundations Packaging because that looks like it may be an issue with the signing of the InRelease file on the apt repo.

Clement_Goubert added projects: Infrastructure-Foundations, Packaging.Aug 17 2023, 1:23 PM

In T344438#9099184, @Clement_Goubert wrote:

Tagging Infrastructure-Foundations Packaging because that looks like it may be an issue with the signing of the InRelease file on the apt repo.

FWIW, a similar error happened as T338952: mwaddlink fails to build because of a missing public key recently, and it seems to only need a "bump" of some intermediate images. Might be something similar?

Raising to UBN, this breaks all mw-on-k8s deployments.

@Joe is investigating

Change 949979 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/docker-images/production-images@master] Re-update artificially images to overcome a docker-pkg bug

https://gerrit.wikimedia.org/r/949979

gerritbot added a project: Patch-For-Review.Aug 17 2023, 1:57 PM

Change 949979 merged by Giuseppe Lavagetto:

[operations/docker-images/production-images@master] Re-update artificially images to overcome a docker-pkg bug

https://gerrit.wikimedia.org/r/949979

Maintenance_bot removed a project: Patch-For-Review.Aug 17 2023, 2:10 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-17T14:44:36Z] <claime> Rolling back 949583 for T344438

Mentioned in SAL (#wikimedia-operations) [2023-08-17T14:49:09Z] <claime> Re-deploying mw-on-k8s T344438

We encountered another issue while fixing, which delayed resolution a bit (details at T343978), but it's all fixed and deployments work again since ~15:00UTC

Joe mentioned this in T344478: Fix how we keep docker-pkg based images up to date.Aug 18 2023, 6:27 AM

scap backport fails to build a image for k8s deploymentClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

scap backport fails to build a image for k8s deployment
Closed, ResolvedPublic
Actions