Thanks for opening this task!

Generally, the Runners—both the standard and the Cloud Runners used by the linked job—should have access to the Wikimedia APT repository. Write access, however, is a bit different and currently only planned for Trusted Runners to the staging APT repository.

That said, we need to distinguish between the Runners themselves and the CI containers. Your job is using the docker-registry.wikimedia.org/wmf-debci-bookworm image. Configuring APT in the Runner and within the container are separate steps. However, both the wmf-debci-bookworm image and the default CI images should already be set up to access and validate packages from apt.wikimedia.org. I did a quick local test with the container to confirm:

docker run -it docker-registry.wikimedia.org/wmf-debci-bookworm /bin/bash
root@7f90450893c2:/# cat /etc/apt/sources.list
deb http://mirrors.wikimedia.org/debian/ bookworm main

deb http://security.debian.org/debian-security bookworm-security main
deb http://mirrors.wikimedia.org/debian/ bookworm-updates main
deb http://mirrors.wikimedia.org/debian/ bookworm-backports main contrib
deb http://apt.wikimedia.org/wikimedia bookworm-wikimedia main
root@7f90450893c2:/etc/apt# apt update
...
Get:4 http://apt.wikimedia.org/wikimedia bookworm-wikimedia InRelease [97.8 kB]
Get:6 http://apt.wikimedia.org/wikimedia bookworm-wikimedia/main amd64 Packages [70.6 kB]
...
Fetched 9743 kB in 3s (3323 kB/s)                        
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.

root@7f90450893c2:/etc/apt# apt list | grep bookworm-wikimedia | wc -l
187

Based on this, it seems the default images and wmf-debci should work as expected. The issue might lie in the EXTRACOMPONENTS configuration. The Merge Request specifies:

Signed-By: /etc/apt/keyrings/wikimedia-archive-keyring.gpg

But in the running container, this file is missing. Instead, I found /etc/apt/trusted.gpg.d/wikimedia-bookworm.pub.gpg with the same key ID (like in the error in this line). I'm not entirely sure about the difference between the keyrings and trusted.gpg.d directories, but it might make sense to use the latter here. What do you think?

Hm, yes, I took the path from a production host, where the key is installed into /etc/apt/keyrings by puppet (apt::package_from_component); you're right that the docker image is using /etc/apt/trusted.gpg.d. I should probably not ask why these two paths are different :)

Anyhow, testing in the bookworm-wikimedia image, if I use that path instead then it works:

# docker run -it docker-registry.wikimedia.org/wmf-debci-bookworm /bin/bash
[...]
root@396c7ff20f41:/# cat /etc/apt/sources.list.d/testing_mcv21.sources
Types: deb deb-src
URIs: http://apt.wikimedia.org/wikimedia
Suites: bookworm-wikimedia
Components: thirdparty/ceph-reef
Signed-By: /etc/apt/trusted.gpg.d/wikimedia-bookworm.pub.gpg
root@396c7ff20f41:/# apt update
[...]
Get:5 http://apt.wikimedia.org/wikimedia bookworm-wikimedia/thirdparty/ceph-reef amd64 Packages [11.0 kB]

I'll put in a MR to fix the keyring path shortly [let's leave aside whether we need to specify Signed-By for this purpose].

mvernon opened https://gitlab.wikimedia.org/repos/sre/wmf-debci/-/merge_requests/15

includes: fix path to WMF repo pubkey

mvernon merged https://gitlab.wikimedia.org/repos/sre/wmf-debci/-/merge_requests/15

includes: fix path to WMF repo pubkey

In T380164#10335127, @MatthewVernon wrote:

Hm, yes, I took the path from a production host, where the key is installed into /etc/apt/keyrings by puppet (apt::package_from_component); you're right that the docker image is using /etc/apt/trusted.gpg.d. I should probably not ask why these two paths are different :)

In T380164#10334895, @Jelto wrote:

But in the running container, this file is missing. Instead, I found /etc/apt/trusted.gpg.d/wikimedia-bookworm.pub.gpg with the same key ID (like in the error in this line). I'm not entirely sure about the difference between the keyrings and trusted.gpg.d directories, but it might make sense to use the latter here. What do you think?

apt-key got deprecated in Debian bullseye. /etc/apt/keyrings is the new and correct way. /etc/apt/trusted.gpg.d should not be used.

-->

The reason for this change is that when adding an OpenPGP key that's used to sign an APT repository to /etc/apt/trusted.gpg or /etc/apt/trusted.gpg.d, the key is unconditionally trusted by APT on all other repositories configured on the system that don't have a signed-by (see below) option, even the official Debian / Ubuntu repositories. As a result, any unofficial APT repository which has its signing key added to /etc/apt/trusted.gpg or /etc/apt/trusted.gpg.d can replace any package on the system. So this change was made for security reasons (your security).

In releases older than Debian 12 and Ubuntu 22.04, /etc/apt/keyrings does not exist by default. It SHOULD be created with permissions 0755 if it is needed and does not already exist.

The certificate MUST NOT be placed in /etc/apt/trusted.gpg.d or loaded by apt-key add.

https://wiki.debian.org/DebianRepository/UseThirdParty#OpenPGP_certificate_distribution

https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-how-to-add.html

https://stackoverflow.com/questions/68992799/warning-apt-key-is-deprecated-manage-keyring-files-in-trusted-gpg-d-instead

@Dzahn This is probably best to be opened as another ticket - wmf-debci isn't handling the placement of those files, it's using the docker-registry.wikimedia.org/${suite} image as a base!

[though if the path in the base image is fixed, wmf-debci will need fixing to match]

I was just trying to answer questions why this is the case and which of the 2 options is the newer/right way to handle it.

Of course - there's always five different ways to do it (four of which are deprecated) in Debian-land so it's good to get all that sorted.