Page MenuHomePhabricator

APT errors when installing custom packages in MediaWiki-Docker
Open, Needs TriagePublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

FROM docker-registry.wikimedia.org/dev/buster-php81-fpm:1.0.1-s2

RUN apt update && \
   apt install -y sqlite3 php8.1-pcov wget vim
  • Run docker compose build

What happens?:
It fails with the following errors:

 => [mediawiki internal] load build definition from Dockerfile                                                                                                                 0.0s
 => => transferring dockerfile: 3.06kB                                                                                                                                         0.0s
 => [mediawiki internal] load metadata for docker-registry.wikimedia.org/dev/buster-php81-fpm:1.0.1-s2                                                                         0.9s
 => [mediawiki-web internal] load build definition from DockerfileWeb                                                                                                          0.0s
 => => transferring dockerfile: 501B                                                                                                                                           0.0s
 => [mediawiki-web internal] load metadata for docker-registry.wikimedia.org/dev/buster-apache2:2.0.1                                                                          0.9s
 => [mediawiki internal] load .dockerignore                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                0.0s
 => [mediawiki-web internal] load .dockerignore                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                0.0s
 => CACHED [mediawiki 1/9] FROM docker-registry.wikimedia.org/dev/buster-php81-fpm:1.0.1-s2@sha256:c4a701d8aeb51c8c99c3dd73a0151598055975ebc64feacb601f3c5d65401876            0.0s
 => => resolve docker-registry.wikimedia.org/dev/buster-php81-fpm:1.0.1-s2@sha256:c4a701d8aeb51c8c99c3dd73a0151598055975ebc64feacb601f3c5d65401876                             0.0s
 => CACHED [mediawiki-web 1/3] FROM docker-registry.wikimedia.org/dev/buster-apache2:2.0.1@sha256:c897ff8d994484163e4c19154b055c23c55ba2e02e44be4954e465b7ae1323e5             0.0s
 => => resolve docker-registry.wikimedia.org/dev/buster-apache2:2.0.1@sha256:c897ff8d994484163e4c19154b055c23c55ba2e02e44be4954e465b7ae1323e5                                  0.0s
 => ERROR [mediawiki 2/9] RUN apt update &&    apt install -y sqlite3 php8.1-pcov wget vim                                                                                     3.7s
 => CANCELED [mediawiki-web 2/3] RUN apt update && apt install -y vim                                                                                                          3.8s
------                                                                                                                                                                              
 > [mediawiki 2/9] RUN apt update &&    apt install -y sqlite3 php8.1-pcov wget vim:                                                                                                
0.258                                                                                                                                                                               
0.258 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.                                                                                               
0.258                                                                                                                                                                               
0.380 Get:1 http://security.debian.org buster/updates InRelease [34.8 kB]                                                                                                           
0.480 Get:2 https://packages.sury.org/php buster InRelease [7559 B]                                                                                                                 
0.556 Get:3 http://apt.wikimedia.org/wikimedia buster-wikimedia InRelease [178 kB]                                                                                                  
0.582 Get:4 http://security.debian.org buster/updates/main amd64 Packages [595 kB]                                                                                                  
0.660 Get:5 http://mirrors.wikimedia.org/debian buster InRelease [122 kB]                                                                                                           
0.673 Err:2 https://packages.sury.org/php buster InRelease                                                                                                                          
0.673   The following signatures were invalid: EXPKEYSIG B188E2B695BD4743 DEB.SURY.ORG Automatic Signing Key <deb@sury.org>
1.098 Get:6 http://mirrors.wikimedia.org/debian buster-updates InRelease [56.6 kB]
1.164 Get:7 http://apt.wikimedia.org/wikimedia buster-wikimedia/main amd64 Packages [94.4 kB]
1.209 Ign:8 http://mirrors.wikimedia.org/debian buster-backports InRelease
1.319 Get:9 http://mirrors.wikimedia.org/debian buster/main amd64 Packages [7909 kB]
2.242 Err:10 http://mirrors.wikimedia.org/debian buster-backports Release
2.242   404  Not Found [IP: 208.80.154.139 80]
2.351 Get:11 http://mirrors.wikimedia.org/debian buster-updates/main amd64 Packages [8788 B]
3.092 Reading package lists...
3.644 W: GPG error: https://packages.sury.org/php buster InRelease: The following signatures were invalid: EXPKEYSIG B188E2B695BD4743 DEB.SURY.ORG Automatic Signing Key <deb@sury.org>
3.644 E: The repository 'https://packages.sury.org/php buster InRelease' is not signed.
3.644 E: The repository 'http://mirrors.wikimedia.org/debian buster-backports Release' does not have a Release file.
------
failed to solve: process "/bin/sh -c apt update &&    apt install -y sqlite3 php8.1-pcov wget vim" did not complete successfully: exit code: 100

What should have happened instead?:
It should complete successfully.

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia): master

Other information (browser name/version, screenshots, etc.):

There seem to be two things going on here: first, it seems that it's using an expired GPG key for sury (related: T360420). Second, there seem to be issues with our own packages; I've been having the sury issue for a while now, but the Wikimedia one seems new (related: T362648, T336680).

Event Timeline

Seems like two of the errors have fixed themselves in the meantime, and only the last one remains:

3.632 E: The repository 'http://mirrors.wikimedia.org/debian buster-backports Release' does not have a Release file.

Should we upgrade to bullseye/bookworm? At any rate, I'd say it's not ideal if our recommended development environment remains broken for 2 months.

Seems like that recently https://packages.sury.org/php/dists/buster/ is now returning a 403

Indeed. The errors I'm currently getting are:

root@85c5f4ab7234:/var/www/html/w# apt update
Hit:1 http://security.debian.org buster/updates InRelease
Err:2 https://packages.sury.org/php buster InRelease                                                                        
  403  Forbidden [IP: 169.150.247.36 443]
Hit:3 http://apt.wikimedia.org/wikimedia buster-wikimedia InRelease                                                         
Hit:4 http://mirrors.wikimedia.org/debian buster InRelease
Hit:5 http://mirrors.wikimedia.org/debian buster-updates InRelease
Ign:6 http://mirrors.wikimedia.org/debian buster-backports InRelease
Err:7 http://mirrors.wikimedia.org/debian buster-backports Release
  404  Not Found [IP: 208.80.154.139 80]
Reading package lists... Done
E: Failed to fetch https://packages.sury.org/php/dists/buster/InRelease  403  Forbidden [IP: 169.150.247.36 443]
E: The repository 'https://packages.sury.org/php buster InRelease' is no longer signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://mirrors.wikimedia.org/debian buster-backports Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

I think mw-docker should just be updated to use bookworm. I haven't been able to rebuild my local environment for the past ~4 months, needless to say how annoying it is.

Hi there, any updates on this task? I haven't been able to rebuild my local docker setup since March, despite needing to do that multiple times. I imagine I could maybe fiddle with the dockerfile locally to work around this issue, but I'm not quite comfortable doing it now, given that my environment is not entirely trivial, and I don't want to mess up with its current state (which is stale, but at least it works).

I really wish I didn't have to spend a whole sunday afternoon figuring this out and unbreaking our official development environment that's been broken for 8 months now. But such is life sometimes, and at least I learned something along the way.

Update: the MR above fixes the buster-backports issue. The next issue is that these images are buster-based and are using sury-php, but buster packages have been dropped from sury on July 1st, when buster became EOL. See for example T369146 for quibble images. So, we'd have to migrate to bookworm-based images (or bullseye if we really have to, but since this is for development environments only, it might be better to just do the big jump now while at it).

Yeah, of stuff still defined in the repo, I think it's probably time to drop everything prior to Bookworm and everything prior to PHP 7.4, then build Bookworm defaults. Working up a couple of MRs for that.