Page MenuHomePhabricator

Sunsetting mirrors.wikimedia.org
Closed, ResolvedPublic

Description

We've been operating a Debian mirror since approx. ten years (it also hosts a mirror of Ubuntu, Tails and Debian Openstack Backports repository). This has historically been a great way to contribute back to our favourite OS. But Debian defaults to the deb.debian.org CDN since many years and that shrank the use of the traditional mirrors significantly.

Despite being part of the ftp.us.debian.org mirror rotation our mirror is hardly used these days, even despite idiotic AI scrapers hitting the server randomly and excessively, our bandwidth usage practically never exceeds 10% of the available bandwidth (and usually even less than that).

The current server is ending it's hardware lifespan, so in the current state it currently does not really make sense to refresh it, so we'll likely decommision it entirely.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Muehlenhoffoperations/puppetproduction+0 -7
Ahmon Dancyintegration/zuul/deploymaster+10 -1
Urbanecmoperations/dnsmaster+0 -1
Muehlenhoffoperations/puppetproduction+0 -197
Muehlenhoffoperations/puppetproduction+0 -1
Muehlenhoffoperations/alertsmaster+0 -49
Muehlenhoffoperations/puppetproduction+1 -1
Muehlenhoffoperations/puppetproduction+3 -2
Dzahnlabs/codesearchmaster+1 -1
Muehlenhoffoperations/puppetproduction+0 -29
Muehlenhoffoperations/puppetproduction+3 -0
Muehlenhoffoperations/puppetproduction+1 -1
Muehlenhoffoperations/puppetproduction+0 -90
Muehlenhoffoperations/puppetproduction+0 -5
Muehlenhoffoperations/puppetproduction+1 -4
Muehlenhoffoperations/puppetproduction+0 -1
Muehlenhoffoperations/puppetproduction+3 -7
Muehlenhoffoperations/puppetproduction+0 -11
Muehlenhoffoperations/puppetproduction+0 -40
Muehlenhoffoperations/puppetproduction+1 -1
Muehlenhoffoperations/puppetproduction+3 -56
Muehlenhoffoperations/puppetproduction+1 -0
Muehlenhoffoperations/alertsmaster+1 -1
Muehlenhoffoperations/puppetproduction+0 -24
Muehlenhoffoperations/puppetproduction+1 -8
Muehlenhoffoperations/puppetproduction+2 -2
Muehlenhoffoperations/puppetproduction+1 -1
Muehlenhoffoperations/puppetproduction+9 -9
Muehlenhoffoperations/puppetproduction+0 -40
Muehlenhoffoperations/puppetproduction+5 -20
Muehlenhoffoperations/puppetproduction+3 -3
Muehlenhoffoperations/puppetproduction+1 -1
Muehlenhoffoperations/puppetproduction+1 -4
Muehlenhoffoperations/puppetproduction+3 -0
Muehlenhoffoperations/puppetproduction+1 -1
Show related patches Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Dockerfile: Bump base bookworm image tag to 20260705repos/releng/buildkit!81dancywmf/build-Iea13c74c3373aeedaced6bb6f3d396f1676b0677wmf/build
Drop source list file referencing mirrors.wikimedia.orgrepos/releng/train-dev!182dancymain-I13237c52463b1d344622520960b0250a8d79496cmain
.pipeline/blubber.yaml: Bump bookworm base imagerepos/releng/gitlab-terraform-images!13dancydancy/bump-bookwormwmf/stable
Customize query in GitLab

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change #1295362 merged by Muehlenhoff:

[operations/puppet@production] package_builder: Switch to deb.debian.org

https://gerrit.wikimedia.org/r/1295362

Change #1295945 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] autoinstall: Switch to deb.debian.org

https://gerrit.wikimedia.org/r/1295945

Change #1295945 merged by Muehlenhoff:

[operations/puppet@production] autoinstall: Switch to deb.debian.org

https://gerrit.wikimedia.org/r/1295945

Change #1295956 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] autoinstall: Switch to deb.debian.org

https://gerrit.wikimedia.org/r/1295956

Change #1295956 merged by Muehlenhoff:

[operations/puppet@production] autoinstall: Switch to deb.debian.org

https://gerrit.wikimedia.org/r/1295956

Change #1294980 merged by Muehlenhoff:

[operations/puppet@production] mirrors: Disable osbpo sync

https://gerrit.wikimedia.org/r/1294980

Change #1301232 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mirrors: Removing remaining bits of osbpo mirror

https://gerrit.wikimedia.org/r/1301232

Change #1301232 merged by Muehlenhoff:

[operations/puppet@production] mirrors: Removing remaining bits of osbpo mirror

https://gerrit.wikimedia.org/r/1301232

Change #1301237 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/alerts@master] Stop monitoring the Ubuntu part of our mirror

https://gerrit.wikimedia.org/r/1301237

Change #1301238 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mirrors: Disable the NRPE check for the Ubuntu mirror

https://gerrit.wikimedia.org/r/1301238

Change #1301237 merged by Muehlenhoff:

[operations/alerts@master] Stop monitoring the Ubuntu part of our mirror

https://gerrit.wikimedia.org/r/1301237

Change #1301238 merged by Muehlenhoff:

[operations/puppet@production] mirrors: Disable the NRPE check for the Ubuntu mirror

https://gerrit.wikimedia.org/r/1301238

Change #1294284 merged by Muehlenhoff:

[operations/puppet@production] Retire the Ubuntu mirror

https://gerrit.wikimedia.org/r/1294284

Change #1302115 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Absent the NRPE mirror check

https://gerrit.wikimedia.org/r/1302115

Change #1302115 merged by Muehlenhoff:

[operations/puppet@production] Absent the NRPE mirror check

https://gerrit.wikimedia.org/r/1302115

Change #1302117 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove the remaining bits of the Ubuntu mirror

https://gerrit.wikimedia.org/r/1302117

Change #1302117 merged by Muehlenhoff:

[operations/puppet@production] Remove the remaining bits of the Ubuntu mirror

https://gerrit.wikimedia.org/r/1302117

I opened a ticket at https://anonticket.torproject.org/ and mirrors.wikimedia.org has been removed from the download mirror rotation for Tails.

Likewise osbpo and the Ubuntu mirror are moved by now, the last remaining part is the Debian mirror.

For future reference: it turned out that our osbpo mirror was one of two stores for those packcages, so we have set up a replacement mirror on cloud-vps. It will be maintained by zigo, the package maintainer, so doesn't fly in the face of this task IMO

Change #1302838 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Disable Debian mirror sync

https://gerrit.wikimedia.org/r/1302838

Change #1302858 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove the black box check for mirrors.wikimedia.org

https://gerrit.wikimedia.org/r/1302858

Change #1302858 merged by Muehlenhoff:

[operations/puppet@production] Remove the black box check for mirrors.wikimedia.org

https://gerrit.wikimedia.org/r/1302858

Change #1302838 merged by Muehlenhoff:

[operations/puppet@production] Disable Debian mirror sync

https://gerrit.wikimedia.org/r/1302838

Change #1303384 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mirrors: Fix SSH key config

https://gerrit.wikimedia.org/r/1303384

Change #1303384 merged by Muehlenhoff:

[operations/puppet@production] mirrors: Fix SSH key config

https://gerrit.wikimedia.org/r/1303384

Mentioned in SAL (#wikimedia-operations) [2026-06-17T11:01:30Z] <moritzm> The Debian mirror on mirrors.wikimedia.org has been disabled T416707

Change #1303386 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mirrors: Add a link to the announcement to the index page until full decom

https://gerrit.wikimedia.org/r/1303386

Change #1303386 merged by Muehlenhoff:

[operations/puppet@production] mirrors: Add a link to the announcement to the index page until full decom

https://gerrit.wikimedia.org/r/1303386

Change #1303391 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mirrors: Remove firewall rule for rsync access

https://gerrit.wikimedia.org/r/1303391

Change #1303391 merged by Muehlenhoff:

[operations/puppet@production] mirrors: Remove firewall rule for rsync access

https://gerrit.wikimedia.org/r/1303391

Change #1303396 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove Debian mirror code

https://gerrit.wikimedia.org/r/1303396

Change #1303396 merged by Muehlenhoff:

[operations/puppet@production] Remove Debian mirror code

https://gerrit.wikimedia.org/r/1303396

Change #1303405 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mirrors: Disable mirror_age_metrics metric

https://gerrit.wikimedia.org/r/1303405

Change #1303913 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] package_builder: Export environment variables for web proxy access

https://gerrit.wikimedia.org/r/1303913

Change #1303405 merged by Muehlenhoff:

[operations/puppet@production] mirrors: Disable mirror_age_metrics metric

https://gerrit.wikimedia.org/r/1303405

Change #1303913 merged by Muehlenhoff:

[operations/puppet@production] package_builder: Export environment variables for web proxy access

https://gerrit.wikimedia.org/r/1303913

Mentioned in SAL (#wikimedia-operations) [2026-06-18T08:01:17Z] <moritzm> regenerate pbuilder environments on build2002 to use deb.debian.org T416707

Mentioned in SAL (#wikimedia-operations) [2026-06-18T08:05:18Z] <moritzm> regenerate pbuilder environments on build2001 to use deb.debian.org T416707

Change #1303959 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mirrors: Remove obsolete users, groups and monitoring

https://gerrit.wikimedia.org/r/1303959

Change #1303959 merged by Muehlenhoff:

[operations/puppet@production] mirrors: Remove obsolete users, groups and monitoring

https://gerrit.wikimedia.org/r/1303959

Change #1304801 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mirrors: Remove rsync

https://gerrit.wikimedia.org/r/1304801

seeing CI errors on labs/codesearch that seem caused by this

#13 0.522 Err:3 http://mirrors.wikimedia.org/debian bullseye Release
#13 0.522   404  Not Found [IP: 208.80.154.139 80]
#13 0.849 Get:4 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main amd64 Packages [75.9 kB]
#13 0.898 Reading package lists...
#13 0.937 E: The repository 'http://mirrors.wikimedia.org/debian bullseye Release' does not have a Release file.

615

Change #1305248 had a related patch set uploaded (by Dzahn; author: Dzahn):

[labs/codesearch@master] update tag for the go image

https://gerrit.wikimedia.org/r/1305248

Change #1305248 merged by jenkins-bot:

[labs/codesearch@master] update tag for the go image

https://gerrit.wikimedia.org/r/1305248

Change #1304801 merged by Muehlenhoff:

[operations/puppet@production] mirrors: Remove rsync

https://gerrit.wikimedia.org/r/1304801

Change #1307176 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/alerts@master] Remove alerts for the mirror lag

https://gerrit.wikimedia.org/r/1307176

Change #1307180 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Move mirror1001 to insetup role for eventual decom

https://gerrit.wikimedia.org/r/1307180

Change #1307180 merged by Muehlenhoff:

[operations/puppet@production] Move mirror1001 to insetup role for eventual decom

https://gerrit.wikimedia.org/r/1307180

Change #1307176 merged by Muehlenhoff:

[operations/alerts@master] Remove alerts for the mirror lag

https://gerrit.wikimedia.org/r/1307176

cookbooks.sre.hosts.decommission executed by jmm@cumin2003 for hosts: mirror1001.wikimedia.org

  • mirror1001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB

Change #1307378 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove the acmechief config for mirrors.wikimedia.org

https://gerrit.wikimedia.org/r/1307378

Change #1307383 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove role:mirror and related Puppet classes

https://gerrit.wikimedia.org/r/1307383

Change #1307395 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove Cumin alias for mirrors

https://gerrit.wikimedia.org/r/1307395

Change #1307395 merged by Muehlenhoff:

[operations/puppet@production] Remove Cumin alias for mirrors

https://gerrit.wikimedia.org/r/1307395

MoritzMuehlenhoff claimed this task.

mirrors.wikimedia.org has been deregistered from all mirror sources, it's internal use in production servers and the container pipeline has been disabled and the hardware was decommissioned. Marking this resolved. There a a few open sub tasks around other former use of mirrors.wikimedia.org, which can be addressed independently of this task.

Change #1307383 merged by Muehlenhoff:

[operations/puppet@production] Remove role:mirror and related Puppet classes

https://gerrit.wikimedia.org/r/1307383

Change #1311584 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/dns@master] Remove mirrors.wikimedia.org

https://gerrit.wikimedia.org/r/1311584

Change #1311584 merged by Dzahn:

[operations/dns@master] Remove mirrors.wikimedia.org

https://gerrit.wikimedia.org/r/1311584

Change #1311933 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[integration/zuul/deploy@master] deploy: Repoint buster build at archive.debian.org

https://gerrit.wikimedia.org/r/1311933

Change #1311933 abandoned by Ahmon Dancy:

[integration/zuul/deploy@master] deploy: Repoint buster build at archive.debian.org

Reason:

Buster support not needed

https://gerrit.wikimedia.org/r/1311933