Page MenuHomePhabricator

CloudVPS: consider mirroring debian repos for openstack packages
Open, LowPublic

Description

The Debian team behind the openstack packaging effort would be interested in someone helping with hosting/mirroring the .deb repositories.
They are trying to provide .deb packages for every openstack version starting from Kilo to the latest. This is an ongoing effort, some of the older releases are still missing, and newer ones too.

Example of some repos already running:

deb http://stretch-queens.debian.net/debian stretch-queens-backports main
deb http://stretch-queens.debian.net/debian stretch-queens-backports-nochanges main

deb http://stretch-rocky.debian.net/debian stretch-rocky-backports main
deb http://stretch-rocky.debian.net/debian stretch-rocky-backports-nochanges main

deb http://buster-stein.debian.net/debian buster-stein-backports main
deb http://buster-stein.debian.net/debian buster-stein-backports-nochanges main

deb http://buster-train.debian.net/debian buster-train-backports main
deb http://buster-train.debian.net/debian buster-train-backports-nochanges main

See also https://wiki.debian.org/OpenStack and also http://osbpo.debian.net/debian/pool/

Technical details:

  • storage requriement should be something like ~4GB
  • a simple rsync should be enough
  • we can be provided an ansible script to set everything up, the setup should be fairly simple (if we have a dedicated VM for this)

We directly benefit from these repos, check for example: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Openstack_source#Version_matrix

If we decide we want this, I'm thinking that this could be done in our side either in Toolforge (a static webservice) or in CloudVPS in a VM with an external FQDN:

  • tools.wmflabs.org/openstack-debian-mirror
  • openstack-debian-mirror.wmflabs.org

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 21 2019, 10:01 AM
aborrero renamed this task from CloudVPS: consider mirrong debian repos for openstack packages to CloudVPS: consider mirroring debian repos for openstack packages.Nov 21 2019, 10:02 AM
aborrero triaged this task as Low priority.
aborrero moved this task from Inbox to Needs discussion on the cloud-services-team (Kanban) board.

We already run a dedicated mirror host (mirrors.wikimedia.org, running on sodium) which currently mirrors Debian, Ubuntu and Tails (and which currently has 7.8 TB of free diskspace). I'm not sure if we have specific criteria for what we mirror, but it seems like a suitable candidate. Adding @faidon for comments.

We don't have specific criteria, it's on a case-by-case basis. This particular one sounds fine to me, let's do it! :)

There is scaffolding for mirrors in puppet for this, I'd use the Tails stuff as a template as that also uses a simple rsync.

Ideally they would introduce round robin DNS having our mirror in the rotation. That would mean sharing the debian/ dir with what we have in the mirror right now (which is vanilla debian). How do we feel about that?

/debian/ is for the official Debian mirror -- and note that we are part of the ftp.us.debian.org/http.us.debian.org rotation. So no, we should not pollute that namespace (and I don't think that would work anyway, ftpsync would just delete it all in the next sync).

They should really stop using /debian/, as this isn't really Debian but a a PPA. Especially if they want mirrors to pick this up :)

You are right. I'm in talks to change the directory.

Waiting for the folks in charge of the repo to rename the directory before enabling the mirror.

aborrero claimed this task.Dec 18 2019, 4:29 PM

It seems we have now this http://osbpo.debian.net/osbpo/ (no more debian/ dir)

Change 566284 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] mirrors: mirror Debian openstack backports repositories

https://gerrit.wikimedia.org/r/566284

Change 566284 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] mirrors: mirror Debian openstack backports repositories

https://gerrit.wikimedia.org/r/566284

Mentioned in SAL (#wikimedia-operations) [2020-01-22T17:46:47Z] <arturo> forcing by hand the first sync on sodium for openstack packages (T238820)

Change 566694 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] mirrors: fix openstack rsync command

https://gerrit.wikimedia.org/r/566694

Change 566694 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] mirrors: fix openstack rsync command

https://gerrit.wikimedia.org/r/566694