CloudVPS: consider mirroring debian repos for openstack packages
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	aborrero
	Nov 21 2019, 10:01 AM

Description

The Debian team behind the openstack packaging effort would be interested in someone helping with hosting/mirroring the .deb repositories.
They are trying to provide .deb packages for every openstack version starting from Kilo to the latest. This is an ongoing effort, some of the older releases are still missing, and newer ones too.

Example of some repos already running:

deb http://stretch-queens.debian.net/debian stretch-queens-backports main
deb http://stretch-queens.debian.net/debian stretch-queens-backports-nochanges main

deb http://stretch-rocky.debian.net/debian stretch-rocky-backports main
deb http://stretch-rocky.debian.net/debian stretch-rocky-backports-nochanges main

deb http://buster-stein.debian.net/debian buster-stein-backports main
deb http://buster-stein.debian.net/debian buster-stein-backports-nochanges main

deb http://buster-train.debian.net/debian buster-train-backports main
deb http://buster-train.debian.net/debian buster-train-backports-nochanges main

Technical details:

storage requriement should be something like ~4GB
a simple rsync should be enough
we can be provided an ansible script to set everything up, the setup should be fairly simple (if we have a dedicated VM for this)

We directly benefit from these repos, check for example: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Openstack_source#Version_matrix

If we decide we want this, I'm thinking that this could be done in our side either in Toolforge (a static webservice) or in CloudVPS in a VM with an external FQDN:

tools.wmflabs.org/openstack-debian-mirror
openstack-debian-mirror.wmflabs.org

Details

	Project	Branch	Lines +/-	Subject
	operations/puppet	production	+1 -1	mirrors: fix openstack rsync command
	operations/puppet	production	+31 -0	mirrors: mirror Debian openstack backports repositories

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Andrew	T241347 upgrade cloud-vps openstack to Openstack version 'Pike'
Resolved	Andrew	T241348 Upgrade cloudservices nodes to openstack Pike
Resolved	aborrero	T238820 CloudVPS: consider mirroring debian repos for openstack packages

Event Timeline

aborrero created this task.Nov 21 2019, 10:01 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 21 2019, 10:01 AM

aborrero renamed this task from CloudVPS: consider mirrong debian repos for openstack packages to CloudVPS: consider mirroring debian repos for openstack packages.Nov 21 2019, 10:02 AM

aborrero triaged this task as Low priority.

aborrero moved this task from Inbox to Needs discussion on the cloud-services-team (Kanban) board.

We already run a dedicated mirror host (mirrors.wikimedia.org, running on sodium) which currently mirrors Debian, Ubuntu and Tails (and which currently has 7.8 TB of free diskspace). I'm not sure if we have specific criteria for what we mirror, but it seems like a suitable candidate. Adding @faidon for comments.

We don't have specific criteria, it's on a case-by-case basis. This particular one sounds fine to me, let's do it! :)

There is scaffolding for mirrors in puppet for this, I'd use the Tails stuff as a template as that also uses a simple rsync.

Ideally they would introduce round robin DNS having our mirror in the rotation. That would mean sharing the debian/ dir with what we have in the mirror right now (which is vanilla debian). How do we feel about that?

/debian/ is for the official Debian mirror -- and note that we are part of the ftp.us.debian.org/http.us.debian.org rotation. So no, we should not pollute that namespace (and I don't think that would work anyway, ftpsync would just delete it all in the next sync).

They should really stop using /debian/, as this isn't really Debian but a a PPA. Especially if they want mirrors to pick this up :)

You are right. I'm in talks to change the directory.

Waiting for the folks in charge of the repo to rename the directory before enabling the mirror.

aborrero claimed this task.Dec 18 2019, 4:29 PM

Andrew mentioned this in T241347: upgrade cloud-vps openstack to Openstack version 'Pike'.Dec 23 2019, 8:30 AM

Andrew added a parent task: T241347: upgrade cloud-vps openstack to Openstack version 'Pike'.

Andrew added a parent task: T241348: Upgrade cloudservices nodes to openstack Pike.Dec 23 2019, 8:32 AM

It seems we have now this http://osbpo.debian.net/osbpo/ (no more debian/ dir)

Change 566284 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] mirrors: mirror Debian openstack backports repositories

https://gerrit.wikimedia.org/r/566284

gerritbot added a project: Patch-For-Review.Jan 21 2020, 1:49 PM

Change 566284 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] mirrors: mirror Debian openstack backports repositories

https://gerrit.wikimedia.org/r/566284

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2020, 2:10 PM

Mentioned in SAL (#wikimedia-operations) [2020-01-22T17:46:47Z] <arturo> forcing by hand the first sync on sodium for openstack packages (T238820)

Change 566694 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] mirrors: fix openstack rsync command

https://gerrit.wikimedia.org/r/566694

gerritbot added a project: Patch-For-Review.Jan 23 2020, 9:26 AM

Change 566694 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] mirrors: fix openstack rsync command

https://gerrit.wikimedia.org/r/566694

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2020, 10:10 AM

This is done. Please reopen if required.