apt: improve apt failover orchestration
Open, MediumPublic
Actions

Assigned To

None

Authored By

	jbond
	Mar 1 2023, 10:42 AM

Description

Some time ago apt was migrated to use active/passive DNS discovery records. this allows us to easily move the apt.wikimedia.org domain between sites. however there are still some aspects of what we consider the primary server to be controlled by puppet (see this change. We should investigate if we can leverage confd to to also control theses aspects.

A simple solution could be to writ a custom fact that interacts with confd to pull this information and feed it into puppet.

This task should also consider the work to move the apt serveres behind the caches (T317178)

We should also update the dc switch over wiki with any changes

Details

Subject	Repo	Branch	Lines +/-
confd::file: drop relative prefix	operations/puppet	production	+6 -15
P:confd: Add support for discovery facts	operations/puppet	production	+21 -2
profile::confd: add a confd profile	operations/puppet	production	+34 -22
P:aptrepo: use new wmflib::discovery::pooled_sites	operations/puppet	production	+18 -9

Customize query in gerrit

Related Objects

Mentioned In: T330843: reprepro uploads should trigger rsync apt job
Mentioned Here: T317178: Move apt to private IPs + CDN

Event Timeline

jbond triaged this task as Medium priority.Mar 1 2023, 10:42 AM

jbond created this task.

Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptMar 1 2023, 10:42 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

jbond updated the task description. (Show Details)Mar 1 2023, 11:28 AM

Change 893468 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] profile::confd: add a confd profile

https://gerrit.wikimedia.org/r/893468

gerritbot added a project: Patch-For-Review.Mar 1 2023, 12:38 PM

Change 893471 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] confd::file: drop relative prefix

https://gerrit.wikimedia.org/r/893471

We should find a standard setup for those use cases, I can see Netbox having exactly the same issue/requirement (some puppet-driver resources based on primary status).

Aklapper renamed this task from apt: improve apt failover ochastration to apt: improve apt failover orchestration.Mar 1 2023, 2:01 PM

Change 893496 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:confd: Add support for discovery facts

https://gerrit.wikimedia.org/r/893496

Change 893522 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] P:aptrepo: use new wmflib::discovery::pooled_sites

https://gerrit.wikimedia.org/r/893522

jbond mentioned this in T330843: reprepro uploads should trigger rsync apt job.Mar 2 2023, 9:48 AM

In T330849#8656648, @Volans wrote:

We should find a standard setup for those use cases, I can see Netbox having exactly the same issue/requirement (some puppet-driver resources based on primary status).

fyi this and the preceding patch is what i have as a proposal, however joe wants to chat about the use of conftool be fore we move forward

Change 893468 abandoned by Jbond:

[operations/puppet@production] profile::confd: add a confd profile

Reason:

this got done recently in a different change