netbox: drop profile::netbox::active_server parameter
Open, MediumPublic
Actions

Assigned To

None

Authored By

	jbond
	May 23 2022, 3:29 PM

Description

With the most recent netbox upgrade we switched netbox to use conftool and DNS discovery for fail over as such we should remove profile::netbox::active_server from the puppet config. currently this parameter is used to enable a bunch of monitoring and systemd::timers. we should investigate if:

its safe to monitor both servers, probably yes?
if its ok to run the reports on both servers, likely no?

in both cases if we really do need to only set things up on the active host then we should use conftool to discover which one is the active node

Details

	Subject	Repo	Branch	Lines +/-
	hieradata: netbox1001 to specify netbox1002 as the active server.	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T309034 netbox: drop profile::netbox::active_server parameter
Resolved	ayounsi	T296452 Upgrade Netbox to 3.2
Stalled	None	T234997 Make Netbox Active/Active
Resolved	ayounsi	T311385 Netbox and Redis
Stalled	None	T330084 gdnsd failures when converting services from active/passive to active/active

Event Timeline

jbond created this task.May 23 2022, 3:29 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMay 23 2022, 3:29 PM

Yes I agree.

its safe to monitor both servers, probably yes?

Actually not really, the monitoring includes the reports monitoring that currently alerts in the dcops IRC channel. Those should probably run only on the active host.

if its ok to run the reports on both servers, likely no?

Not only those but also the ganeti sync systemd timers and such.

So I think we'll need to have that information somewhere.
The simplest thing to use would be the DNS, if the PTR of the IP returned by the discovery record is the FQDN of the host then it's the active one.
The other one is to have a confd template write a file in /etc/$NAME.

One approach could be to have Puppet deploy everything, but export resources only for the active one based on the DNS check above.
On the host the scripts should check the chosen method (either directly the dns or the file generated by confd) and act accordingly.

Volans triaged this task as Medium priority.May 23 2022, 3:52 PM

Middle/longer term the reports status should go through Prometheus so we could revisit at this point. Until then I agree with Riccardo.

As it's the same database, the Ganeti sync could run from both servers but for example as half their current frequency (similar to the Homer diff emails).

Change 805125 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] hieradata: netbox1001 to specify netbox1002 as the active server.

https://gerrit.wikimedia.org/r/805125

gerritbot added a project: Patch-For-Review.Jun 13 2022, 1:05 PM

Change 805125 merged by Jbond:

[operations/puppet@production] hieradata: netbox1001 to specify netbox1002 as the active server.

https://gerrit.wikimedia.org/r/805125

ayounsi removed a parent task: T296452: Upgrade Netbox to 3.2.Jun 20 2022, 4:05 PM

ayounsi added a subtask: T296452: Upgrade Netbox to 3.2.

ayounsi removed ayounsi as the assignee of this task.Jun 24 2022, 7:32 AM

ayounsi closed subtask T296452: Upgrade Netbox to 3.2 as Resolved.Jul 19 2022, 12:56 PM