cloudcontrol: review connectivity with backup system
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	aborrero
	Jun 5 2023, 10:37 AM

Description

Following the network rework in the parent task, cloudcontrols aren't getting backups:

P48707 network or service issue on cloud-dev

1	04-Jun 02:08 backup1001.eqiad.wmnet JobId 513205: Fatal error: bsockcore.c:208 Unable to connect to Client: cloudcontrol2004-dev.codfw.wmnet-fd on cloudcontrol2004-dev.codfw.wmnet:9102. ERR=Interrupted system call
2	04-Jun 02:08 backup1001.eqiad.wmnet JobId 513205: Fatal error: No Job status returned from FD.
3	04-Jun 02:08 backup1001.eqiad.wmnet JobId 513205: Error: Bacula backup1001.eqiad.wmnet 9.6.7 (10Dec20)

This resulted in the hosts being depooled from the backup system monitoring for now see https://gerrit.wikimedia.org/r/c/operations/puppet/+/927119 for example.

There are likely firewalling / routing considerations here.

Details

	Subject	Repo	Branch	Lines +/-
	backup: Revert the addition of codfw-dev cloud hosts to the ignore list	operations/puppet	production	+0 -3
	cr-labs: Permit bacula backup traffic	operations/homer/public	master	+15 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	aborrero	T296411 cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet
Resolved	aborrero	T297596 have cloud hardware servers in the cloud realm using a dedicated LB layer
Resolved	aborrero	T324992 cloudlb: create PoC on codfw
Resolved	aborrero	T332153 cloudlb PoC: prepare backends
Resolved	aborrero	T338132 cloudcontrol: review connectivity with backup system

Event Timeline

aborrero triaged this task as Low priority.Jun 5 2023, 10:37 AM

aborrero created this task.

aborrero added a project: User-aborrero.

• jcrespo subscribed.Jun 5 2023, 10:39 AM

aborrero removed a subscriber: Papaul.Jun 5 2023, 10:40 AM

@aborrero: I would start by some sanity checks- confirming it is not just the puppet profile being absent, or the systemd daemon failing, before going to network config (as this was a common case super easy to check)

aborrero mentioned this in T339894: cloudservices: codfw1dev: fix backups.Jun 20 2023, 10:47 AM

Change 931576 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/homer/public@master] cr-labs: Permit bacula backup traffic

https://gerrit.wikimedia.org/r/931576

gerritbot added a project: Patch-For-Review.Jun 20 2023, 10:48 AM

Change 931576 merged by Arturo Borrero Gonzalez:

[operations/homer/public@master] cr-labs: Permit bacula backup traffic

https://gerrit.wikimedia.org/r/931576

Maintenance_bot removed a project: Patch-For-Review.Jun 20 2023, 2:29 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-20T14:36:19Z] <arturo> homer run for CR eqiad/codfw to allow bacula traffic in from cloud-hosts (T338132, T339894)

Could you please try again?

aborrero reassigned this task from aborrero to • jcrespo.Jun 20 2023, 2:37 PM

It worked now:

Terminated Jobs:
 JobId  Level      Files    Bytes   Status   Finished        Name
515898  Full          16    54.00 M  OK       20-Jun-23 15:34 cloudcontrol2001-dev.codfw.wmnet-Monthly-1st-Wed-productionEqiad-mysql-srv-backups
515899  Full           3    6.316 M  OK       20-Jun-23 15:35 cloudservices2004-dev.codfw.wmnet-Monthly-1st-Fri-productionEqiad-openldap

I will revert the monitoring hiding.

Change 931634 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] backup: Revert the addition of codfw-dev cloud hosts to the ignore list

https://gerrit.wikimedia.org/r/931634

gerritbot added a project: Patch-For-Review.Jun 20 2023, 3:44 PM

• jcrespo added projects: Data-Persistence-Backup, bacula.Jun 20 2023, 3:44 PM

Change 931634 merged by Jcrespo:

[operations/puppet@production] backup: Revert the addition of codfw-dev cloud hosts to the ignore list

https://gerrit.wikimedia.org/r/931634

Thanks!

• jcrespo reassigned this task from • jcrespo to aborrero.Jun 20 2023, 4:04 PM

Maintenance_bot removed a project: Patch-For-Review.Jun 20 2023, 4:11 PM