Maniphest T206951

Puppet doesn't restart ferm on failure
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Volans
	Oct 14 2018, 5:44 PM

Description

After the reboot of some eqsin hosts as described in the parent task, ferm failed to start on bast5001 and dns5001 due to a DNS lookup failure. The problem is that Puppet didn't restart it in any of the following runs, I had to manually do sudo systemctl start ferm and as a result the 2 hosts have been for ~1h without ferm rules applied.

We should improve this behaviour to ensure to minimize the potential exposure of a host without ferm rules applied on reboot, if ferm fails to start for any transient reason.

Details

Subject	Repo	Branch	Lines +/-
sre.wdqs.data-transfer: fix syntax, simplify rule	operations/cookbooks	master	+1 -2
sre.wdqs.data-transfer: use proper systemctl path	operations/cookbooks	master	+4 -4
sre.wdqs.data-transfer: fix syntax, simplify rule	operations/cookbooks	master	+1 -2
sre.wdqs.data-transfer: manage ferm rules required for transfer	operations/cookbooks	master	+33 -6
ferm: enable ferm status script	operations/puppet	production	+5 -2
ferm: Add status check	operations/puppet	production	+179 -1
ferm: add a very basic status check	operations/puppet	production	+23 -3
ferm_status: use ip_network to normalise src and dst addresses	operations/puppet	production	+3 -9
ferm_status: store each chain in its own hash	operations/puppet	production	+3 -2

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		ayounsi	T206861 Power incident in eqsin
		Resolved		jbond	T206951 Puppet doesn't restart ferm on failure

Event Timeline

Volans triaged this task as Medium priority.Oct 14 2018, 5:44 PM

Volans created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 14 2018, 5:44 PM

Paladox subscribed.Oct 14 2018, 6:41 PM

• ema moved this task from Backlog to General on the Traffic board.Oct 22 2018, 8:37 AM

BBlack mentioned this in T206861: Power incident in eqsin.Nov 27 2018, 7:27 PM

greg added a project: Wikimedia-Incident.Nov 27 2018, 7:28 PM

greg moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:30 PM

I don't know if this is related but today I noticed that, if iptables rules are cleared (iptables -F), subsequent puppet runs will not re-apply them. I also had to run systemctl restart ferm to get them re-applied.

Dzahn subscribed.Feb 9 2019, 1:11 AM

elukey subscribed.Feb 9 2019, 2:16 PM

• GTirloni unsubscribed.Mar 21 2019, 9:11 PM

Change 573335 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] ferm: add a very basic status check

https://gerrit.wikimedia.org/r/573335

gerritbot added a project: Patch-For-Review.Feb 19 2020, 5:05 PM

Change 576101 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] ferm: Add status check

https://gerrit.wikimedia.org/r/576101

Change 576102 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] ferm: enable ferm status script

https://gerrit.wikimedia.org/r/576102

Change 589289 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/cookbooks@master] sre.wdqs.data-transfer: manage ferm rules required for transfer

https://gerrit.wikimedia.org/r/589289

Change 576101 merged by Jbond:
[operations/puppet@production] ferm: Add status check

https://gerrit.wikimedia.org/r/576101

Change 591036 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] ferm_status: store each chain in its own hash

https://gerrit.wikimedia.org/r/591036

Change 591036 merged by Jbond:
[operations/puppet@production] ferm_status: store each chain in its own hash

https://gerrit.wikimedia.org/r/591036

Change 591049 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] ferm_status: use ip_network to normalise src and dst addresses

https://gerrit.wikimedia.org/r/591049

Change 591049 merged by Jbond:
[operations/puppet@production] ferm_status: use ip_network to normalise src and dst addresses

https://gerrit.wikimedia.org/r/591049

Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM

Change 573335 abandoned by Jbond:
ferm: add a very basic status check

Reason:
we now have ferm-status.py

https://gerrit.wikimedia.org/r/573335

Change 576102 merged by Jbond:
[operations/puppet@production] ferm: enable ferm status script

https://gerrit.wikimedia.org/r/576102

Change 589289 merged by Ryan Kemper:
[operations/cookbooks@master] sre.wdqs.data-transfer: manage ferm rules required for transfer

https://gerrit.wikimedia.org/r/589289

Maintenance_bot removed a project: Patch-For-Review.May 7 2020, 7:10 PM

Change 595061 had a related patch set uploaded (by Ryan Kemper; owner: Ryan Kemper):
[operations/cookbooks@master] sre.wdqs.data-transfer: fix missing commas

https://gerrit.wikimedia.org/r/595061

gerritbot added a project: Patch-For-Review.May 7 2020, 10:35 PM

Change 595649 had a related patch set uploaded (by Ryan Kemper; owner: Ryan Kemper):
[operations/cookbooks@master] sre.wdqs.data-transfer: fix syntax, simplify rule

https://gerrit.wikimedia.org/r/595649

Change 595061 merged by Gehel:
[operations/cookbooks@master] sre.wdqs.data-transfer: fix syntax, simplify rule

https://gerrit.wikimedia.org/r/595061

Change 596073 had a related patch set uploaded (by Ryan Kemper; owner: Ryan Kemper):
[operations/cookbooks@master] sre.wdqs.data-transfer: use proper systemctl path

https://gerrit.wikimedia.org/r/596073

Change 596073 merged by Ryan Kemper:
[operations/cookbooks@master] sre.wdqs.data-transfer: use proper systemctl path

https://gerrit.wikimedia.org/r/596073

Change 595649 abandoned by Ryan Kemper:
sre.wdqs.data-transfer: fix syntax, simplify rule

Reason:
this is a "duplicate", a different gerrit patch of mine fixed this

https://gerrit.wikimedia.org/r/595649

Maintenance_bot removed a project: Patch-For-Review.Jun 2 2020, 12:10 AM

ayounsi unsubscribed.Jun 2 2020, 5:46 AM

Dzahn unsubscribed.Jun 2 2020, 6:42 AM

BBlack moved this task from General to Bug Reports on the Traffic board.Sep 29 2020, 8:37 PM

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!