Page MenuHomePhabricator
Create Task
Maniphest T148986

Firewall sets not being loaded post-reboot due to a @resolve race
Closed, ResolvedPublic
Actions

Description

I found three different, recently-rebooted hosts today (eeden, radon, es2015) that had no firewall loaded and only a WARNING alert for conntrack/sysctl.

This looks to me like a boot-time race between networking and the ferm init script, as it looks like ferm failed to load due to @resolve failing.

At minimum, we need a CRITICAL Icinga alert that is included in base::firewall and complains very very loudly if there is no firewall set loaded.

As for potential solutions:

  • Fixing the race, somehow.
  • Deprecating @resolve and falling-back to e.g. subnets/static mappings or Hiera or something.
  • Replacing @resolve with a puppet-time resolving of hostnames, via a custom parser function.

I'm setting this to UBN for security reasons until we have the aforementioned Icinga alert. After we have that, we can probably lower this to High.

Details

SubjectRepoBranchLines +/-
haproxy: Increase workaround for ferm starting too lateoperations/puppetproduction+1 -1
haproxy: Add workaround for ferm starting too lateoperations/puppetproduction+2 -0
Check whether ferm has been correctly startedoperations/puppetproduction+39 -0
Customize query in gerrit

Related Objects

Event Timeline

faidon created this task.Oct 24 2016, 4:17 PM
Restricted Application added subscribers: Jay8g, Luke081515, TerraCodes, Aklapper. · View Herald TranscriptOct 24 2016, 4:17 PM
MoritzMuehlenhoff added a comment.Oct 25 2016, 12:49 PM

I'll create an Icinga check. As a secondary step it's worth looking into backporting the systemd unit from stretch.

MoritzMuehlenhoff claimed this task.Oct 25 2016, 12:50 PM
Joe subscribed.Oct 25 2016, 4:09 PM

FTR, we already have a parser function called ipresolve() that mostly does what we need here.

gerritbot subscribed.Oct 28 2016, 12:23 PM

Change 318527 had a related patch set uploaded (by Muehlenhoff):
Check whether ferm has been correctly started

https://gerrit.wikimedia.org/r/318527

gerritbot added a project: Patch-For-Review.Oct 28 2016, 12:23 PM
faidon mentioned this in T136094: Race condition in setting net.netfilter.nf_conntrack_tcp_timeout_time_wait.Oct 28 2016, 3:12 PM
greg subscribed.Nov 11 2016, 12:53 AM

Obligatory UBN! priority check-in after 2.5 weeks. Is that prio still valid? Should this be prioritized within some team more highly? There's a related patch here and a couple more on the other task this was mentioned in that are not yet merged. (I haven't looked any deeper)

MoritzMuehlenhoff added a comment.Nov 11 2016, 7:52 AM

We already have monitoring for this (implicitly via the connection tracking Icinga check), but more explicit monitoring is under way via https://gerrit.wikimedia.org/r/#/c/318527/

gerritbot added a comment.Nov 14 2016, 3:27 PM

Change 318527 merged by Muehlenhoff:
Check whether ferm has been correctly started

https://gerrit.wikimedia.org/r/318527

MoritzMuehlenhoff lowered the priority of this task from Unbreak Now! to High.Nov 14 2016, 3:42 PM

We now have an Icinga check which tests whether ferm is loaded.

MoritzMuehlenhoff added a comment.Nov 14 2016, 3:45 PM

As for fixing the actual race; systemd.special(7) lists an nss-lookup.target, which looks promising. I'll test this.

fgiunchedi subscribed.May 31 2017, 4:33 PM

The @resolve "race" is also present in stretch, though more severe since ferm reliably fails when @resolve is used. See also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863802

MoritzMuehlenhoff added a comment.May 31 2017, 4:40 PM

The stretch issue is tracked separatetly via T166653

fgiunchedi renamed this task from Firewall sets not being loaded post-reboot due to a @resolve race to Firewall sets not being loaded post-reboot due to a @resolve race on jessie.May 31 2017, 5:19 PM
greg unsubscribed.May 31 2017, 5:36 PM
jcrespo subscribed.Dec 20 2017, 10:32 AM

We believe the workaround for T166653, resulting on ferm loading late also affects network state negatively, causing haproxy automatic restart to failover due to bad state of the firewall. We have to introduce a 10 second delay on haproxy systemd unit to workaround the network issues- proxy does not go back to the previous UP state to prevent flapping on a real scenario.

gerritbot added a comment.Dec 20 2017, 10:49 AM

Change 399377 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] haproxy: Add workaround for ferm starting too late

https://gerrit.wikimedia.org/r/399377

gerritbot added a comment.Dec 20 2017, 11:08 AM

Change 399377 merged by Jcrespo:
[operations/puppet@production] haproxy: Add workaround for ferm starting too late

https://gerrit.wikimedia.org/r/399377

gerritbot added a comment.Dec 20 2017, 12:23 PM

Change 399388 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] haproxy: Increase workaround for ferm starting too late

https://gerrit.wikimedia.org/r/399388

gerritbot added a comment.Dec 20 2017, 12:27 PM

Change 399388 merged by Jcrespo:
[operations/puppet@production] haproxy: Increase workaround for ferm starting too late

https://gerrit.wikimedia.org/r/399388

jijiki merged a task: T207417: ferm fail to start at boot in some cases.Oct 25 2018, 11:36 AM
jijiki added a subscriber: Volans.
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 8:53 PM
fgiunchedi closed this task as Resolved.Apr 14 2020, 3:32 PM

Boldly resolving since Jessie is deprecated

MoritzMuehlenhoff reopened this task as Open.Apr 14 2020, 4:11 PM

I think we've still seen these on Stretch, let's keep this open until we're sure that this got fixed in Buster.

MoritzMuehlenhoff renamed this task from Firewall sets not being loaded post-reboot due to a @resolve race on jessie to Firewall sets not being loaded post-reboot due to a @resolve race.Apr 14 2020, 4:11 PM
jcrespo added a comment.Apr 14 2020, 4:55 PM

Also this would need to be reverted: https://phabricator.wikimedia.org/T148986#3850836 before closing the ticket, but I don't think we have a proxy with buster yet to test it.

TerraCodes unsubscribed.Apr 15 2020, 12:09 PM
Aklapper removed a project: Patch-For-Review.Oct 14 2020, 7:26 PM
MoritzMuehlenhoff closed this task as Resolved.Oct 29 2020, 10:28 AM

We haven't seen these for a while to be a general problem. Also, there's monitoring in place, so if it happens again we can revisit specific cases. Closing.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL