Ferm: leftovers on hosts were it was enabled and then removed
Closed, ResolvedPublic

Description

While testing Cumin I discovered an inconsistent state on dbproxy1011, where there are some leftovers of ferm that probably was enabled in the past and was removed at some point. The same happened on multatuli (where it was fixed re-adding the firewall).

It seems that the role applied to dbproxy1011 (::role::mariadb::proxy::master) includes role::mariadb::ferm that in turn defines a ferm::service and a ferm::rule resources that in turn define a virtual file resource each that requires the File['/etc/ferm/conf.d'] resource that in turn requires the Package['ferm'] resource.
But those hosts don't seems to use the ferm class resource, that is the one that realizes the virtual resources into concrete ones, hence do not trigger all this chain of events.

Other dbproxies that I've checked don't have the ferm module installed at all and also looking at a puppet-compiler result for both dbproxy1010 and dbproxy1011 they don't show any ferm package installation, although iptable_filter kernel module is loaded on both of them.

I'm then assuming that the presence of ferm on dbproxy1011 is a leftover of a previous configuration that was including the ferm class that was reverted at some point.

If this is the case, I think that it shows a problem in our firewall configuration because leaving stale applied iptables rules might have weird and unexpected issues. Maybe we should consider or to have a rule to reimage any host that goes from having a firewall to not having it, or to ensure that if not used, the whole firewall dependencies are cleared (ferm pacakge, iptables loaded modules, etc.).

Thoughts?

Volans created this task.Feb 22 2017, 7:41 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 22 2017, 7:41 PM

The dbproxy hosts don't use base::firewall yet, we talked about that in the past, but haven't scheduled a date yet. But that's hardly limited to the ferm classes, almost all other parts of our puppet code keeps things around if a role of a host is changed (starting with installed Debian packages and also conf files), the only clean to switch to a different role without manual cleanups is to reimage.

@jcrespo @Marostegui are you ok with the manual removal of any ferm rule and manual restore of a clean iptables table on dbproxy1011?

Sure. Even if it breaks something, it is considered beta. You can give it a check and compare it to dbproxy1010, it should have the same rules.

Mentioned in SAL (#wikimedia-operations) [2017-02-24T16:55:15Z] <volans> manually cleaning ferm leftovers on dbproxy1011 - T158798

Volans closed this task as Resolved.Feb 24 2017, 5:20 PM
Volans claimed this task.

Cleanup completed and all looks good so far. Resolving