Page MenuHomePhabricator

Labs instances rely on unpuppetized firewall setup to connect to databases
Closed, ResolvedPublic

Description

Script to inject needed DNAT rules to connect to labsdb

After rebooting wikimetrics-dev1, the instance could not connect to the databases.

The reason was missing DNATs in firewall configuration.

I could not find this requirement documented, nor puppetized.

Is it documented somewhere?

If not ... let's puppetize DNAT rules (if they are not yet).

(Since I needed something right away, I wrote setup_dnat_rules.sh (see attachment), which contains the (somewhat redundant) DNAT rules from staging.
Maybe it helps someone else in the future)


Version: unspecified
Severity: normal

Attached:

Details

Reference
bz69042

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:30 AM
bzimport set Reference to bz69042.
bzimport added a subscriber: Unknown Object (MLST).
QChris created this task.Aug 2 2014, 10:10 AM
Nuria added a comment.Aug 5 2014, 9:31 AM

I do not remember us having to do this when we set up neither dev or staging when we fist set them up, which indicates that something might have changed on labs setup.

I do not think this should be a bug on our end but we should confirm with otto 1st whether this needs to be puppetized.

QChris added a comment.Aug 5 2014, 9:57 AM

(In reply to nuria from comment #1)

I do not remember us having to do this when we set up neither dev or staging
when we fist set them up, [...]

It might be a new thing or not. I do not know.

But we're fighting it once in a while:

  • at least on 2014-07-24 there was also need to do it [1],
  • when I rebooted a machine some days ago,
  • just now (2014-08-05), we again needed to reboot a machine.

Tim's bug that I linked above is from 2014-02-25.
So I doubt it's a new thing.

I do not think this should be a bug on our end [...]

I wanted a place to track it on our end.
And I wanted a place to put the DNAT script.
Hence, I filed it for us for now, and linked Tim's bug.

[1] http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-analytics/20140724.txt

[13:53:26] <milimetric> qchris: do you have any idea how to iptables-restore this: http://paste.ubuntu.com/7847723/
scfc added a comment.Aug 5 2014, 3:43 PM

AFAIK, the replica DB servers were never accessible under the enwiki.labsdb:$STANDARDPORT scheme without additions to /etc/hosts and iptables on the client side.

Automatic loading of iptables settings is getting implemented in

https://gerrit.wikimedia.org/r/#/c/156599/

Once that has been merged, the issue decreases to how to create
/etc/iptables.conf automatically.

coren closed this task as Resolved.Mar 6 2015, 6:22 PM
coren claimed this task.
coren added a subscriber: coren.

The iptables have been obsoleted some time ago as the replica databases were merged.