labs security rules are flappy / invalid cause network communications issues
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Feb 9 2015, 9:14 AM

Description

Since Friday Feb 6th 23:30 UTC, puppet runs on instance of the integration labs project started failing. The instances puppet agent point to integration-puppetmaster.eqiad.wmflabs but timeout connecting to it. The puppet agent local to integration-puppetmaster works fine though.

On one of the instance:

Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Connection timed out - connect(2)
Info: Retrieving plugin
...
Error: Could not retrieve catalog from remote server: Connection timed out - connect(2)
Notice: Using cached catalog
Info: Applying configuration version '1423261026'

Additionally the Jenkins master on gallium ( 208.80.154.135 ) is no more able to ssh to integration-slave1001.eqiad.wmflabs ( 10.68.16.60 ) after I rebooted the instance.

gallium:~$ telnet 10.68.16.60 22
Trying 10.68.16.60...
telnet: Unable to connect to remote host: Connection timed out

Though I can ssh to it from the labs bastion. The instance has some iptables rules but they allow connections from gallium on port 22 (ssh).

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		coren	T88960 labs security rules are flappy / invalid cause network communications issues
		Resolved		hashar	T88972 Bring integration-slave1001.eqiad.wmflabs back in the pool

Event Timeline

hashar created this task.Feb 9 2015, 9:14 AM

hashar raised the priority of this task from to High.

hashar updated the task description. (Show Details)

hashar added a project: Continuous-Integration-Infrastructure.

hashar subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 9 2015, 9:14 AM

I have stopped the puppet master on integration-puppetmaster and started listening for connections using netcat -l 8140. From integration-slave1002 I tried to telnet 10.68.16.96 8140 and have a connection timeout.

Seems there is some firewall rule in between that prevents network communications between the instances and the puppetmaster. I can't find any specific iptables rule that could cause the problem.

Maybe ops have some idea? :-(

hashar renamed this task from integration-puppetmaster does not respond to other instances to labs security rules are flappy / invalid cause network communications issues.Feb 9 2015, 10:27 AM

hashar updated the task description. (Show Details)

hashar set Security to None.

hashar mentioned this in T88972: Bring integration-slave1001.eqiad.wmflabs back in the pool.Feb 9 2015, 10:29 AM

The project security group did not (was changed not to?) include allowing ssh from gallium. Fixing that did the trick.

hashar closed subtask T88972: Bring integration-slave1001.eqiad.wmflabs back in the pool as Resolved.Feb 9 2015, 3:54 PM

The integration labs project was missing a security rule to allow ssh from gallium for some reason. I have added it back: allow from gallium to port 22 (ssh) and that fixed the connection issue to integration-slave1001.eqiad.wmflabs

The puppetmaster issue must be related though I have yet to figure out the reason.

The puppetmaster issue did appear related: adding an explcit rule to allow it fixed the immediate problem, but seems to point at something having changed about the applied defaults. This will need further investigation.

coren closed this task as Resolved.Feb 9 2015, 4:07 PM

The deployment-prep labs project also uses a local puppetmaster but it does not need any specific security rule to allow puppet connections (port 8140).

@hashar
root@integration-slave1002:~# telnet 10.68.16.96 8140
Trying 10.68.16.96...
Connected to 10.68.16.96.
Escape character is '^]'.

I assume everything is OK ? (or at least some my testing says)

Yeah, things are working fine now with an explicit rule - but the necessity of having the explicit rule seems to be new.

Filled another ticket for investigation of the underlying issue: T88995: Labs security rules changed on integration labs project around Friday Feb 6th 23:30 UTC.

hashar moved this task from Untriaged to Done on the Continuous-Integration-Infrastructure board.Feb 9 2015, 4:18 PM

The doc publishing jobs are failing as well and there is no workaround for it :( T89026

Aklapper added a project: Cloud-Services.Apr 29 2015, 12:20 PM

labs security rules are flappy / invalid cause network communications issuesClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

labs security rules are flappy / invalid cause network communications issues
Closed, ResolvedPublic
Actions

Related Objects
Search...