Page MenuHomePhabricator

pc2007.codfw.wmnet network blip?
Closed, ResolvedPublic

Description

Hello,

This is from last night

01:47 <+icinga-wm> PROBLEM - Check systemd state on pc2007 is CRITICAL: connect to address 10.192.0.104 port 5666: Connection refused
01:47 <+icinga-wm> PROBLEM - puppet last run on pc2007 is CRITICAL: connect to address 10.192.0.104 port 5666: Connection refused
01:47 <+icinga-wm> PROBLEM - Check size of conntrack table on pc2007 is CRITICAL: connect to address 10.192.0.104 port 5666: Connection refused
01:47 <+icinga-wm> PROBLEM - MariaDB disk space on pc2007 is CRITICAL: connect to address 10.192.0.104 port 5666: Connection refused
01:47 <+icinga-wm> PROBLEM - Check whether ferm is active by checking the default input chain on pc2007 is CRITICAL: connect to address 10.192.0.104 port 5666: Connection refused
01:47 <+icinga-wm> PROBLEM - mysqld processes on pc2007 is CRITICAL: connect to address 10.192.0.104 port 5666: Connection refused
01:47 <+icinga-wm> PROBLEM - MariaDB Slave IO: pc1 on pc2007 is CRITICAL: connect to address 10.192.0.104 port 5666: Connection refused
01:47 <+icinga-wm> PROBLEM - MariaDB Slave SQL: pc1 on pc2007 is CRITICAL: connect to address 10.192.0.104 port 5666: Connection refused
01:47 <+icinga-wm> PROBLEM - dhclient process on pc2007 is CRITICAL: connect to address 10.192.0.104 port 5666: Connection refused

Looks like it had a network blip?
I cannot find anything on the logs related to its interface, maybe there are some logs on the switch side?

Event Timeline

See IRC/internal channel, during the setup of logstash2001.codfw.wmnet it accidentally reused the 10.192.0.104 A record.

See IRC/internal channel, during the setup of logstash2001.codfw.wmnet it accidentally reused the 10.192.0.104 A record.

Ah! Thank you :-)
Mystery solved