Page MenuHomePhabricator

Too many connection errors in mysql
Closed, ResolvedPublic


Recently, the deployment-prep cluster is experiencing many technical failures with regards to the MySQL server (I believe?). A sample output error:

Can't contact the database server: Host 'i-0000026f.pmtpa.wmflabs' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts' (deployment-sql)

This blocks *everyone* (including bots) from accessing the beta cluster, and defeats the purpose of being an alternative beta cluster.

Please look into this issue. Its a blocker.

Version: unspecified
Severity: blocker



Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 12:26 AM
bzimport set Reference to bz37173.
bzimport added a subscriber: Unknown Object (MLST).

Someone spammed the beta cluster which caused the apaches to generate a ton of connections which got them blacklisted.

I ran FLUSH HOSTS; and raised max_connect_errors;

For the record, the incident occurred between 2012/05/28 21:45 UTC and roughly 2012/05/29 02:00 UTC. Every apaches show a nice CPU spike.

I have banned all public IP ranges from Amazon Elastic Cloud. That will prevents their crawlers to hammer our servers.

How did you perform the ban?

squid config, apache config, iptables, rules from labsconsole... ?

From squid, see /etc/squid/squid.conf on deployment-squid.

I have just fixed the config right now.