Improve external storage MySQL configuration for connection spikes
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Volans
	Apr 21 2016, 9:52 AM

Description

To mitigate overload during switchover and in case of memcached failures improving the connections and thread pool.

Details

	Subject	Repo	Branch	Lines +/-
	MariaDB: separate external storage my.cnf	operations/puppet	production	+171 -1
	Add parameter to specify larger threadpools	operations/puppet/mariadb	master	+11 -10

Customize query in gerrit

Event Timeline

Volans created this task.Apr 21 2016, 9:52 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 21 2016, 9:52 AM

Change 284662 had a related patch set uploaded (by Volans):
Add parameter to specify larger threadpools

https://gerrit.wikimedia.org/r/284662

gerritbot added a project: Patch-For-Review.Apr 21 2016, 9:53 AM

Change 284666 had a related patch set uploaded (by Volans):
MariaDB: separate external storage my.cnf

https://gerrit.wikimedia.org/r/284666

Change 284662 abandoned by Volans:
Add parameter to specify larger threadpools

https://gerrit.wikimedia.org/r/284662

Change 284666 merged by Volans:
MariaDB: separate external storage my.cnf

https://gerrit.wikimedia.org/r/284666

Executed successfully (and verified) on active ES DBs in eqiad:

sudo salt 'es101*' cmd.run 'mysql --defaults-file=/root/.my.cnf -A -BN -e"SET GLOBAL thread_pool_size=40"'
sudo salt 'es101*' cmd.run 'mysql --defaults-file=/root/.my.cnf -A -BN -e"SET GLOBAL thread_pool_max_threads=2000"'
sudo salt 'es101*' cmd.run 'mysql --defaults-file=/root/.my.cnf -A -BN -e"SET GLOBAL max_connections=10000"'

Executed successfully:

sudo salt -E 'es201[1-9]' cmd.run 'mysql --defaults-file=/root/.my.cnf -A -BN -e"SET GLOBAL thread_pool_size=40"'
sudo salt -E 'es201[1-9]' cmd.run 'mysql --defaults-file=/root/.my.cnf -A -BN -e"SET GLOBAL thread_pool_max_threads=2000"'
sudo salt -E 'es201[1-9]' cmd.run 'mysql --defaults-file=/root/.my.cnf -A -BN -e"SET GLOBAL max_connections=10000"'

Improve external storage MySQL configuration for connection spikesClosed, ResolvedPublicActions

Description

Details

Event Timeline

Improve external storage MySQL configuration for connection spikes
Closed, ResolvedPublic
Actions