Page MenuHomePhabricator

tools.wikiloves (s52927) overloading labsdb1003 and making it lag
Closed, ResolvedPublic

Description

Hey,

This is the second time I see s52927 creating the following amount of heavy queries:

[root@labsdb1003 10:49 /root]
# mysql --skip-ssl -e "show processlist;" | grep s52927
114593256	s52927	10.68.18.9:45753	commonswiki_p	Execute	71707	Queried about 322170000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
114725576	s52927	10.68.18.208:49900	commonswiki_p	Execute	68099	Queried about 264350000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
114882051	s52927	10.68.20.54:35109	commonswiki_p	Execute	64507	Queried about 222660000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
114999634	s52927	10.68.17.202:42612	commonswiki_p	Execute	60905	Queried about 191790000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
115144355	s52927	10.68.23.166:34751	commonswiki_p	Execute	57299	Queried about 169340000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
115264646	s52927	10.68.18.208:57117	commonswiki_p	Execute	53699	Queried about 138620000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
115366183	s52927	10.68.18.137:48723	commonswiki_p	Execute	50099	Queried about 124360000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
115469429	s52927	10.68.16.94:46145	commonswiki_p	Execute	46506	Queried about 104130000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
115569847	s52927	10.68.17.202:59277	commonswiki_p	Execute	42907	Queried about 90180000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
115667383	s52927	10.68.23.223:39938	commonswiki_p	Execute	39298	Queried about 79360000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
115763672	s52927	10.68.18.83:50577	commonswiki_p	Execute	35699	Queried about 68910000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
115863568	s52927	10.68.18.12:42182	commonswiki_p	Execute	32098	Queried about 60800000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
115957039	s52927	10.68.23.223:57951	commonswiki_p	Execute	28507	Queried about 52030000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
116052871	s52927	10.68.18.208:59466	commonswiki_p	Execute	24897	Queried about 46100000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
116143364	s52927	10.68.18.12:45166	commonswiki_p	Execute	21307	Queried about 36610000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
116218246	s52927	10.68.18.12:35570	commonswiki_p	Execute	17707	Queried about 31650000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
116298221	s52927	10.68.19.235:37034	commonswiki_p	Execute	14099	Queried about 23740000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
116385305	s52927	10.68.18.83:40500	commonswiki_p	Execute	10507	Queried about 17880000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
116486551	s52927	10.68.23.223:39153	commonswiki_p	Execute	6907	Queried about 12630000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000
116580598	s52927	10.68.21.169:35281	commonswiki_p	Execute	3307	Queried about 7450000 rows	SELECT\n img_timestamp,\n img_name IN (SELECT DISTINCT gil_to FROM globalimagelinks) AS image_in_use,\n	0.000

The first time I had to kill them as labsdb1003 was lagging around 7h behind.
I will keep an eye on how they go and if needed I will need to kill them again + set a limit of connections for this user as it is affecting other labsdb1003 users (who should migrate to the new hosts anyways)

Event Timeline

This user has now 68 threads running. Exactly the same heavy queries. The server is lagging for 3h now. I am going to kill them and set a limit of 10 connections to avoid disturbing other users during the weekend.

# mysql --skip-ssl -e "show full processlist;" | grep s52927 | wc -l
68

Hey, I believe I’m s52927 (I inherited Tool-wikiloves last year). Thanks @jcrespo for flagging this to me.

First off, apologies for making a mess (I gather I made a mess ^_^"). While Wiki Loves Monuments was ongoing I increased the cron to run these queries *very* often (every 10 minutes I believe) − I have now reduced it to run every 12 hours, which is plenty enough for the tool functionality, and hopefully will relieve this.

(More generally, I need to rewrite the entire way the tool uses the database…)

bd808 renamed this task from User overloading labsdb1003 and making it lag to tools.wikiloves (s52927) overloading labsdb1003 and making it lag.Nov 28 2017, 4:54 AM
bd808 assigned this task to JeanFred.
bd808 triaged this task as Medium priority.
bd808 added a project: Tool-wikiloves.

JeanFred- there is a way to avoid multiple runs. On your tools, if they can run on a single machine at a time, use a lock to avoid multiple concurrent instances (e.g. creating a directory is a guaranteed atomic operation). If you need a central place to lock a thing, you can use mysql's GET_LOCK(str,timeout) with a short timeout and kill the process if the lock doesn't succeed.

This can probably be closed.
Labsdb1003 no longer exists and we have now a query killer on the new lasdb hosts + all the users have now a limit of connections :-)