Page MenuHomePhabricator

s51053 (tools.jackbot) is abusing resources on labsdbs, throttle his grants
Closed, ResolvedPublic

Description

s51053 is executing very costly queries, sometimes exact duplicates of 7h queries in parallel.

Throttle his grants to restrict to 1 concurrent query per server and a 1 hour of execution limit.

Event Timeline

jcrespo claimed this task.
jcrespo raised the priority of this task from to Needs Triage.
jcrespo updated the task description. (Show Details)
jcrespo added projects: Cloud-Services, Toolforge, DBA.
jcrespo subscribed.
valhallasw renamed this task from s51053 is abusing resources on labsdbs, throttle his grants to s51053 (tools.jackbot) is abusing resources on labsdbs, throttle his grants.Oct 3 2015, 9:50 AM
valhallasw set Security to None.
valhallasw added a subscriber: JackPotte.

The site https://tools.wmflabs.org/jackbot/snottywong/ has a very high number of visitors, it had been developed by snottywong for the English Wikipedia needs.

But I won't hesitate to shut it down definitely if this is a problem for you:

tools.jackbot@tools-bastion-01:~$ tail access.log

10.68.21.81 tools.wmflabs.org - [03/Oct/2015:09:23:08 +0000] "GET /jackbot/snottywong/usersearch.html?page=Kategori:Kaliforniya_%C3%BCniversitesi,_San_Diego HTTP/1.1" 200 14658 "https://tr.wikipedia.org/w/index.php?title=Kategori:Kaliforniya_%C3%BCniversitesi,_San_Diego&action=info" "Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit (KHTML, like Gecko)"
10.68.21.81 tools.wmflabs.org - [03/Oct/2015:09:31:31 +0000] "GET /jackbot/snottywong/cgi-bin/votecounter.cgi?page=Wikipedia:Articles_for_deletion/Neils_Hogenson_House HTTP/1.1" 200 1306 "https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion/Log/2015_September_27" "Mozilla/5.0 (Windows NT 6.1; rv:40.0) Gecko/20100101 Firefox/40.0"
10.68.21.81 tools.wmflabs.org - [03/Oct/2015:09:31:31 +0000] "GET /jackbot/snottywong/cgi-bin/menubar3.css HTTP/1.1" 404 345 "https://tools.wmflabs.org/jackbot/snottywong/cgi-bin/votecounter.cgi?page=Wikipedia:Articles_for_deletion/Neils_Hogenson_House" "Mozilla/5.0 (Windows NT 6.1; rv:40.0) Gecko/20100101 Firefox/40.0"
10.68.21.81 tools.wmflabs.org - [03/Oct/2015:09:31:31 +0000] "GET /jackbot/snottywong/cgi-bin/greyscale.css HTTP/1.1" 200 0 "https://tools.wmflabs.org/jackbot/snottywong/cgi-bin/votecounter.cgi?page=Wikipedia:Articles_for_deletion/Neils_Hogenson_House" "Mozilla/5.0 (Windows NT 6.1; rv:40.0) Gecko/20100101 Firefox/40.0"
10.68.21.81 tools.wmflabs.org - [03/Oct/2015:09:41:30 +0000] "GET /jackbot/snottywong/cgi-bin/editorinteract.cgi?user1=Hammerpleasedonthurtem&user2=Bangabandhu&user3=&user4=&user5=&user6=&user7=&user8=&user9=&user10=&ns=none&startdate=&enddate= HTTP/1.1" 200 1800 "https://tools.wmflabs.org/jackbot/snottywong/editorinteract.html" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0"
10.68.21.81 tools.wmflabs.org - [03/Oct/2015:09:42:06 +0000] "GET /jackbot/snottywong/cgi-bin/editorinteract.cgi?user1=Hammerpleasedonthurtem&user2=Bangabandhu&user3=&user4=&user5=&user6=&user7=&user8=&user9=&user10=&ns=none&startdate=&enddate= HTTP/1.1" 200 1800 "https://tools.wmflabs.org/jackbot/snottywong/editorinteract.html" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0"
10.68.21.81 tools.wmflabs.org - [03/Oct/2015:09:42:41 +0000] "GET /jackbot/snottywong/cgi-bin/editorinteract.cgi?user1=StAnselm&user2=Ian.thomson&user3=&user4=&user5=&user6=&user7=&user8=&user9=&user10=&ns=none&startdate=&enddate= HTTP/1.1" 200 1820 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0"
10.68.21.81 tools.wmflabs.org - [03/Oct/2015:09:42:53 +0000] "GET /jackbot/snottywong/usersearch.html?page=Kategori_tart%C4%B1%C5%9Fma:Kaliforniya_%C3%BCniversitesi,_San_Diego HTTP/1.1" 200 14658 "https://tr.wikipedia.org/w/index.php?title=Kategori_tart%C4%B1%C5%9Fma:Kaliforniya_%C3%BCniversitesi,_San_Diego&action=info" "Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit (KHTML, like Gecko)"
10.68.21.81 tools.wmflabs.org - [03/Oct/2015:09:43:01 +0000] "GET /jackbot/snottywong/cgi-bin/editorinteract.cgi?user1=StAnselm&user2=Ian.thomson&user3=&user4=&user5=&user6=&user7=&user8=&user9=&user10=&ns=none&startdate=&enddate= HTTP/1.1" 200 1820 "https://tools.wmflabs.org/jackbot/snottywong/editorinteract.html" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0"
10.68.21.81 tools.wmflabs.org - [03/Oct/2015:09:57:06 +0000] "HEAD /jackbot/snottywong/cgi-bin/votecounter.cgi?page=Wikipedia:Articles_for_deletion/Kim_Davis_(county_clerk HTTP/1.1" 200 0 "-" "Mozilla/5.0"
tools.jackbot@tools-bastion-01:~$

Hi, @JackPotte, can you confirm you are s51053?

High number of hits are not currently an issue.

However, mainly in labsdb1001 (enwiki, cannot remember if something else by default), we regularly get 7h queries from your user, sometimes several of them with the same SQL. Large transactions take a huge amount of resources, can produce lag and can end up filling up the disk.

On Wikimedia production, requests that take more than 300 seconds are automatically killed. I suggest that your application should do the same. Otherwise, we will have to enforce limits on the quantity and size of them. For long query execution, a queue should be implemented, similarly to https://quarry.wmflabs.org/.

Hi, @JackPotte, can you confirm you are s51053?

You can check this stuff like so:

krenair@tools-bastion-01:~$ ldaplist -l servicegroups | grep 51053 -B 5
dn: cn=tools.jackbot,ou=servicegroups,dc=wikimedia,dc=org
	objectClass: groupofnames
	objectClass: posixgroup
	objectClass: top
	member: uid=jackpotte,ou=people,dc=wikimedia,dc=org
	gidNumber: 51053
krenair@tools-bastion-01:~$ ldaplist -l passwd jackpotte | grep cn
	cn: JackPotte

Which is the linked LDAP user in @JackPotte's Phabricator profile.

@Krenair, yes, but many people do not use Phabricator, or IRC, or email, or the Wikis,... there is no single point of contact. Hence T114560. (I wasn't "authenticating" him, it was just an informal question, but thank you for the input).

So I propose to study the queue system with https://github.com/wikimedia/analytics-quarry-web during my next free time: in six months.

Ok, meanwhile, I will reduce the impact on db side (as I intended originally with this ticket), because we have received complains of slowness from other labs users.

This should be resolved now, do you know a monitoring on which I could check it please?

This is resolved, I only edited it because of admin purposes (correct tracking). Sorry for the spam, email is automatic.