Page MenuHomePhabricator

s51580 blocked from access to wikireplicas with suspicion of having tried a DOS
Closed, ResolvedPublic

Description

I have the suspicion that s51580 may have been used (willingly or unwillingly) to perform a DOS attack against the databases, which has been partially succesful, forcing labsdb1010 to failover service to labsdb1009 through its load balancer. If that was intentional or mistake or a compromised account, I do not know.

This was the report from tendril:

Hits	Tmax	Tavg	Tsum	Hosts	Users	Schemas
589	18,335	15,395	9,067,658	labsdb1010	s51580	
select count(*), fr_user from dewiki_p.flaggedrevs where ( fr_flags = 'dynamic' or fr_flags = ', dynamic' ) and fr_timestamp like '201712%' group by fr_user order by count(*) /* SLOW_OK */ /* 8572d5fb0ce04c1550e67b48fa7322c9 labsdb1010 12455s */
1066	661	261	278,254	labsdb1009, labsdb1010	s51580	
#select all articles that were never reviewed #1 min 20, 335 select page_id, page_title from dewiki_p.page #not flagged where page_id not in (select distinct fp_page_id from dewiki_p.flaggedpages) #not a redirect and page_is_redirect = 0 #and page_id not in (select distinct rd_from from dewiki_p.redirect) #and in article namespace and page_namespace = 0 /* 50752df04c3acd7da0baddaacc3d1773 labsdb1010 2s */

Event Timeline

jcrespo created this task.Dec 17 2017, 7:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 17 2017, 7:33 AM
Marostegui changed the visibility from "Public (No Login Required)" to "WMF-NDA (Project)".
Marostegui added a subscriber: Marostegui.

I have changed this to only visible for NDA members, just in case for now.

I would guess unintentional - that query is a fairly logical way to ask "what is the breakdown of people waiting for flagged revisions review in december 2017". If it was intentional I would expect the query to be less directly asking a question that wiki users would be so likely to have.

The query is not such a big problem, as much as making 500 of those within seconds.

jcrespo changed the visibility from "WMF-NDA (Project)" to "Public (No Login Required)".

Were the users behind these s number already informed?

Hi

sorry about that, that was definitely unintentional and the idea was never to execute so many queries. That query should have been executed once a day using a cronjob. The result is stored in a text file and then displayed to the users to show a ranking of which users have reviewed how many pages. I currently dont know how this could amount to 500 queries in a second, I see the cronjob running once a day and producing the expected data.

So, the query itself is part of normal operations but I could not figure out yet why there are so many requests and I can only see the regular cronjob on my side doing one query a day.

Hi, @Hannes_Rost_MW , being a mistake, if you disable the cronjob and promise to check why it happened and/or put measures to avoid it (e.g. https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Query_Limits ) I can reenable your account right away.

@jcrespo thanks, I will do that and enable the query limits.

jcrespo closed this task as Resolved.Jan 3 2018, 6:32 PM
jcrespo claimed this task.

I have reenabled the account, but I will impose permanent account limits if a DOS-like pattern reappears.

great, thanks. I will keep an eye out.