Text version for easy copy paste:

ListeriaBot notified in https://bitbucket.org/magnusmanske/listeria/issues/58/very-high-login-rate

https://ar.wikipedia.org/wiki/%D9%85%D9%88%D8%B6%D9%88%D8%B9:Vp2mudl4jr44aduw
https://commons.wikimedia.org/wiki/User_talk:Schlurcher#High_SchlurcherBot_login_rate
https://fr.wikipedia.org/wiki/Discussion_utilisateur:Hexasoft/Hexabot#High_Hexabot_login_rate
https://www.wikidata.org/wiki/Topic:Vp2n54xa08diok81
https://meta.wikimedia.org/wiki/User_talk:FaFlo#High_login_rate%3F
https://en.wikipedia.org/wiki/User_talk:Oleg_Alexandrov#High_WP_1.0_bot_login_rate

That's everyone notified with over 10K logins per 48H.

IMHO, anything over 100 in a 48H period (that's just over 2 per hour) is probably doing something wrong

IMHO, anything over 100 in a 48H period (that's just over 2 per hour) is probably doing something wrong

Thanks. I'll review AlaaBot, as in real it's weird to make 2,667!

In T256533#6261590, @alanajjar wrote:

IMHO, anything over 100 in a 48H period (that's just over 2 per hour) is probably doing something wrong

Thanks. I'll review AlaaBot, as in real it's weird to make 2,667!

Thanks!

2668/48 is like 55 per hour... so almost one a minute. So something doesn't feel quite right :)

https://www.wikidata.org/wiki/User_talk:Emaus#High_EmausBot_login_rate
https://fr.wikinews.org/wiki/Discussion_utilisateur:Mattho69#High_Matthobot_login_rate
https://bitbucket.org/magnusmanske/commons-delinquent/issues/64/high-login-rate
https://bn.wikipedia.org/wiki/%E0%A6%AC%E0%A7%8D%E0%A6%AF%E0%A6%AC%E0%A6%B9%E0%A6%BE%E0%A6%B0%E0%A6%95%E0%A6%BE%E0%A6%B0%E0%A7%80_%E0%A6%86%E0%A6%B2%E0%A6%BE%E0%A6%AA:Wikitanvir#High_%E0%A6%85%E0%A6%AD%E0%A7%8D%E0%A6%AF%E0%A6%B0%E0%A7%8D%E0%A6%A5%E0%A6%A8%E0%A6%BE_%E0%A6%95%E0%A6%AE%E0%A6%BF%E0%A6%9F%E0%A6%BF_%E0%A6%AC%E0%A6%9F_login_rate

Those over 5K

https://arz.wikipedia.org/wiki/%D9%86%D9%82%D8%A7%D8%B4_%D8%A7%D9%84%D9%85%D8%B3%D8%AA%D8%AE%D8%AF%D9%85:HitomiAkane#High_login_rate%3F
https://en.wikipedia.org/wiki/User_talk:AmandaNP#High_DeltaQuadBot_login_rate
https://zh.wikipedia.org/wiki/User_talk:Antigng#High_Antigng-bot_login_rate

https://en.wikipedia.org/wiki/Special:CentralAuth?target=EntretenimientatoBOT bot blocked?

https://commons.wikimedia.org/wiki/User_talk:Zhuyifei1999#High_FlickreviewR_2_login_rate
https://de.wikipedia.org/wiki/Benutzer_Diskussion:Luke081515#High_Luke081515Bot_login_rate
https://sv.wikipedia.org/wiki/Anv%C3%A4ndardiskussion:Fluff#High_Perfect_login_rate

Thanks for notification. The problem is what I'm running is not a single task, but ten individual tasks instead. To minimize common mode failures, they were designed to run as individual processes, each being able to do its own task by itself, from login to querying to editing, without relying on sessions/tokens from other processes. On finishing, they logout and exit. A batch file is created to run all of them every 10 mins.

Reusing sessions or sharing sessions between these tasks would require, at least, communication between processes, which is completely beyond my scope, or at worst, code refactoring.

So at this moment, the only thing I can do is to reduce the frequency at which each of them is run, and my bottom line is once every 30 minutes. Otherwise there'll be service degradation, something not acceptable. This will lead to a login every 30/10=3 mins, or 960/48hrs.
That's it. And I cannot go further beyond that. Also I don't think such a login rate could cause threats to your server.

Hundreds or thousands of logins a day is generally a sign of bigger issues.

You don't necessarily need to share sessions between different processes/tasks. You should be able to persist login sessions across different runs of the same tasks though, which may require some work.

If they're long running processes, they don't need to login repeatedly, just at the start, and use the assert method (and error handling) to check they're still logged in

You don't necessarily need to reduce the run rate, your count isn't that particularly high. The example of JarBot logging in 114K in 48 hours is a bigger one. Nearly 2.4K an hour. Which is 40 per minute, basically every 1.5 seconds... There is absolutely no reason a bot needs to login that frequently

It's not necessarily threats to the server, but more that it should be unnecessary to login that frequently. How often do you get logged out of your browser session, for example?

Hi, I will fix my bot Mr.Ibrahembot .

Hello, is the information in Wikiscan correct?

"Per T256532"

so what's the actual problem now here? I cannot view this task.

For a few months now, every time I start a Pywikibot script, it starts with

WARNING: No user is logged in on site wikipedia:hu
Logging in to wikipedia:hu as <account>

I don’t appear on this list as I only start bots manually, so no more than at most a few dozens of logins a day, but scheduled bots can reach quite large numbers this way.

I thought I would mention there is now an Upstream request for mwclient
https://github.com/mwclient/mwclient/issues/256

For the wikibase-cli toolkit I opened https://github.com/maxlath/wikibase-cli/issues/120

In T256533#6261565, @Ladsgroup wrote:

Text version for easy copy paste:

JarBot	114,698
ListeriaBot	70,143
Mr.Ibrahembot	53,443
SchlurcherBot	36,728
Hexabot	34,736
Scidudebot	22,998
FaFlo	16,436
WP 1.0 bot	13,605
EmausBot	7,084
Matthobot	6,846
CommonsDelinker	6,763
অভ্যর্থনা কমিটি বট	5,360
HitomiAkane	3,731
DeltaQuadBot	2,965
AlaaBot	2,667
Antigng-bot	2,632
FlickreviewR 2	2,616
EntretenimientatoBOT	2,492
Luke081515Bot	2,327
Perfect	1,918

Can you check again the last 12 hours? I fixed some bugs and I want to know if the bugs related to this issue. Thank you.

I just checked and it got much better but it still logins 10 times a minute and still most of logins in Arabic Wikipedia

In T256533#6262569, @Ladsgroup wrote:

I just checked and it got much better but it still logins 10 times a minute and still most of logins in Arabic Wikipedia

Good to hear that, I fixed new bugs. If you can post the new logins number of JarBot every 12 hours for the next couple days to know if the issue still I will be thankful.

In T256533#6262902, @Jar wrote:

In T256533#6262569, @Ladsgroup wrote:

I just checked and it got much better but it still logins 10 times a minute and still most of logins in Arabic Wikipedia

Good to hear that, I fixed new bugs. If you can post the new logins number of JarBot every 12 hours for the next couple days to know if the issue still I will be thankful.

Last 12 hours: 1,206 logins (~1.67 per minute, almost 1/60th of the original number). Thanks!

Using this to log in with a bot password for the wikiwho API. Logins should not happen THAT often. Are data available if/when the login rates for my account drastically increased? Will try to fix.

In T256533#6264581, @FaFlo wrote:

Using this to log in with a bot password for the wikiwho API. Logins should not happen THAT often. Are data available if/when the login rates for my account drastically increased? Will try to fix.

Does this help?

This is for the past seven days and each bar is for three-hour timespan (I see a noticeable drop at 25th June went from 20,000 logins every 3 hours to 600 per three hours). It still logs in 10,000 times in the past 48 hours.

In T256533#6264355, @Ladsgroup wrote:

In T256533#6262902, @Jar wrote:

In T256533#6262569, @Ladsgroup wrote:

I just checked and it got much better but it still logins 10 times a minute and still most of logins in Arabic Wikipedia

Good to hear that, I fixed new bugs. If you can post the new logins number of JarBot every 12 hours for the next couple days to know if the issue still I will be thankful.

Last 12 hours: 1,206 logins (~1.67 per minute, almost 1/60th of the original number). Thanks!

Hello Ladsgroup, does it get lower than that?

In T256533#6271433, @Jar wrote:

In T256533#6264355, @Ladsgroup wrote:

Last 12 hours: 1,206 logins (~1.67 per minute, almost 1/60th of the original number). Thanks!

Hello Ladsgroup, does it get lower than that?

YES, last 12 hours only 15. THANK YOU.

Hi, SchlurcherBot should also be fixed. I switched approx. 2 days ago to OAuth verification for the majority of my tasks that need login. Schlurcher

In T256533#6273878, @Schlurcher wrote:

Hi, SchlurcherBot should also be fixed. I switched approx. 2 days ago to OAuth verification for the majority of my tasks that need login. Schlurcher

Yes, it went from 200/hour to 10/hour. Thanks!

The average number of logins (everything, everywhere) in the past seven days went from 20K per hour to 7k per hour. The new list in the past 48 hours:

If we can reduce the top four the below 5K per day, I think the first phase can be called done.

Only AGbot is now in that list that is above the 5K a day

https://www.wikidata.org/wiki/User_talk:Andrew_Gray#High_AGBot_login_rate

In T256533#6261957, @Luke081515 wrote:

"Per T256532"

so what's the actual problem now here? I cannot view this task.

Can somebody give me an answer concerning this question?

In T256533#6284985, @Luke081515 wrote:

In T256533#6261957, @Luke081515 wrote:

"Per T256532"

so what's the actual problem now here? I cannot view this task.

Can somebody give me an answer concerning this question?

That task is private for security reasons. Suffice to say that there is a potential for abuse that we are trying to prevent.

@Ladsgroup When you have a chance, can I get another count? Want to know how i'm doing in reducing my footprint and want to get it as low as possible.

In T256533#6383223, @DeltaQuad wrote:

@Ladsgroup When you have a chance, can I get another count? Want to know how i'm doing in reducing my footprint and want to get it as low as possible.

It's 645 in the past 24 hours, in comparison @Mr.Ibrahem's bot did 50K logins in the past 24 hours. @Ibrahem: Will you fix your bot or should I block it?

@DeltaQuad In other words, it's pretty good, it's lower than most bots. Thank you!

In T256533#6383310, @Ladsgroup wrote:

In T256533#6383223, @DeltaQuad wrote:

@Ladsgroup When you have a chance, can I get another count? Want to know how i'm doing in reducing my footprint and want to get it as low as possible.

It's 645 in the past 24 hours, in comparison @Mr.Ibrahem's bot did 50K logins in the past 24 hours. @Ibrahem: Will you fix your bot or should I block it?

Per conversation with Ladsgroup, the bot is now locked until the issue is fixed.

Thanks for the update @Urbanecm

Do we need to refresh the data and see if there are any other accounts that are still logging in too frequently?

Here are current logins ordered by frequency

from https://logstash.wikimedia.org/goto/d7fcb59c2cc892b96bf1100fd77994df (last 2 days)

In T256533#6407377, @Urbanecm wrote:

In T256533#6383310, @Ladsgroup wrote:

In T256533#6383223, @DeltaQuad wrote:

@Ladsgroup When you have a chance, can I get another count? Want to know how i'm doing in reducing my footprint and want to get it as low as possible.

It's 645 in the past 24 hours, in comparison @Mr.Ibrahem's bot did 50K logins in the past 24 hours. @Ibrahem: Will you fix your bot or should I block it?

Per conversation with Ladsgroup, the bot is now locked until the issue is fixed.

@Urbanecm
Hi, sorry for that, I had stop the most active jobs in my bot and I had made some changes .. Can you unblock the bot ?

And Is there any way to track my bot logins ?

In T256533#6409637, @Mr.Ibrahem wrote:

In T256533#6407377, @Urbanecm wrote:

In T256533#6383310, @Ladsgroup wrote:

In T256533#6383223, @DeltaQuad wrote:

@Ladsgroup When you have a chance, can I get another count? Want to know how i'm doing in reducing my footprint and want to get it as low as possible.

It's 645 in the past 24 hours, in comparison @Mr.Ibrahem's bot did 50K logins in the past 24 hours. @Ibrahem: Will you fix your bot or should I block it?

Per conversation with Ladsgroup, the bot is now locked until the issue is fixed.

Hi, sorry for that, I had stop the most active jobs in my bot and I had made some changes .. Can you unblock the bot ?

Hello, thanks for the changes you made. I've removed the account lock then.

And Is there any way to track my bot logins ?

Sadly, that's not possible. The data are recorded by the servers, but they are intentionally non-public. I'm happy to query your account data for you, you can just ping me at IRC, or ask here :).

In T256533#6409637, @Mr.Ibrahem wrote:
And Is there any way to track my bot logins ?

If you are using pywikibot to create your bots, then you should know that pywikibot reports every log in as a log on the command line. You should look at the output of your bots, not just through it away.

In T256533#6410837, @Huji wrote:

In T256533#6409637, @Mr.Ibrahem wrote:
And Is there any way to track my bot logins ?

If you are using pywikibot to create your bots, then you should know that pywikibot reports every log in as a log on the command line. You should look at the output of your bots, not just through it away.

FYI: According to user agents, the only PWB-based bot is Matthias Winkelmann (who is not really a bot, but a flooder, I don't know if that changes something)

In T256533#6410580, @Urbanecm wrote:

Sadly, that's not possible. The data are recorded by the servers, but they are intentionally non-public. I'm happy to query your account data for you, you can just ping me at IRC, or ask here :).

@Urbanecm Can you tell me the last logins in last 24h and 12h ?

In the past 24 and 12 hours you had 1,024 and 436 logins respectively

I sent notices to users from the current list according to https://logstash.wikimedia.org/goto/d7fcb59c2cc892b96bf1100fd77994df. Let's see what happens in a week.

Wanted to come here and acknowledge this issue. I've opened a bug against the WP 1.0 project to fix this. It should be taken care of this weekend. Thank you @DeltaQuad for the link to mwclient workaround!

Thanks @Audiodude! Let me know if we can support you in some way.

Okay I've deployed the fix to WP 1.0 bot as of tonight's update. It should be good the next time you run the numbers. Please let me know.

In T256533#6421842, @Audiodude wrote:

Okay I've deployed the fix to WP 1.0 bot as of tonight's update. It should be good the next time you run the numbers. Please let me know.

Confirmed, in the last two days, I see 28 logins. Thank you!

In T256533#6408832, @Urbanecm wrote:

Here are current logins ordered by frequency

Mr.Ibrahembot	127162
ListeriaBot	74121
WP 1.0 bot	20387
FaFlo	15889
EmausBot	12735
CommonsDelinker	11952
Matthias Winkelmann	5351
AlaaBot	4356
FlickreviewR 2	3776
Luke081515Bot	3482
YouTubeReviewBot	3172
Antigng-bot	2058
Lê Lợi (bot)	1963
DeltaQuadBot	1895
Olafbot	1638
WikitanvirBot	1593
MusikBot	1585
Jembot	1561
AlbeROBOT	1351
MusikBot II	1270

from https://logstash.wikimedia.org/goto/d7fcb59c2cc892b96bf1100fd77994df (last 2 days)

JFYI: On behalf of @FaFlo bot, we 've planned to work around the problem towards the end of the week.

As of today, FaFlo is the only bot that sends moe than 10k requests/3 days. I sent him a follow-up message via both email and talk page, including a note the account may be disabled. Apart from that, CommonsDelinker is very close to 10k (it made 9612 requests). On the positive side, ListeriaBot disapppeared from the list.

Thanks @Olgazgovora for the info.

In T256533#6435090, @Urbanecm wrote:

As of today, FaFlo is the only bot that sends moe than 10k requests/3 days. I sent him a follow-up message via both email and talk page, including a note the account may be disabled. Apart from that, CommonsDelinker is very close to 10k (it made 9612 requests). On the positive side, ListeriaBot disapppeared from the list.

Thanks @Olgazgovora for the info.

Dear @Urbanecm,
Could you, please, provide statistics on how the requests changed for FaFlo bot during last 3-4 hours? Is there any difference? We removed it for one project (WhoColor) but not solved yet for the second (WikiWho). Thank you.

Sincerely,
Olya

Dear @Olgazgovora,

sure. In short, the bot dramatically increased its amount of login requests (from about 5 % of login requests to more than 70 % of login requests), and as such, it has been temporarily blocked.

[urbanecm@mwlog1001 /srv/mw-log]$ head -n 1 goodpass.log | grep -Eo '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
2020-09-06 08:34:43
[urbanecm@mwlog1001 /srv/mw-log]$ grep FaFlo goodpass.log | wc -l
52325
[urbanecm@mwlog1001 /srv/mw-log]$ tail -n1 goodpass.log | grep -Eo '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
2020-09-06 15:02:22
[urbanecm@mwlog1001 /srv/mw-log]$ wc -l < goodpass.log
72722
[urbanecm@mwlog1001 /srv/mw-log]$

According to the logs, your bot made 52325 successful login attempts (out of 72722 attempts in total) between 2020-09-06 08:34:43 and 2020-09-06 15:02:22 (ie. in 6,5 hours). That means it made over 70 % of requests alone, and that it logs in more than twice per second. Very similar numbers apply for both yesterday (September 05) and the day before (September 04).

As such, I have temporarily locked the account. Please fix that issue, and let me know.

Sincerely,

Martin Urbanec

In T256533#6438978, @Urbanecm wrote:

Dear @Olgazgovora,

sure. In short, the bot dramatically increased its amount of login requests (from about 5 % of login requests to more than 70 % of login requests), and as such, it has been temporarily blocked.

[urbanecm@mwlog1001 /srv/mw-log]$ head -n 1 goodpass.log | grep -Eo '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
2020-09-06 08:34:43
[urbanecm@mwlog1001 /srv/mw-log]$ grep FaFlo goodpass.log | wc -l
52325
[urbanecm@mwlog1001 /srv/mw-log]$ tail -n1 goodpass.log | grep -Eo '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
2020-09-06 15:02:22
[urbanecm@mwlog1001 /srv/mw-log]$ wc -l < goodpass.log
72722
[urbanecm@mwlog1001 /srv/mw-log]$

According to the logs, your bot made 52325 successful login attempts (out of 72722 attempts in total) between 2020-09-06 08:34:43 and 2020-09-06 15:02:22 (ie. in 6,5 hours). That means it made over 70 % of requests alone, and that it logs in more than twice per second. Very similar numbers apply for both yesterday (September 05) and the day before (September 04).

As such, I have temporarily locked the account. Please fix that issue, and let me know.

Sincerely,

Martin Urbanec

I've changed the code for the Wikiwho project too. Could you, please, unlock FaFlo and also tell us how the number of login requests changed? Thank you!

Sincerely,
Olya

I have unlocked the user account, and I will monitor the logins.

In T256533#6440407, @Urbanecm wrote:

I have unlocked the user account, and I will monitor the logins.

Thank you. And, please, let me know if there are changes with login amounts.

In T256533#6440777, @Olgazgovora wrote:

In T256533#6440407, @Urbanecm wrote:

I have unlocked the user account, and I will monitor the logins.

Thank you. And, please, let me know if there are changes with login amounts.

Sure. So far, there were three user logins. Tanks!

I think we call this resolved. I don't see a bot that logins extremely frequently now.

Should we open a new one that would focus on creating an automated mechanism to detect these?

Do we even have similar automated measures? For instance, do DBAs get a notification if a DB's growth suddenly speeds up?

In T256533#6456740, @Huji wrote:

Should we open a new one that would focus on creating an automated mechanism to detect these?

Once T256532 is completed, there shouldn't be any need to manually hunt those.

Do we even have similar automated measures? For instance, do DBAs get a notification if a DB's growth suddenly speeds up?

That'd be a question for the DBAs, @Marostegui?

AFAIK bots are not logged in cu tables (yet). So this ticket is not relevant to size of CU tables. Am I missing something obvious?

You are not. But recall that this ticket is about accounts with very high login rate, not just bots. Right now, it happens that all culprits were bots. But in a future state, a non-bot account (read: a bot without a bot flag) might do the same, and we want to identify that proactively.

In T256533#6456773, @Urbanecm wrote:

In T256533#6456740, @Huji wrote:

Should we open a new one that would focus on creating an automated mechanism to detect these?

Once T256532 is completed, there shouldn't be any need to manually hunt those.

Do we even have similar automated measures? For instance, do DBAs get a notification if a DB's growth suddenly speeds up?

That'd be a question for the DBAs, @Marostegui?

We have an alert based on backups size, so if there's a sudden increase on a backup size from one week to the next week, it will get triggered. It obviously depends on how big the increase (on disk) is.

Entretenimientato and EntretenimientoBorrarBOT were both vandal bots operated by a WMF-legal banned user.