I instead think we should:

• Remove noratelimit from all bots
• Add accounts that require noratelimit to "accountcreator" group

Before this we should:

• Find all bots that edits more than 90/minute
• Communiate to them

In T258354#6317978, @Bugreporter wrote:

Before this we should:

• Find all bots that edits more than 90/minute
• Communiate to them

I am already monitoring user edit rates occasionally with a custom Python script that reads the Wikimedia RC stream, and I have contacted several bot operators regarding their edit rates when I saw excessively high values. I wasn't particularly successful, I have to admit.

Unfortunately the situation with maxlag>5 for considerable fractions of a day due to the WDQS bottleneck has brought us to a situation where there is sort of a competition for as many editing resources as possible among bot operators. As long as there is neither a policy enforcing a limit nor a hard technical limit, the greediest operator will simply win this competition. There are already quite some bot operators who challenge the Wikidata bot policy by editing up to maxlag=6 or even 7, as they probably think that nobody will notice.

We also have to remember that exact throttling to a targeted edit rate is not simple at all. Some tools such as QuickStatements do not allow to control the edit rate at all and they just hammer as many edits as possible to the servers, particularly for users that are not rate limited; quite some bot operators simply use a framework such as Pywikibot and might not be overly proficient when it comes to throttling (although pywikibot in particular does not create that many issues I think); and finally, it is quite difficult to monitor the own edit rate at all, and to realize that one causes issues.

I'm afraid I have to say that I do not see a chance for a fair distribution of our limited edit resources among users and bot operators as long as we do not have a hard technical edit rate limit.

Regarding the proposed solution:

• "flooders" should be treated the same as bots
• I would like to see a way to limit all other unlimited users' edit rates as well (sysops, apparently global rollbackers, ...). When I use my sysop account with QuickStatements and run a single batch, it can easily go up to 400 or 500 edits per minute and there is no possibility for me to slow down. Not fair for all users without such elevated rights.

It needs a thorough assessment which functionality categorically needs to be unlimited (mass message, nuke?, ...). Potential users of these functionality should be able to temporarily add no-ratelimit to their accounts *for the purpose of using these functions only*, and should remove it thereafter.

What would happen if a bot reached a rate limit? Would its edits just be delayed, the same as what happens currently when maxlag is reached, or would the edits be dropped/result in an error message? Hopefully it would be the former.

+1 to what @MisterSynergy said.

What would happen if a bot reached a rate limit?

I believe they will be receiving an error message with ratelimited until they have waited enough time. Whether the bot crashes or waits depends on the used framework.

Add accounts that require noratelimit to "accountcreator" group

Quite odd solution.

In T258354#6317977, @Bugreporter wrote:

I instead think we should:

• Remove noratelimit from all bots
• Add accounts that require noratelimit to "accountcreator" group

We discussed it just now and unfortunately this will break mass message at least so is not an option :(

Change 618245 had a related patch set uploaded (by Tobias Andersson; owner: Tobias Andersson):
[operations/mediawiki-config@master] DNM WIP: add new limited bot group

https://gerrit.wikimedia.org/r/618245

ANswers for remaining questions: Let's call it "Slow bots" and announce it before creation. I can write something up. Let's coordinate via chat for a date.

I think the approach is wrong. Let's start with the first assumption. Nuke ( https://www.wikidata.org/wiki/Special:Nuke ) runs under your own account. Used it plenty of times. As an admin I have noratelimit so having noratelimit on a completely different group is not relevant. MassMessage is just an extension that runs in the background. It is a bot. It shouldn't go too fast and it shouldn't crash on ratelimits. Relevant historic bugs: T192690 and T184948 .

Just add a sane (relatively high) ratelimit to the current bots group or keep it as is. The current proposed solution is to try to work around the problem.

When it is created and current bots moved to new group it would be a breaking change.

In T258354#6365838, @Multichill wrote:

Just add a sane (relatively high) ratelimit to the current bots group or keep it as is. The current proposed solution is to try to work around the problem.

Well, that was exactly what we did in T184948: limit page creation and edit rate on Wikidata and you disagreed with it (T184948#4145480)

In T258354#6387030, @Ladsgroup wrote:

In T258354#6365838, @Multichill wrote:

Just add a sane (relatively high) ratelimit to the current bots group or keep it as is. The current proposed solution is to try to work around the problem.

Well, that was exactly what we did in T184948: limit page creation and edit rate on Wikidata and you disagreed with it (T184948#4145480)

No, that's not what you did. You added a ratelimit to a group which had the "noratelimit" right assigned to it causing all sorts of breakage.

Help me understand your proposal better. Sorry if I miss anything obvious. Bots and sysops have "noratelimit" right attached to their user rights groups. We can either have another bots right group that doesn't have the "noratelimit" right or apply a forced ratelimit to the current bot group (which has noratelimit right attached to it). The former is this ticket, the latter is the other. What is the third option here? Stripping bots group out of its noratelimit right?

In T258354#6365968, @Bugreporter wrote:

When it is created and current bots moved to new group it would be a breaking change.

https://www.wikidata.org/wiki/Wikidata:Stable_Interface_Policy#Notification_Policy

Changes to non-stable interfaces may not be announced, even if they are breaking changes.

And these changes does not fall under any stable interface.

Awaiting announcement

In T258354#6317977, @Bugreporter wrote:

I instead think we should:

• Remove noratelimit from all bots
• Add accounts that require noratelimit to "accountcreator" group

I do strongly agree with this, as this would avoid the burden of re-assigning bots from a group to a group, and also i feel this is systematicaly better solution.

In T258354#6317978, @Bugreporter wrote:

Before this we should:

• Find all bots that edits more than 90/minute
• Communiate to them

+1

Instead of a new user group, a better solution would be to remove, for the Wikidata bot group, the noratelimit permission and adjust in rOMWC wmf-config/InitialiseSettings.php an acceptable rate limit for all bots if needed. Currently non-autoconfirmed users gets 8 edits per minute, while autoconfirmed gets 90 edits per minute. 90 edits/minute. Should be an acceptable limit, and just requires a simple config patch instead of (a) create a new group and (b) move accounts between groups.

Ok so here's where we're at:

• asking bots to be nice: This has happened. It's playing whack-a-mole. Admins who care are rightfully upset that this is not a workable solution and we've had complaints asking to not make this a whack-a-mole game anymore. It's also not fair to the people who are actually trying to be good citizens and abite by the rules just to then be outrun by people who don't care or know what they're doing. This is not an acceptable state for me.
• just removing noratelimit from the existing bot group: that's been done in the past and caused bad side-effects with extensions that actually require it like MassMessage and Nuke (?).
• adding the new bot group: I think it'd have been an elegant solution as it gives control to the admins on-wiki but apparently not :D

So that leaves the suggestion to remove noratelimit from all bots and add accounts that require noratelimit to "accountcreator" group. @Ladsgroup: Would this work?

Ok, I see, we only put misbehaving bots in this group.

In T258354#6497417, @Lydia_Pintscher wrote:

So that leaves the suggestion to remove noratelimit from all bots and add accounts that require noratelimit to "accountcreator" group. @Ladsgroup: Would this work?

Definitely.

In T258354#6499293, @Multichill wrote:

Ok, I see, we only put misbehaving bots in this group.

I think the proposal is the other way around, we add important bots that needs bypassing ratelimit to accountcreator group (like massmessage)

I assume we could define a higher ratelimit for bots in general but not too high.

In T258354#6501501, @Ladsgroup wrote:

In T258354#6497417, @Lydia_Pintscher wrote:

So that leaves the suggestion to remove noratelimit from all bots and add accounts that require noratelimit to "accountcreator" group. @Ladsgroup: Would this work?

Definitely.

Ok then let's do this. I'll adapt the ticket accordingly.

I still find a bit ridiculous that fast editing bots will be in a group for account creators...

I like this solution. The only right provided by the flag of account creator is noratelimit, and right now no user account has this flag, so we can freely change the group/flag name with MediaWiki:Group-accountcreator and MediaWiki:Group-accountcreator-member to fit its future purpose on Wikidata.

What should this group/flag be called? Superbot? Something more creative? Ideally, the name shouldn't suggest that it's redundant with the bot group/flag, but additional.

See https://commons.wikimedia.org/wiki/Commons:Village_pump/Proposals/Archive/2018/01#Proposal_to_rename_account_creator_group_to_batch_uploaders

In T258354#6509703, @abian wrote:

What should this group/flag be called? Superbot? Something more creative? Ideally, the name shouldn't suggest that it's redundant with the bot group/flag, but additional.

Ignore rate limits? No rate limit? Fast editors? Fast users? I don't think we should go for superbot, because there are use cases for noratelimit too, such as creating accounts at outreach events.

In T258354#6528169, @Urbanecm wrote:

Ignore rate limits? No rate limit? Fast editors? Fast users? I don't think we should go for superbot, because there are use cases for noratelimit too, such as creating accounts at outreach events.

According to what I've just checked, in the history of Wikidata the accountcreator flag has been granted four times, and on none of those occasions has it been used for anything, the users didn't create any account with it. If we anticipate that such a need may arise in the future, we could introduce the event coordinator group, but for now that need doesn't seem to exist.

I would prefer it to be understood that belonging to this group is an undesirable exception, and I wouldn't like users who don't belong to it (almost everyone) to be considered handicapped or punished because they're categorised as "slow" or "limited", as opposed to "fast" users; in fact, all users can continue to edit at a high rate. Other names as mediocre as "superbot" come to mind, from more serious to more irreverent: structural user/account, greedy user/account, exhausting/draining user/account, speed(ing) offender, freeloader, racehorse (racing goat? 🤨)…

Per announcement, unless there is a strong opposition from the community to this change we will implement it on October 20th.
Thanks a lot to everyone giving constructive feedback under this ticket!

Change 634938 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/mediawiki-config@master] Remove noratelimit from Wikidata bot group

https://gerrit.wikimedia.org/r/634938

Actually, @hoo might be able to add User:MediaWiki message delivery to Account creators, as a steward? I’m not sure – global groups confuse me.

Yeah if hoo can't do it then let's ask ymblanter.

Change 634938 merged by jenkins-bot:
[operations/mediawiki-config@master] Remove noratelimit from Wikidata bot group

https://gerrit.wikimedia.org/r/634938

Mentioned in SAL (#wikimedia-operations) [2020-10-20T11:09:56Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:634938|Remove noratelimit from Wikidata bot group (T258354)]] (duration: 00m 56s)

Change deployed. So far, the edit rate at Wikidata Edits hasn’t come crashing down, which is probably good.

In T258354#6559861, @Lucas_Werkmeister_WMDE wrote:

Actually, @hoo might be able to add User:MediaWiki message delivery to Account creators, as a steward? I’m not sure – global groups confuse me.

Note: Stewards can technically do literally anything (ie. anything the web interface allows for) – so yes, @hoo would be technically able to do the change. However, stewards will never use their authority on projects with locals who can do the change - in this case, Wikidata's bureaucrats.

I would support using createAndPromote.php, since this is in fact a sysadmin decision, and not something that can be decided about by any community.

I'm however not sure why is this necessary: AFAICS, https://github.com/wikimedia/mediawiki-extensions-MassMessage/blob/master/includes/Job/MassMessageServerSideJob.php#L68 doesn't actually call User::pingLimiter, so it doesn't seem to be affected?

Well, IIRC we’ve had problems with MassMessage being rate-limited in the past, so it’s probably better to be safe than sorry and still add it to the group. I’ve left a message on Ymblanter’s talk page.

In T258354#6563989, @Lucas_Werkmeister_WMDE wrote:

Well, IIRC we’ve had problems with MassMessage being rate-limited in the past, so it’s probably better to be safe than sorry and still add it to the group. I’ve left a message on Ymblanter’s talk page.

I think the usual place to post such requests is https://www.wikidata.org/wiki/Wikidata:Bureaucrats%27_noticeboard - but that's mostly Ymblanter anyway!

Change 618245 abandoned by Tobias Andersson:
[operations/mediawiki-config@master] Add new slow-bot group for Wikidata

Reason:
https://phabricator.wikimedia.org/T258354#6509213

https://gerrit.wikimedia.org/r/618245