Page MenuHomePhabricator
Create Task
Maniphest T243844

[SUPPORT] wikibase change edit rate / wikibase scalability
Closed, ResolvedPublic
Actions

Description

Hello,

I'm currently importing a bigger dataset into a wikibase. I would say it is like a batch import. I'm using a pywikibot and setting:

minthrottle = 0
maxthrottle = 0

So as far I can see, he is trying to write in the wikibase as fast as he can. I also minimised the number of API calls but still it is "slow". I can make between 50-100 edits per minute, each edit with around 10 statements and some labels. Is there a way to make it faster? I saw this ticket:

https://phabricator.wikimedia.org/T184948

is there a maximum edit limit for the wiki per user? Is it a matter of infrastructure? If I could make 500-1000 edits per minute would be great! Do you think this is feasible?

Salut
D063520

Related Objects

Event Timeline

DD063520 created this task.Jan 28 2020, 3:16 PM
Restricted Application added a project: User-Addshore. · View Herald TranscriptJan 28 2020, 3:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
DD063520 updated the task description. (Show Details)Jan 28 2020, 3:16 PM
Bugreporter closed this task as Invalid.Jan 28 2020, 4:11 PM
Bugreporter subscribed.

Please use an account with "bot" flag to do the import. Otherwise the rate of edits may not exceed 90/minute. Note anything described in T184948: limit page creation and edit rate on Wikidata is no longer in effect.

If this is not enough, try to run multiple parallel process with pt:0, provided that you have monitering the server to keep it healthy (Please only use this in your local Wikibase installation, NEVER do this in production Wikidata!)

Addshore reopened this task as Open.EditedJan 28 2020, 4:12 PM

Sounds like this is talking about Wikibase not wikidata.
Am I correct @DD063520 ?

Addshore moved this task from incoming to in progress on the Wikidata board.Jan 28 2020, 4:13 PM
Bugreporter added a comment.EditedJan 28 2020, 4:14 PM

This user probably does not use a bot account to import, so it is affected by default MediaWiki rate limit (90/min).

Addshore renamed this task from wikibase change edit rate / wikibase scalability to [SUPPORT] wikibase change edit rate / wikibase scalability.Jan 28 2020, 4:14 PM
Addshore added a project: Wikidata-Campsite.
Addshore moved this task from Incoming to Blocked / Waiting / External on the Wikidata-Campsite board.
DD063520 added a comment.EditedJan 28 2020, 4:16 PM

Hi,

I think I use a bot account. I created it using Special Pages / Bot passwords. And I set high volume editing. In this scenario, is the rate limit not set? i.e. should I be able to edit more?

Merci
D063520

Addshore added a comment.Jan 28 2020, 4:16 PM

I can advice you give these pages a read:

is there a maximum edit limit for the wiki per user?

See above

Is it a matter of infrastructure?

Infrastructure can also be a limiting factor

If I could make 500-1000 edits per minute would be great! Do you think this is feasible?

Yes, but probably not with only 1 mediawiki / wikibase instance running

Addshore added a comment.Jan 28 2020, 4:16 PM
In T243844#5836700, @DD063520 wrote:

Hi,

I think I use a bot account. I created it using Special Pages / Bot passwords. And I set high volume editing. In this scenario, is the rate limit not set?

Merci
D063520

Is this talking about Wikidata.org or your own wikibase?

DD063520 added a comment.Jan 28 2020, 4:16 PM

My own wikibase .....

DD063520 added a comment.Jan 28 2020, 4:21 PM

Thank you for these pointers:

https://www.mediawiki.org/wiki/Manual:Edit_throttling
https://www.mediawiki.org/wiki/Manual:$wgRateLimits

I more or less understand. Question .... let's say I set 500 edits per minute per user? How can I monitor that it is not too much and that I do not break the installation?

Bugreporter added a comment.Jan 28 2020, 4:26 PM

You can remove the limit entirely, if you are watching your server's health.

DD063520 added a comment.Jan 28 2020, 4:26 PM

what you mean with server's health? that the cpu is not always 100%?

Bugreporter added a comment.Jan 28 2020, 4:27 PM
In T243844#5836700, @DD063520 wrote:

Hi,

I think I use a bot account. I created it using Special Pages / Bot passwords. And I set high volume editing. In this scenario, is the rate limit not set? i.e. should I be able to edit more?

Merci
D063520

"Bot passwords" does not mean your account have bot flag. See https://www.mediawiki.org/wiki/Manual:Bots for what it means.

Addshore added a comment.Jan 29 2020, 3:06 PM
In T243844#5836717, @DD063520 wrote:

what you mean with server's health? that the cpu is not always 100%?

Yes, I mean, at some point your resources will just not allow clients to edit any faster.
This could be your database gets overloaded with writes, this could be the PHP mediawiki server can not deal with any more requests, or many other things.

Addshore closed this task as Resolved.Feb 3 2020, 1:45 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL