Page MenuHomePhabricator
Create Task
Maniphest T139618

Petscan is being used with excesive parallelism by a user on Wikidata
Closed, ResolvedPublic
Actions

Description

See the background: https://www.wikidata.org/wiki/User_talk:GZWDer#GZWDer_.28flood.29

The tool was creating 300-400 new pages per minute, so many that it was creating too many database locks with itself.

According to the user, it run with 5 parallel threads- that is not ok for our API- whose etiquette suggest to do only serial requests.

While it was not causing major infrastructure issues, it creates problems for other users. The users, however, cannot change the source code as this is a hosted tool. Can this be limited in any way? Globally? Per user? Per wiki? Per shard?

Related Objects

Event Timeline

jcrespo created this task.Jul 7 2016, 3:22 PM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJul 7 2016, 3:22 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald Transcript
jcrespo added projects: DBA, Wikidata.Jul 7 2016, 3:23 PM
yuvipanda subscribed.Jul 7 2016, 3:23 PM

@Magnus since I think he wrote petscan?

jcrespo moved this task from incoming to monitoring on the Wikidata board.Jul 7 2016, 3:40 PM
Magnus added a comment.EditedJul 7 2016, 4:00 PM

The five threads are for bot accounts only. Normal user accounts get single thread with delay.
I have used my own bot account a lot over the years, with previous tools (also 5 threads) and PetScan, without issue.
As far as I understand, GZWDer used (at least) two tabs running in parallel. Even if I reduce that to single thread, he could just open 10 tabs. This is not a code issue, but a user issue.

jcrespo added a comment.Jul 7 2016, 4:07 PM

@Magnus, as you can see on the discussion I agreed with you initially, and in no way I am giving you any responsibility for this particular incident. However, the user says he doesn't have the *option* of running it slower, in a single thread, if I understood correctly.

So consider this a *feature request* from that user, which I happen to be indirectly interested in.

Magnus added a comment.Jul 7 2016, 5:18 PM

Just run it as a normal user and not a bot user!

Sjoerddebruin subscribed.Jul 7 2016, 5:19 PM
Bugreporter subscribed.Jul 7 2016, 5:51 PM
jcrespo moved this task from Triage to Blocked external/Not db team on the DBA board.Jul 7 2016, 5:52 PM

I will see what the user responds, and act depending on it.

Bugreporter added a comment.Jul 7 2016, 5:58 PM

Running it as a normal user will flood recent change, See https://www.wikidata.org/wiki/Wikidata:Administrators%27_noticeboard/Archive/2014/05#Flooding_of_Special:RecentChanges

However the current Recent Changes is worse than in 2014

Bugreporter added a comment.EditedJul 7 2016, 6:03 PM

I try to limit the negative effect of running the tools.

At the beginning at most 6-7 tabs are running. Then I keep only one tab after warning. Now I use (at most) two tabs and I want jcrespo report any issue to my talk page. Currently nothing occurs.

Bugreporter added a subscriber: • Jura1.Jul 7 2016, 6:03 PM

oops this user is not active at phabricator.

Magnus added a comment.Jul 7 2016, 6:08 PM

" 6-7 tabs are run"

I believe we found the root problem :-)

Bugreporter added a comment.EditedJul 7 2016, 6:10 PM

This is previously, as I didn't know how many tabs should be run at most, and what problem would occur when too many tabs are running. After warning at most two tabs are running.

jcrespo added a comment.Jul 7 2016, 6:10 PM

I think we agreed to use only one "tab" at a time to follow API:Etiquette. I will block all your queries if they continue producing errors in the next 10 minutes, as I have warned you 3 or 4 times.

Bugreporter added a comment.EditedJul 7 2016, 6:12 PM

Currently (and since 20+ minutes ago) only one is running.

jcrespo closed this task as Resolved.Jul 7 2016, 6:13 PM
jcrespo claimed this task.
jcrespo mentioned this in T139636: Wikidata Database contention under high edit rate.

Thank you. I see lower amount of errors in the last 20 minutes. I will be monitoring the logs in case the errors return.

jcrespo renamed this task from Petscan is running too fast for Wikidata to Petscan is being used with excesive parallelism by a user on Wikidata.Jul 7 2016, 6:21 PM
zhuyifei1999 subscribed.Jul 8 2016, 4:35 AM
jcrespo reopened this task as Open.Jul 11 2016, 8:57 AM

This is still ongoing.

Bugreporter added a comment.Jul 11 2016, 9:04 AM

Now only one tab is running.

Bugreporter added a comment.Jul 11 2016, 9:07 AM

As there're issue even if there're only one tab, code must be modified.

Bugreporter changed the task status from Open to Stalled.Jul 11 2016, 9:34 AM

There're nothing I can do other than stop creating items (and probably other semi-automatic work) until https://bitbucket.org/magnusmanske/petscan/issues/48/make-number-of-threads-an-option-for-bot is fixed.

Magnus closed this task as Resolved.Jul 11 2016, 10:48 AM

Added option for bot-account users to set concurrent threads (1-5).
https://bitbucket.org/magnusmanske/petscan/commits/dbefbbd3dab0

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL