Maniphest T204471

WikibaseQualityConstraints include extension name in UA for query service requests
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

In T204267 the extension flooded the query service with requests and ended up getting all mediawiki requests banned.
In the future other less floody SPARQL requests may come from mediawiki and thus have the same UA & get banned (unless the banning is done by IP only?)
In order to avoid these blankety bans each sub product / service within mediawiki making the requests should add its own product to the UA in addition to the default mediawiki one.

We should also include the name of the wiki that the request is coming from. This is planning for a future where QualityConstraints may also be run on another wikibase instance within the WMF cluster (for example commons).

Current example UA:

MediaWiki/1.32.0-wmf.20

Proposed UA:

MediaWiki/1.32.0-wmf.20 WikibaseQualityConstraints

User-Agent RFC: https://tools.ietf.org/html/rfc7231#section-5.5.3

Pointers

The option probably needs to be added to https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikibaseQualityConstraints/+/refs/heads/master/src/ConstraintCheck/Helper/SparqlHelper.php#583 however other usage should be checked..

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Set custom user agent in SparqlHelper	Ladsgroup	mediawiki/extensions/WikibaseQualityConstraints	master	+14 -4

Customize query in gerrit

Related Objects

Mentioned Here: T204267: Flood of WDQS requests from wbqc

Event Timeline

Addshore created this task.Sep 17 2018, 7:52 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 17 2018, 7:52 AM

@Smalyshev In order to decide if there is any point in this one....

Does the query service actually differentiate requests from a single IP with different UAs for rate limiting / banning?

Addshore added a project: [DEPRECATED] wdwb-tech.Sep 17 2018, 7:53 AM

Addshore moved this task from Inbox to Investigate & Discuss on the [DEPRECATED] wdwb-tech board.

Addshore moved this task from incoming to needs discussion or investigation on the Wikidata board.Sep 17 2018, 8:11 AM

TBD / TBA does the query service actually differentiate requests from a single IP with different UAs?

That sounds like a really easy way for any spammer to circumvent a ban, so I would be very surprised if we actually did this tbh.

Proposed UA:

MediaWiki/1.32.0-wmf.20 WikibaseQualityConstraints

I would think that WikibaseQualityConstraints should come first, since MediaWiki acts more like the “library” component here:

WikibaseQualityConstraints MediaWiki/1.32.0-wmf.20

Rate limiting is bucketing based on IP+User agent, so having distinctive user agent for WBQC is certainly a good idea.

Addshore updated the task description. (Show Details)Sep 18 2018, 9:08 AM

Addshore moved this task from Investigate & Discuss to Goals on the [DEPRECATED] wdwb-tech board.

Addshore moved this task from needs discussion or investigation to ready to go on the Wikidata board.

Addshore added a project: Wikidata-Campsite.

Addshore moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.

Lucas_Werkmeister_WMDE updated the task description. (Show Details)Sep 18 2018, 1:43 PM

Lucas_Werkmeister_WMDE set the point value for this task to 5.

Lucas_Werkmeister_WMDE added a project: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)).

Addshore moved this task from Ready to estimate to In Progress on the Wikidata-Campsite board.Sep 18 2018, 1:48 PM

Addshore moved this task from ready to go to in progress on the Wikidata board.Sep 18 2018, 2:23 PM

Change 463923 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/WikibaseQualityConstraints@master] Set custom user agent in SparqlHelper

https://gerrit.wikimedia.org/r/463923

gerritbot added a project: Patch-For-Review.Oct 2 2018, 10:09 AM

Ladsgroup claimed this task.Oct 2 2018, 10:11 AM

Ladsgroup moved this task from To Do (prioritised from top to bottom) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptOct 2 2018, 10:11 AM

Ladsgroup moved this task from Incoming to Blocked on others on the User-Ladsgroup board.Oct 2 2018, 8:19 PM

• Jonas moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Oct 4 2018, 10:08 AM

Ladsgroup moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Oct 4 2018, 11:32 AM

Addshore moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Oct 4 2018, 12:36 PM

Change 463923 merged by jenkins-bot:
[mediawiki/extensions/WikibaseQualityConstraints@master] Set custom user agent in SparqlHelper

https://gerrit.wikimedia.org/r/463923

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)).Oct 4 2018, 2:00 PM

This should be verified once deployed.
This should be deployed on the evening of the 17th October.
We can verify by making sure the query service gets requests with the new UA

Addshore moved this task from Goals to Triaged High (100+) on the [DEPRECATED] wdwb-tech board.Oct 8 2018, 12:46 PM

Addshore removed a project: Wikidata-Campsite.Nov 1 2018, 11:21 AM

	F27078771: image.png
	Nov 6 2018, 12:06 PM

WikibaseQualityConstraints include extension name in UA for query service requestsClosed, ResolvedPublic5 Estimated Story PointsActions

Description

Details

Related Objects

Event Timeline

WikibaseQualityConstraints include extension name in UA for query service requests
Closed, ResolvedPublic5 Estimated Story Points
Actions