Page MenuHomePhabricator

Add user agent to Wikidata Query UI code examples
Open, Needs TriagePublic

Description

As a developer, I want to use the code examples provided by the Wikidata Query UI in order to use the Wikidata Query Service from my tool/bot/script/whatever, without running afoul of any Wikimedia policies or service constraints.

Problem:
The code examples generated by the Wikidata Query UI do not set any special User-Agent header (except for the Perl example, which prepends “MyApp/0.1 ” to the default user agent). This violates the User-Agent policy and can result in the script getting blocked (for example, the default python-requests user agent is currently blocked completely.)

Acceptance criteria:

  • Every code example sets a custom user agent
  • The code advises the developer to adjust that user agent further (e. g. via a TODO comment)

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 27 2019, 12:22 PM

Untested, but I believe the correct code should be:

def get_results(endpoint_url, query):
    sparql = SPARQLWrapper(endpoint_url, agent='MyCoolTool/0.1 foo@example.org')
    sparql.setQuery(query)
    sparql.setReturnFormat(JSON)
    return sparql.query().convert()

Based off of https://github.com/RDFLib/sparqlwrapper/blob/master/SPARQLWrapper/Wrapper.py#L446

Where should I send a patch to?

abian added a subscriber: abian.Jul 10 2019, 7:49 PM

The pywikibot code should handle this already. Should a comment be added to the example snippet to make that explicit?

The pywikibot code should handle this already. Should a comment be added to the example snippet to make that explicit?

A comment to indicate where to adjust the user agent would be nice, it took me a while just to find the code that does it (pywikibot/comms/http.py, I think?) and I still wouldn’t be sure how to adjust it if I was running my own bot (but perhaps that’s just because I’m not very familiar with pywikibot).

But the comment in the task description is just from me searching for things like “user agent” in the code samples and not finding anything outside of Ruby – the pywikibot default already sounds better than nothing, so that’s not as urgent.

Change 523172 had a related patch set uploaded (by Lokal Profil; owner: Lokal Profil):
[wikidata/query/gui@master] Add user-agent note to pywikibot code example

https://gerrit.wikimedia.org/r/523172

Change 523172 merged by jenkins-bot:
[wikidata/query/gui@master] Add user-agent note to pywikibot code example

https://gerrit.wikimedia.org/r/523172

Change 538375 had a related patch set uploaded (by Seb35; owner: Seb35):
[wikidata/query/gui@master] Fix tests for "Add user-agent note to pywikibot code example"

https://gerrit.wikimedia.org/r/538375

Change 538375 merged by jenkins-bot:
[wikidata/query/gui@master] Fix tests for "Add user-agent note to pywikibot code example"

https://gerrit.wikimedia.org/r/538375

Change 538592 had a related patch set uploaded (by WDQSGuiBuilder; owner: WDQSGuiBuilder):
[wikidata/query/gui-deploy@production] Merging from 37ea21e226ec3dcf654f3b2c19adc1edc9117a3c:

https://gerrit.wikimedia.org/r/538592

Change 552541 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[wikidata/query/gui@master] Add default user agent to some more code examples

https://gerrit.wikimedia.org/r/552541

Change 552541 merged by jenkins-bot:
[wikidata/query/gui@master] Add default user agent to some more code examples

https://gerrit.wikimedia.org/r/552541

Addshore closed this task as Resolved.Tue, Dec 3, 1:51 PM
Addshore claimed this task.
Restricted Application added a project: User-Addshore. · View Herald TranscriptTue, Dec 3, 1:51 PM
Lucas_Werkmeister_WMDE added a subscriber: Addshore.

Still needs to be deployed, though. (And does it make sense to have a Done task in the Campsite’s Verification column?)

Addshore reopened this task as Open.Thu, Dec 5, 10:07 AM

Will reopen until we can verify it in production.

(And does it make sense to have a Done task in the Campsite’s Verification column?)

Many closed tasks end up being left all over the place https://phabricator.wikimedia.org/project/board/3539/?filter=LO033Mf5mq7L