Page MenuHomePhabricator

Add user agent to Wikidata Query UI code examples
Closed, ResolvedPublic

Description

As a developer, I want to use the code examples provided by the Wikidata Query UI in order to use the Wikidata Query Service from my tool/bot/script/whatever, without running afoul of any Wikimedia policies or service constraints.

Problem:
The code examples generated by the Wikidata Query UI do not set any special User-Agent header (except for the Perl example, which prepends “MyApp/0.1 ” to the default user agent). This violates the User-Agent policy and can result in the script getting blocked (for example, the default python-requests user agent is currently blocked completely.)

Acceptance criteria:

  • Every code example sets a custom user agent
  • The code advises the developer to adjust that user agent further (e. g. via a TODO comment)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 27 2019, 12:22 PM

Untested, but I believe the correct code should be:

def get_results(endpoint_url, query):
    sparql = SPARQLWrapper(endpoint_url, agent='MyCoolTool/0.1 foo@example.org')
    sparql.setQuery(query)
    sparql.setReturnFormat(JSON)
    return sparql.query().convert()

Based off of https://github.com/RDFLib/sparqlwrapper/blob/master/SPARQLWrapper/Wrapper.py#L446

Where should I send a patch to?

abian added a subscriber: abian.Jul 10 2019, 7:49 PM

The pywikibot code should handle this already. Should a comment be added to the example snippet to make that explicit?

The pywikibot code should handle this already. Should a comment be added to the example snippet to make that explicit?

A comment to indicate where to adjust the user agent would be nice, it took me a while just to find the code that does it (pywikibot/comms/http.py, I think?) and I still wouldn’t be sure how to adjust it if I was running my own bot (but perhaps that’s just because I’m not very familiar with pywikibot).

But the comment in the task description is just from me searching for things like “user agent” in the code samples and not finding anything outside of Ruby – the pywikibot default already sounds better than nothing, so that’s not as urgent.

Change 523172 had a related patch set uploaded (by Lokal Profil; owner: Lokal Profil):
[wikidata/query/gui@master] Add user-agent note to pywikibot code example

https://gerrit.wikimedia.org/r/523172

Change 523172 merged by jenkins-bot:
[wikidata/query/gui@master] Add user-agent note to pywikibot code example

https://gerrit.wikimedia.org/r/523172

Change 538375 had a related patch set uploaded (by Seb35; owner: Seb35):
[wikidata/query/gui@master] Fix tests for "Add user-agent note to pywikibot code example"

https://gerrit.wikimedia.org/r/538375

Change 538375 merged by jenkins-bot:
[wikidata/query/gui@master] Fix tests for "Add user-agent note to pywikibot code example"

https://gerrit.wikimedia.org/r/538375

Change 538592 had a related patch set uploaded (by WDQSGuiBuilder; owner: WDQSGuiBuilder):
[wikidata/query/gui-deploy@production] Merging from 37ea21e226ec3dcf654f3b2c19adc1edc9117a3c:

https://gerrit.wikimedia.org/r/538592

Change 552541 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[wikidata/query/gui@master] Add default user agent to some more code examples

https://gerrit.wikimedia.org/r/552541

Change 552541 merged by jenkins-bot:
[wikidata/query/gui@master] Add default user agent to some more code examples

https://gerrit.wikimedia.org/r/552541

Addshore closed this task as Resolved.Dec 3 2019, 1:51 PM
Addshore claimed this task.
Restricted Application added a project: User-Addshore. · View Herald TranscriptDec 3 2019, 1:51 PM
Lucas_Werkmeister_WMDE added a subscriber: Addshore.

Still needs to be deployed, though. (And does it make sense to have a Done task in the Campsite’s Verification column?)

Addshore reopened this task as Open.Dec 5 2019, 10:07 AM

Will reopen until we can verify it in production.

(And does it make sense to have a Done task in the Campsite’s Verification column?)

Many closed tasks end up being left all over the place https://phabricator.wikimedia.org/project/board/3539/?filter=LO033Mf5mq7L

Addshore changed the task status from Open to Stalled.Dec 23 2019, 1:00 PM

Still not yet deployed.

Just checked back on this and still waiting for a deployment :)

I’m not sure what the current state of wdqs-gui deployment is – I saw @dcausse and @Mathew.onipe push several new snapshot versions to the repo, but I guess those don’t have any effect?

New build available at https://gerrit.wikimedia.org/r/562497, if anyone wants to merge+deploy it.

Just checked again and this is still not deployed.
Probably because of T242640

I see no reason why T242640: query/wikidata/gui jenkins build broken should block deploys, since lately we’ve been building the deploy commit manually anyways.

In fact https://gerrit.wikimedia.org/r/562497 is already merged, so I think all that’s missing is a git pull and scap sync on the deployment server? Or something along those lines, at least.

4:12 PM <addshore> gehel: did the build being broken stop the query service deploy by any chance?
4:13 PM <addshore> Just seeing when those UI changes we had could be expected and what if anything is blocking them
4:13 PM <+gehel> dcausse, zpapierski: ^^
4:13 PM <addshore> =]
4:13 PM <+gehel> addshore: yep, we've delayed the release until this build is fixed
4:13 PM <addshore> Thanks, I'll pop that in my ticket!
4:13 PM <+gehel> we have the option to just drop the GUI submodule right know as well

So for now blocked on the build being broken.
But also might accelerate the detaching of the UI and backend.

Addshore triaged this task as Low priority.Jan 14 2020, 4:20 PM
Silvan_WMDE closed this task as Resolved.Feb 6 2020, 11:38 AM
Silvan_WMDE added a subscriber: Silvan_WMDE.

deployed and verified:

  • PHP, Java, Perl, Python, Ruby, R, Matlab are covered,
  • URL, HTML, Wikilink, listeria are not applicable,
  • JavaScript (jQuery and modern) doesn’t specify a user agent because the browser’s default user agent must be used.