Page MenuHomePhabricator

Enable different URL shorteners for WDQS
Closed, ResolvedPublic

Assigned To
Authored By
Smalyshev
Sep 15 2015, 11:47 PM
Referenced Files
None
Tokens
"Like" token, awarded by deryckchan."Love" token, awarded by WalterKlossew."Like" token, awarded by Ivanhercaz."Love" token, awarded by Liuxinyu970226."Love" token, awarded by Pintoch.

Description

WDQS features ability to produce query short URL via URL shortener. Unfortunately, tinyurl.com is banned in wikidata.org space because of the spam, so we may want to support another one like us.wmflabs.org

See: https://www.wikidata.org/wiki/Wikidata:Project_chat#URL_shortener.3F

Related Objects

StatusSubtypeAssignedTask
ResolvedLadsgroup
ResolvedBBlack
ResolvedSmalyshev
ResolvedLadsgroup
Resolvedmatmarex
Resolvedmatmarex
ResolvedLegoktm
Resolvedmatmarex
DuplicateNone
Resolvedori
ResolvedBBlack
ResolvedBBlack
Resolved dpatrick
Resolved Prtksxna
ResolvedLegoktm
ResolvedLegoktm
ResolvedArielGlenn
ResolvedJoe
ResolvedBBlack
ResolvedSmalyshev
ResolvedLadsgroup
ResolvedMarostegui
ResolvedLegoktm
Resolved Prtksxna
ResolvedLadsgroup
DeclinedNone
DeclinedNone
ResolvedSmalyshev

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

We just got a demo of https://w.wiki/2 so this task can probably be started soon!

As soon as URL creation is enabled we can support it :)

Instead of a URL shortener, wouldn't a Quarry-like (quarry.wmflabs.org) interface a solution? It would permit to save queries, share them, even fix typos in them without changing the public URL.

Such kind of interface certainly wouldn't hurt but I think having short URLs is not entirely the same. It's a bit simpler case.

I think both ways are desirable. Quarry-like interface for WDQS is already tasked at T104762. Quarry is exteremely useful in lots of ways. The easy typo fixing or code commenting without URL change as mentioned, being able to promise someone to write a query and provide a link there beforehand (much like with "red" pages, you can say "hey, look here tomorrow, there'll be something really cool"), being able to reuse old URLs which you needed only to show some number in a chat which noone would care about in 5 minutes and lots of other stuff.

URL shortening on the other hand is open to all, not just registered users, [considering current Quarry shortcomings] allows not just query author to run it and so forth.

We definitely would benefit from having both is what I believe.

Instead of a URL shortener, wouldn't a Quarry-like (quarry.wmflabs.org) interface a solution? It would permit to save queries, share them, even fix typos in them without changing the public URL.

There was some more discussion about this at https://phabricator.wikimedia.org/T108557#3056937

Note that similar service provide their own URL shortening, such as osm.org short links (which IMHO make for a very good sharing feature, better than alternatives such as the huge Google Maps URLs) or http://overpass-turbo.eu/s/ice which exists alongside the format http://overpass-turbo.eu/?Q=node%0A%20%20%5Bamenity%3Ddrinking_water%5D%0A%20%20(%7B%7Bbbox%7D%7D)%3B%0Aout%3B&C=45.46976;9.18637;13&R

I'd rather avoid reinventing the wheel and baking (and maintaining) yet another homebrew URL shortening scheme when there's so many existing solutions for it. It's not like it is hard to do - it just needs to be done.

Just cheer to the guy who set up the tinyurl.com links, a blacklisted domain on wikipedia. Looks terribly logic that it's still active...

Smalyshev added a project: User-Smalyshev.
Smalyshev moved this task from Backlog to Waiting/Blocked on the User-Smalyshev board.

Personally I still don't understand why the Wikidata-query-service doesn't generate their own shortened urls and own the resolution. It shouldn't have external security issues, and less in the way of abuse. Magnus has done that with petscan. Waiting for the broader WMF to resolve this is where there is no evident poltical will or impetus is becomng ... <crickets>.

If we can't get it unstuck then yes. However the dev team can't put time into it right now unfortunately :/

I'd rather not create a duplicate service for something we already have perfectly good implementation for, and then have to support it in production. Everybody agrees we need this service, and not only for WDQS. The right thing to do is not to concoct some homebrew half-baked version of it for each special use case but getting it working in a generic case. Let's do the right thing.

Instead of a URL shortener, wouldn't a Quarry-like (quarry.wmflabs.org) interface a solution? It would permit to save queries, share them, even fix typos in them without changing the public URL.

There was some more discussion about this at https://phabricator.wikimedia.org/T108557#3056937

Note that similar service provide their own URL shortening, such as osm.org short links

FWIW, people are using petscan to share short links to save SPARQL queries, e.g. http://petscan.wmflabs.org/?psid=5653431#num_results

What about installing on wmflabs an existing url shortener service.
Examples https://github.com/YOURLS/YOURLS || https://github.com/cydrobolt/polr

@Sabas88 may be a viable interim solution if somebody volunteered to set up and install it. If nobody does, I might do it in my copious free time, but no specific promises.

Note that this system should be limited to wikimedia URLs only (not sure whether the solutions above allow it, if not, one should choose the solution which does), otherwise it will be banned on Wiki very soon, and also have an active maintainer willing to deal with abuse, for the same reason.

I just ran into a problem with the URL shortener: my query was simply too long to be crunched by TinyURL (and most other URL shorteners for that matter, the only one I tried that worked was cpc.cx)

(This is IMHO one further proof that the ability to save queries (like we can do on quarry) would be better than shortening URLs.)

Quick update: The amazing @Ladsgroup is currently putting time into the big remaining blocker: T133109: Add basic abuse prevention to UrlShortener

I wonder does UrlShortener require login? If yes, we'd have to add OAuth setup to WDQS GUI I presume for this URL shortener - which means, the code needs to become way smarter with regard to how URL shorteners work, because some may require login and some not.

Also I think if we had some Labs setup where UrlShortener is already enabled this would probably allow to work on developing the above in parallel, not waiting for UrlShortener be officially deployed in production.

If we enable CORS between wikidata query service and Wikimedia projects, it shouldn't need any OAuth dance.

I created a separate task to enable CORS for query.wikidata.org: T218568

Change 500833 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/gui@master] Refactor URL shortener as separate API

https://gerrit.wikimedia.org/r/500833

Change 503071 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/gui@master] Add WMF url shortener on Meta

https://gerrit.wikimedia.org/r/503071

Change 500833 merged by jenkins-bot:
[wikidata/query/gui@master] Refactor URL shortener as separate API

https://gerrit.wikimedia.org/r/500833

Change 503071 merged by jenkins-bot:
[wikidata/query/gui@master] Add WMF url shortener on Meta

https://gerrit.wikimedia.org/r/503071

It looks nice but when I click on the short URL, I can't copy the short link, the popup goes away when I hover it.

@Ladsgroup which one? With the query (top left) or the results (down right in the menu)? In any case, probably makes sense to open new task with a screen shot.

Just tested, thanks for bringing the feature to the QS <3
I couldn't reproduce the issue @Ladsgroup mentions.

Question; what kind of error message is displayed when the URL is too long?

So there is no difference between "failed because URL is too long" and "failed because so many people use the service that it's been blocked by rate limit" ^^

It looks nice but when I click on the short URL, I can't copy the short link, the popup goes away when I hover it.

Same for me (Firefox 66), I have to hold the left click and ctrl-c.

So there is no difference between "failed because URL is too long" and "failed because so many people use the service that it's been blocked by rate limit"

Yes, error reporting right now is most rudimentary. We probably should fix that.

P.S. Created T221126 for that.