Page MenuHomePhabricator

Enable different URL shorteners for WDQS
Closed, ResolvedPublic

Description

WDQS features ability to produce query short URL via URL shortener. Unfortunately, tinyurl.com is banned in wikidata.org space because of the spam, so we may want to support another one like us.wmflabs.org

See: https://www.wikidata.org/wiki/Wikidata:Project_chat#URL_shortener.3F

Related Objects

StatusAssignedTask
ResolvedLadsgroup
ResolvedBBlack
ResolvedSmalyshev
ResolvedLadsgroup
Resolvedmatmarex
Resolvedmatmarex
ResolvedLegoktm
Resolvedmatmarex
DuplicateNone
Resolvedori
ResolvedBBlack
ResolvedBBlack
Resolved dpatrick
ResolvedPrtksxna
ResolvedLegoktm
ResolvedLegoktm
ResolvedArielGlenn
ResolvedJoe
ResolvedBBlack
ResolvedSmalyshev
ResolvedLadsgroup
ResolvedMarostegui
ResolvedLegoktm
ResolvedPrtksxna
ResolvedLadsgroup
DeclinedNone
DeclinedNone
ResolvedSmalyshev

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

We just got a demo of https://w.wiki/2 so this task can probably be started soon!

As soon as URL creation is enabled we can support it :)

debt added a subscriber: debt.Jun 28 2016, 11:08 AM
Jklamo added a subscriber: Jklamo.Sep 20 2016, 6:48 PM

Instead of a URL shortener, wouldn't a Quarry-like (quarry.wmflabs.org) interface a solution? It would permit to save queries, share them, even fix typos in them without changing the public URL.

Such kind of interface certainly wouldn't hurt but I think having short URLs is not entirely the same. It's a bit simpler case.

Base added a subscriber: Base.Dec 21 2016, 2:36 PM

I think both ways are desirable. Quarry-like interface for WDQS is already tasked at T104762. Quarry is exteremely useful in lots of ways. The easy typo fixing or code commenting without URL change as mentioned, being able to promise someone to write a query and provide a link there beforehand (much like with "red" pages, you can say "hey, look here tomorrow, there'll be something really cool"), being able to reuse old URLs which you needed only to show some number in a chat which noone would care about in 5 minutes and lots of other stuff.

URL shortening on the other hand is open to all, not just registered users, [considering current Quarry shortcomings] allows not just query author to run it and so forth.

We definitely would benefit from having both is what I believe.

Instead of a URL shortener, wouldn't a Quarry-like (quarry.wmflabs.org) interface a solution? It would permit to save queries, share them, even fix typos in them without changing the public URL.

There was some more discussion about this at https://phabricator.wikimedia.org/T108557#3056937

Note that similar service provide their own URL shortening, such as osm.org short links (which IMHO make for a very good sharing feature, better than alternatives such as the huge Google Maps URLs) or http://overpass-turbo.eu/s/ice which exists alongside the format http://overpass-turbo.eu/?Q=node%0A%20%20%5Bamenity%3Ddrinking_water%5D%0A%20%20(%7B%7Bbbox%7D%7D)%3B%0Aout%3B&C=45.46976;9.18637;13&R

I'd rather avoid reinventing the wheel and baking (and maintaining) yet another homebrew URL shortening scheme when there's so many existing solutions for it. It's not like it is hard to do - it just needs to be done.

Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 25 2017, 11:25 PM

Just cheer to the guy who set up the tinyurl.com links, a blacklisted domain on wikipedia. Looks terribly logic that it's still active...

abian added a subscriber: abian.Jan 19 2018, 10:38 PM
Ivanhercaz added a subscriber: Ivanhercaz.
Smalyshev removed Smalyshev as the assignee of this task.Jan 30 2018, 8:06 AM
Smalyshev added a project: User-Smalyshev.
Smalyshev moved this task from Backlog to Waiting/Blocked on the User-Smalyshev board.

Personally I still don't understand why the Wikidata-query-service doesn't generate their own shortened urls and own the resolution. It shouldn't have external security issues, and less in the way of abuse. Magnus has done that with petscan. Waiting for the broader WMF to resolve this is where there is no evident poltical will or impetus is becomng ... <crickets>.

If we can't get it unstuck then yes. However the dev team can't put time into it right now unfortunately :/

I'd rather not create a duplicate service for something we already have perfectly good implementation for, and then have to support it in production. Everybody agrees we need this service, and not only for WDQS. The right thing to do is not to concoct some homebrew half-baked version of it for each special use case but getting it working in a generic case. Let's do the right thing.

Instead of a URL shortener, wouldn't a Quarry-like (quarry.wmflabs.org) interface a solution? It would permit to save queries, share them, even fix typos in them without changing the public URL.

There was some more discussion about this at https://phabricator.wikimedia.org/T108557#3056937
Note that similar service provide their own URL shortening, such as osm.org short links

FWIW, people are using petscan to share short links to save SPARQL queries, e.g. http://petscan.wmflabs.org/?psid=5653431#num_results

Sabas88 added a subscriber: Sabas88.Sep 5 2018, 8:09 AM

What about installing on wmflabs an existing url shortener service.
Examples https://github.com/YOURLS/YOURLS || https://github.com/cydrobolt/polr

@Sabas88 may be a viable interim solution if somebody volunteered to set up and install it. If nobody does, I might do it in my copious free time, but no specific promises.

Note that this system should be limited to wikimedia URLs only (not sure whether the solutions above allow it, if not, one should choose the solution which does), otherwise it will be banned on Wiki very soon, and also have an active maintainer willing to deal with abuse, for the same reason.

WalterKlossew awarded a token.

I created a proposal on Community Wishlist Survey for URL shortener for Wikimedia wikis.

I created a proposal on Community Wishlist Survey for URL shortener for Wikimedia wikis.

Please see T108557

I just ran into a problem with the URL shortener: my query was simply too long to be crunched by TinyURL (and most other URL shorteners for that matter, the only one I tried that worked was cpc.cx)

(This is IMHO one further proof that the ability to save queries (like we can do on quarry) would be better than shortening URLs.)

Quick update: The amazing @Ladsgroup is currently putting time into the big remaining blocker: T133109: Add basic abuse prevention to UrlShortener

Smalyshev added a comment.EditedMar 17 2019, 7:04 PM

I wonder does UrlShortener require login? If yes, we'd have to add OAuth setup to WDQS GUI I presume for this URL shortener - which means, the code needs to become way smarter with regard to how URL shorteners work, because some may require login and some not.

Also I think if we had some Labs setup where UrlShortener is already enabled this would probably allow to work on developing the above in parallel, not waiting for UrlShortener be officially deployed in production.

If we enable CORS between wikidata query service and Wikimedia projects, it shouldn't need any OAuth dance.

I created a separate task to enable CORS for query.wikidata.org: T218568

Change 500833 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/gui@master] Refactor URL shortener as separate API

https://gerrit.wikimedia.org/r/500833

Change 503071 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/gui@master] Add WMF url shortener on Meta

https://gerrit.wikimedia.org/r/503071

Change 500833 merged by jenkins-bot:
[wikidata/query/gui@master] Refactor URL shortener as separate API

https://gerrit.wikimedia.org/r/500833

Change 503071 merged by jenkins-bot:
[wikidata/query/gui@master] Add WMF url shortener on Meta

https://gerrit.wikimedia.org/r/503071

Smalyshev closed this task as Resolved.Apr 15 2019, 6:41 PM
Smalyshev claimed this task.

It looks nice but when I click on the short URL, I can't copy the short link, the popup goes away when I hover it.

@Ladsgroup which one? With the query (top left) or the results (down right in the menu)? In any case, probably makes sense to open new task with a screen shot.

Just tested, thanks for bringing the feature to the QS <3
I couldn't reproduce the issue @Ladsgroup mentions.

Question; what kind of error message is displayed when the URL is too long?

“URL shortening failed”

So there is no difference between "failed because URL is too long" and "failed because so many people use the service that it's been blocked by rate limit" ^^

It looks nice but when I click on the short URL, I can't copy the short link, the popup goes away when I hover it.

Same for me (Firefox 66), I have to hold the left click and ctrl-c.

Smalyshev added a comment.EditedApr 16 2019, 5:23 PM

So there is no difference between "failed because URL is too long" and "failed because so many people use the service that it's been blocked by rate limit"

Yes, error reporting right now is most rudimentary. We probably should fix that.

P.S. Created T221126 for that.