Page MenuHomePhabricator

Split out the 3 standalone tools from the DMCA toolbox and deploy them on tool labs
Closed, ResolvedPublic5 Story Points

Description

The code is at https://github.com/jamesryanalexander/tools-standalone/tree/776e429770615ee9f2bb6f1e39186af57fe6c42e

The 3 tools are:

  • globalsearch.php - Tool for searching for a phrase on all wikimedia wikis, currently requires and uses mediawiki OAuth.
  • globallinks.php - Tool for searching for a specific url linked from any wiki.
  • globalpagetext.php - Tool to show the text on a specific page given across all wikis (usually used to compare the status of a mediaWiki interface page).
What this task includes:
  • Splitting out the requisite code (but not doing any refactoring unless absolutely necessary, we can do that separately if needed)
  • Adding some barebones bootstrap UI for them
  • Deploying on tool labs (all 3 as part of the same tool)

Note: This task does not involve adding OAuth support for using the tools.

Event Timeline

Niharika created this task.May 26 2017, 5:37 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 26 2017, 5:37 PM
Niharika updated the task description. (Show Details)May 26 2017, 5:37 PM
Niharika added a project: Anti-Harassment.
DannyH triaged this task as Normal priority.May 31 2017, 5:11 PM
DannyH set the point value for this task to 5.
dbarratt claimed this task.Jun 7 2017, 7:42 PM
dbarratt edited projects, added Community-Tech-Sprint; removed Community-Tech.
dbarratt moved this task from Ready to In Development on the Community-Tech-Sprint board.

Where should the code live? Should I request a new repository on https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests ?

Also, there is no license in that code repository, so as far as I know, we don't have any right to modify it. :(

I've asked for a license to be added:
https://github.com/jamesryanalexander/tools-standalone/issues/1

Where should the code live? Should I request a new repository on https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests ?

I don't think you should modify it for this task. You can create a new repo under your account in Github and use the code in the standalone-tools folder and start from there.

For T167187: Epic ⚡️ : Implement DMCA and CP takedown report tools you can fork the current repo and work on it under your account.

You don't need to request Gerrit repositories for either, I think. You can keep working on it on Github.

You do need to request a tool-labs tool for the current task. You can read more about it here: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs

What should we call the "Service Group"?

Also, what user should I authenticate as (which seems to be an environment variable)? And where would I get a "consumer key"?

bd808 added a subscriber: bd808.Jun 9 2017, 2:48 AM

What should we call the "Service Group"?

I'd vote for "globalsearch" since that is what the three endpoints attempt to do.

bd808 added a comment.Jun 9 2017, 2:49 AM

Also, what user should I authenticate as (which seems to be an environment variable)? And where would I get a "consumer key"?

I think this can be done without auth at all. I don't think the Action API actions being used return anything different based on user permissions.

+1 to globalsearch and not using Oauth.

Also, is it within this ticket to remove the dependency on https://github.com/jamesryanalexander/tools-coreinclude ? (Or whichever other code this tool uses.) If not, that dependency should be refactored and loaded with Composer, I reckon.

So it looks like it uses OAuth in order to get a list of Wikimedia sites. Then once it has the list, it performs the search against each one of the site's API's and then merges the results. I'll see if I can track down where it's getting this list and if OAuth is really required or not.

I added Composer to the library so now it can be loaded (and autoloaded!)
https://github.com/davidbarratt/tools-coreinclude

Ah! https://www.mediawiki.org/wiki/Extension:SiteMatrix I am just learning more and more every day!

Welp! doesn't need OAuth. https://meta.wikimedia.org/w/api.php?action=sitematrix&format=json

Honestly I think this should be rewritten as a SPA in JavaScript. :)

@dbarratt: Looks good. A few loose ends to tie up:

  • Please add myself (kaldari) and Jalexander as maintainers of the Tool Labs project (via this page)
  • Please move the repo under https://github.com/wikimedia/ (this might involve folking the old repo and then deleting it)
  • Change the Tool Labs tool to pull from the new GitHub repo

Once that is done, feel free to mark as resolved and move to the Done column.

Welp! doesn't need OAuth. https://meta.wikimedia.org/w/api.php?action=sitematrix&format=json
Honestly I think this should be rewritten as a SPA in JavaScript. :)

Yeah, it should be, the ONLY reason it was using OAuth was because I could reuse shared code. It totally shouldn't be and was planned to be rewritten when moved out.

Done!
https://tools.wmflabs.org/globalsearch/
I also created T167544: Rewrite Global Search (globalsearch) in JavaScript because the tool should really be rewritten. :)

\o/

@dbarratt If you add me (https://github.com/bd808) as a "Collaborator" on the current GitHub repo you are using I can migrate it to the Wikimedia organization for you. The migration will leave a redirect for the current URLs that transparently forward to the new location. See https://www.mediawiki.org/wiki/Manual:Developing_libraries#Transferring_an_existing_GitHub_repo_to_Wikimedia for the procedure.

dbarratt added a comment.EditedJun 12 2017, 6:56 PM

@dbarratt If you add me (https://github.com/bd808) as a "Collaborator" on the current GitHub repo you are using I can migrate it to the Wikimedia organization for you.

@bd808 I added you as a collaborator on https://github.com/davidbarratt/tools-standalone and https://github.com/davidbarratt/tools-coreinclude

@bd808: Can you change the name of the repos while you're at it, to globalsearch-standalone and globalsearch-coreinclude?

@dbarratt: Why are there 2 different repos?

@dbarratt: Why are there 2 different repos?

tools-standalone has a dependency on tools-coreinclude which is here:
https://github.com/davidbarratt/tools-standalone/blob/master/composer.json#L7

However, now that I've removed OAuth, I think it's a single function that it's depending on, which I could just copy over if you would prefer.

However, now that I've removed OAuth, I think it's a single function that it's depending on, which I could just copy over if you would prefer.

That would be awesome if we could consolidate them.

bd808 added a comment.Jun 12 2017, 8:09 PM

@bd808: Can you change the name of the repos while you're at it, to globalsearch-standalone and globalsearch-coreinclude?

The normal naming convention would be to call the consolidated repo "tools-globalsearch".

That would be awesome if we could consolidate them.

Done.
https://github.com/davidbarratt/tools-standalone

dbarratt reassigned this task from dbarratt to bd808.Jun 12 2017, 9:01 PM
dbarratt added a subscriber: dbarratt.

@bd808 Let me know when it's moved over on GitHub and I'll change the origin on Tools.

Restricted Application added a project: User-bd808. · View Herald TranscriptJun 12 2017, 9:01 PM
bd808 added a comment.Jun 12 2017, 9:43 PM

https://github.com/wikimedia/tools-globalsearch

I gave the Community Tech team admin permission over the repo.

Only things left to do are update the README and change the tool instance to pull from the new repo.

Tool updated

tools.globalsearch@tools-bastion-03:~/public_html$ git remote -v
origin	https://github.com/wikimedia/tools-globalsearch.git (fetch)
origin	https://github.com/wikimedia/tools-globalsearch.git (push)

doh! let me update the README

kaldari closed this task as Resolved.Jun 12 2017, 10:46 PM

Looks great!

DannyH moved this task from Estimated to Archive on the Community-Tech board.