Page MenuHomePhabricator

Toolforge: domain migration: track down missing OAuth grants
Closed, ResolvedPublic

Description

The actions in this task are related to https://wikitech.wikimedia.org/wiki/News/Toolforge.org

Specifically, tools listed here should need to take https://wikitech.wikimedia.org/wiki/News/Toolforge.org#OAuth_not_working_at_new_domain seriously, and request a new OAuth grant (if actually used by the tool).

This is a list of tools that may require a new OAuth grant for the new domain.
Some of the tool might be unmaintained or dead.

If marked, there is nothing else to do for it (either fixed, or non working, or dead tool).

Related:

1wikiadmin@10.64.48.153(metawiki)> SELECT DISTINCT
2 -> oarc_user_id,
3 -> SUBSTRING_INDEX(SUBSTRING_INDEX(oarc_callback_url, '/', 4), '/', -1) AS tool
4 -> FROM oauth_registered_consumer
5 -> WHERE oarc_stage = 1
6 -> AND oarc_callback_url LIKE 'https://tools.wmflabs.org/%'
7 -> AND oarc_name NOT IN (
8 -> SELECT oarc_name
9 -> FROM oauth_registered_consumer
10 -> WHERE oarc_stage = 1
11 -> AND oarc_callback_url LIKE 'https://%.toolforge.org/%'
12 -> )
13 -> ORDER BY oarc_user_id;
14+--------------+---------------------------------+
15| oarc_user_id | tool |
16+--------------+---------------------------------+
17| 160 | costar |
18| 160 | pagepile |
19| 160 | quickstatements |
20| 160 | tooltranslate |
21| 160 | widar |
22| 160 | wikishootme |
23| 160 | wlmuk |
24| 232 | as-info |
25| 232 | histsearch |
26| 232 | usernamesearch |
27| 461 | popularpages |
28| 527 | wlxjury |
29| 751 | submitter |
30| 2552 | editful |
31| 2645 | globalprefs |
32| 2773 | videoconvert |
33| 3318 | oauth-hello-world |
34| 4637 | citations |
35| 5005 | gerrit-patch-uploader |
36| 6398 | flickrdash |
37| 6398 | pg2ws |
38| 9468 | ash-django |
39| 12370 | aleph |
40| 14707 | itemfinder |
41| 15105 | locator-tool |
42| 15129 | pb |
43| 15129 | simplecommonstransfer |
44| 23411 | jimmy |
45| 25445 | wdbeoupdate |
46| 28500 | shexia |
47| 30834 | magog |
48| 271287 | meta |
49| 756525 | kokolores |
50| 1235163 | declare |
51| 1403237 | svgtranslate |
52| 1464021 | visualcategories |
53| 1877991 | ccm |
54| 1877991 | edgarsdev |
55| 1877991 | npp-lv |
56| 2200842 | dspull |
57| 2629548 | jorobot |
58| 2851948 | ptwikis |
59| 2851948 | wmopbot |
60| 2886067 | jembot |
61| 2888483 | fawiki-editathon |
62| 2888483 | idwiki-marathon |
63| 2888483 | maiwiki-editathon |
64| 2888483 | newiki-editathon |
65| 2888483 | rang |
66| 2888483 | wam |
67| 2888483 | wm-cee-spring |
68| 4170774 | my-first-django-oauth-app |
69| 5261941 | calling-card |
70| 5300473 | suggestor |
71| 5311143 | wikigrade |
72| 5330648 | kasparbot |
73| 5972740 | iluvatarbot |
74| 5972740 | swviewer |
75| 6024474 | montage |
76| 6024474 | monumental |
77| 6024474 | monumental-glam |
78| 7787507 | gratitude |
79| 7835020 | crosswatch |
80| 7835020 | watchr |
81| 8303885 | hd |
82| 9278481 | fountain |
83| 9278481 | fountain-test |
84| 9291471 | nullzerobot |
85| 10517190 | iabot |
86| 10517190 | supercount |
87| 10517190 | xtools |
88| 10975188 | hauki |
89| 10975188 | lexeme-senses |
90| 10975188 | wikidata-senses |
91| 11056975 | wikidata-slicer |
92| 11176046 | gyan |
93| 11176046 | twitter-to-commons |
94| 12256150 | import-500px |
95| 12256150 | merge2pdf |
96| 13567335 | wiper |
97| 13808645 | sbot |
98| 13876265 | dtz |
99| 14053179 | pltools |
100| 14494445 | cobot |
101| 14524051 | montage-beta |
102| 14524051 | montage-dev |
103| 16098390 | outofband |
104| 16293129 | commonsarchive |
105| 16293129 | video2commons |
106| 17347449 | paws |
107| 18474758 | fatameh |
108| 27851124 | spiarticleanalyzer |
109| 28456187 | my-first-flask-oauth-tool |
110| 28587080 | my-threads |
111| 28587080 | sparrow |
112| 28587080 | yabbr |
113| 28678913 | directory |
114| 28678913 | wikiviewstats |
115| 28678913 | xtools |
116| 29743591 | nordic-museum-depicts |
117| 29743591 | smv-description-translations |
118| 31931882 | yichengtry |
119| 34722510 | commons-mass-description |
120| 34722510 | commons-mass-description-test |
121| 34722510 | harvesting-data-refinery |
122| 34722510 | ipwatcher |
123| 34722510 | watch-translations |
124| 34722510 | weapon-of-mass-description |
125| 34722510 | weapon-of-mass-description-test |
126| 34722510 | wiki2email |
127| 34722510 | wikinity-test |
128| 38131863 | request |
129| 42818326 | jayprakashbot |
130| 42864640 | ircredirector |
131| 43193007 | niosh |
132| 44510157 | olympics |
133| 46762471 | copypatrol |
134| 46762471 | grantmetrics |
135| 46762471 | grantmetrics-test |
136| 46762471 | plagiabot |
137| 46899540 | analytics |
138| 47501133 | povoconta |
139| 47817072 | sqid |
140| 48055740 | nodejs-mw-oauth-tool |
141| 48311503 | isa |
142| 49267185 | glam2commons |
143| 49267185 | sibutest |
144| 49625466 | tsbot |
145| 50487796 | patrall |
146| 51711782 | outreachy-wikicv |
147| 51711782 | worklist-tool |
148| 52439465 | video-cut-tool-back-end |
149| 58667759 | articles-needing-links |
150+--------------+---------------------------------+
151133 rows in set (0.01 sec)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
aborrero updated the task description. (Show Details)
aborrero added subscribers: Addshore, Legoktm.
aborrero updated the task description. (Show Details)
aborrero updated the task description. (Show Details)
aborrero added subscribers: Prolineserver, Lokal_Profil.

I think asking everyone to create a new OAuth grant is a bad idea. Not joy will it create needless work for stewards and tool developers, but it clutters up an already messy collection of OAuth grants. Why not just run a DB script on production to just change the paths internally when the domain changes? That’s much easier, less work, and provides a more seamless experience for everyone.

I think asking everyone to create a new OAuth grant is a bad idea. Not joy will it create needless work for stewards and tool developers, but it clutters up an already messy collection of OAuth grants. Why not just run a DB script on production to just change the paths internally when the domain changes? That’s much easier, less work, and provides a more seamless experience for everyone.

This was already considered, see T244473#5866634

aborrero updated the task description. (Show Details)
aborrero added a subscriber: Ash_Crow.
bd808 triaged this task as High priority.Jun 9 2020, 2:52 PM
bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero raised the priority of this task from High to Needs Triage.Jun 9 2020, 2:55 PM
aborrero updated the task description. (Show Details)

This was already considered, see T244473#5866634

Although the main reason it was rejected was the difficulty of handling both the old and new domain with the same consumer (or at least coordinating for the consumer update and the tool domain change to happen simultaneously). AIUI the old domains will be killed in two weeks; that would make a DB update feasible.

Hi there, I'm a maintainer of some of the projects listed in the task description. What specific actions should I take? and what is the due date? Thanks

What's the best way to have the webservice automatically redirect to the new domain? I'm not too familiar .htaccess and the like.

My tools: iluvatarbot and swviewer.

  1. What exactly should I do? Just replace oAuth callback to swviewer.toolforge.org/...?
  2. Is it just about oAuth? If I ignore this notification, will my tools continue working? Will only oAuth be broken or tools is too?

Well I have new grants pending approval, now I just need to know how to configure the webserver to redirect to the new domain. Any help would be appreciated.

This was already considered, see T244473#5866634

Although the main reason it was rejected was the difficulty of handling both the old and new domain with the same consumer (or at least coordinating for the consumer update and the tool domain change to happen simultaneously). AIUI the old domains will be killed in two weeks; that would make a DB update feasible.

Yes, that's a good idea. Coordinate the force-redirection with the DB update. Will talk with @bd808 about this. Thanks!

Hi there, I'm a maintainer of some of the projects listed in the task description. What specific actions should I take? and what is the due date? Thanks

Please read https://wikitech.wikimedia.org/wiki/News/Toolforge.org#OAuth_not_working_at_new_domain

What's the best way to have the webservice automatically redirect to the new domain? I'm not too familiar .htaccess and the like.

Please read https://wikitech.wikimedia.org/wiki/News/Toolforge.org#Temporary_--canonical_switch_for_webservice.

i.e, try webservice start [..] --canonical.

My tools: iluvatarbot and swviewer.

  1. What exactly should I do? Just replace oAuth callback to swviewer.toolforge.org/...?
  2. Is it just about oAuth? If I ignore this notification, will my tools continue working? Will only oAuth be broken or tools is too?

Yes, the oauth callback URL is changing with the new domain. Please read more information here: https://wikitech.wikimedia.org/wiki/News/Toolforge.org#OAuth_not_working_at_new_domain

aborrero triaged this task as High priority.Jun 9 2020, 3:54 PM

I just realized that redirecting to the new URL, with the canonical redirect doesn't actually break OAuth callbacks. I'm wondering why this ticket is needed?

I just realized that redirecting to the new URL, with the canonical redirect doesn't actually break OAuth callbacks. I'm wondering why this ticket is needed?

If it does not break them, then your tool is not actually using its runtime config to tell the OAuth server where to send the response. As a concrete example, a typical Python + flask tool built along the model of https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool would send the OAuth server a return URL matching the inbound request's URL ($TOOL.toolforge.org vs tools.wmflabs.org/$TOOL) and that would not match a grant configured for the opposite URL scheme.

I just realized that redirecting to the new URL, with the canonical redirect doesn't actually break OAuth callbacks. I'm wondering why this ticket is needed?

If it does not break them, then your tool is not actually using its runtime config to tell the OAuth server where to send the response. As a concrete example, a typical Python + flask tool built along the model of https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool would send the OAuth server a return URL matching the inbound request's URL ($TOOL.toolforge.org vs tools.wmflabs.org/$TOOL) and that would not match a grant configured for the opposite URL scheme.

I'm not sure I follow. My tool sends the user to oauthcallback.php. It creates a request, and then sends the user of to Wikipedia. Wikipedia then returns to oauthcallback.php. Since both the old grant and the new grant go to the same oauthcallback script, MW is feeding it valid tokens. As long as the old domain redirects, OAuth continues to function even with the old grant going to the old domain, since all tokens are passed during the redirect.

I'm not sure I follow. My tool sends the user to oauthcallback.php. It creates a request, and then sends the user of to Wikipedia. Wikipedia then returns to oauthcallback.php. Since both the old grant and the new grant go to the same oauthcallback script, MW is feeding it valid tokens. As long as the old domain redirects, OAuth continues to function even with the old grant going to the old domain, since all tokens are passed during the redirect.

This is really off topic discussion here. The "It creates a request" part includes signing the request and the signed data includes scheme, host, and port information about where the request is coming from. The old and new grant go to different URLs. If your tool is signing a request containing foo.toolforge.org with a key from tools.wmflabs.org/foo then it will be rejected. Further tutorials on how OAuth 1.0a works can be found online, but please do trust me that I'm not confused and that new grants are needed.

I'm not sure I follow. My tool sends the user to oauthcallback.php. It creates a request, and then sends the user of to Wikipedia. Wikipedia then returns to oauthcallback.php. Since both the old grant and the new grant go to the same oauthcallback script, MW is feeding it valid tokens. As long as the old domain redirects, OAuth continues to function even with the old grant going to the old domain, since all tokens are passed during the redirect.

This is really off topic discussion here. The "It creates a request" part includes signing the request and the signed data includes scheme, host, and port information about where the request is coming from. The old and new grant go to different URLs. If your tool is signing a request containing foo.toolforge.org with a key from tools.wmflabs.org/foo then it will be rejected. Further tutorials on how OAuth 1.0a works can be found online, but please do trust me that I'm not confused and that new grants are needed.

Anomie just explained it to me. Since I'm using "oob" as the OAuth callback URL, it works despite being redirected.

Euku added a subscriber: Euku.
Euku removed a subscriber: Euku.

Hey there, just wanted to drop a note that we've (cc @Slaporte) started on migration work for Hatnote projects (montage, monumental), but didn't get around to finishing it. The June 11 mention was the first I heard of it (thanks @Lokal_Profil otherwise I might have heard from a user), and we put it on the backlog for this past weekend.

I noticed the redirects haven't kicked in per the original timeline (today was supposed to be the hard deadline) Any chance of an extension for a week (or two)?

I noticed the redirects haven't kicked in per the original timeline (today was supposed to be the hard deadline) Any chance of an extension for a week (or two)?

The deadline has been extended until 2020-06-30. We hope that folks will be able to use this additional time well. Unfortunately we know that not all tools will be actively updated even with the extra time, but we hope that this will reduce the amount of disruption for the community and the work that tools support.

What does "working, changes required" actually mean? I got the new Oauth grant, restarted the webservice with --canonical, and it seems to work (I have a problem when accessing the tool via proxy, but I have no clue about what's wrong). What else do we need to do to get the tick?

What does "working, changes required" actually mean? I got the new Oauth grant, restarted the webservice with --canonical, and it seems to work (I have a problem when accessing the tool via proxy, but I have no clue about what's wrong). What else do we need to do to get the tick?

You need to tick the boxes yourself when you have made the changes needed to work on the new domain.