Page MenuHomePhabricator
Create Task
Maniphest T234617

Toolforge. introduce new domain toolforge.org
Closed, ResolvedPublic
Actions

Description

In https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/DNS_domain_usage#Resolution we agreed on introducing the domain toolforge.org as the new general domain for Toolforge.

The path to introducing such domain contains several steps, first of all is probably how and when we will be introducing it. The most sensible option seems to be pairing the introduction of the new domain with the new kubernetes cluster. DNS will be managed via OpenStack Designate.

For the record, this is the current setup at whois level for wmflabs.org vs toolforge.org:

$ whois wmflabs.org | grep "Name Server"
Name Server: CLOUD-NS0.WIKIMEDIA.ORG
Name Server: CLOUD-NS1.WIKIMEDIA.ORG
$ whois toolforge.org | grep "Name Server"
Name Server: NS0.WIKIMEDIA.ORG
Name Server: NS1.WIKIMEDIA.ORG
Name Server: NS2.WIKIMEDIA.ORG

Announced to community: https://wikitech.wikimedia.org/wiki/News/Toolforge.org

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
Restricted Task
 ResolvedaborreroT125589 Allow each tool to have its own subdomain for browser sandbox/cookie isolation
 ResolvedaborreroT234617 Toolforge. introduce new domain toolforge.org
 ResolvedKrenairT235252 Toolforge: SSL support for new domain toolforge.org
 ResolvedNoneT235304 Create a service account to manage toolforge.org. from acme-chief
 ResolvedAndrewT235303 Update authoratiative nameservers for the toolforge.org domain to point to Designate
 ResolvedaborreroT244473 Toolforge: both domains in parallel and OAuth
 ResolvedaborreroT254857 Toolforge: domain migration: track down missing OAuth grants
 ResolvedaborreroT168677 Add new Cloud Services domains to public suffix list
 Declined• BstormT245572 Update Grid Engine and grid_configurator.py for the new domain names
 ResolvedaborreroT247236 Toolforge: introduce a system to preserve old tools.wmflabs.org URLs
 Resolvedbd808T247432 Preserve the ability to make interwiki links to Toolforge tools under the host based routing scheme
 ResolvedBUG REPORTbd808T257104 https://iw.toolforge.org breaks when not given any values
 Resolved• BstormT249390 Regression in webservice 0.65 in argument and config parsing
 ResolvedUrbanecmT249815 maps: whitelist/reduce ratelimit from requests with toolforge.org referrer
ResolvedaborreroT249843 Toolforge.org: k8s canonical temporal redirect uses HTTP 301
 Resolvedbd808T245804 Reassign base URLs for toolinfo records' web service links
 Resolvedbd808T254640 Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical`
 ResolvedtaaviT257386 Fix the proxy behavior for Kubernetes on toolsbeta
 ResolvedBUG REPORTDannyS712T257481 Refill results page returning '404 - Not Found'
 ResolvedBUG REPORTCyberpower678T257471 Refill stuck at 'Submitting your task...'
 ResolvedBUG REPORTParaT257576 toolserver.org/~para/ coordinate redirection tools return 404 error
 Resolvedbd808T262072 Update Cloud Services Terms of use to cover toolforge.org and wmcloud.org and subdomains thereof
 OpenNoneT285364 Audit wmf-config and add toolforge.org as needed where tools.wmflabs.org is used

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Apr 8 2020, 11:35 PM

Mentioned in SAL (#wikimedia-cloud) [2020-04-08T23:35:38Z] <bstorm_> deploy toollabs-webservice v0.66 T154504 T234617

• Bstorm closed subtask T249390: Regression in webservice 0.65 in argument and config parsing as Resolved.Apr 8 2020, 11:36 PM
Stashbot added a comment.Apr 8 2020, 11:49 PM

Mentioned in SAL (#wikimedia-cloud) [2020-04-08T23:49:21Z] <bd808> Running rebuild_all for Docker images to pick up toollabs-webservice v0.66 (T154504, T234617)

Stashbot added a comment.Apr 9 2020, 4:29 AM

Mentioned in SAL (#wikimedia-cloud) [2020-04-09T04:29:55Z] <bd808> Running rebuild_all for Docker images to pick up toollabs-webservice v0.66 [try #2] (T154504, T234617)

• Bstorm closed subtask T245572: Update Grid Engine and grid_configurator.py for the new domain names as Declined.Apr 9 2020, 6:35 PM
MBH subscribed.Apr 13 2020, 11:44 AM

There is interwiki prefix [[toollabs:...]] now, that still leads to old url. Will it be updated or will be introduced a new prefix?

Stashbot added a comment.Apr 13 2020, 12:41 PM

Mentioned in SAL (#wikimedia-cloud) [2020-04-13T12:41:18Z] <arturo> adding a new ingress 'default-route-toolforge.org' to redirect http://toolforge.org to the wikitech front page (T234617)

Stashbot added a comment.Apr 13 2020, 12:43 PM

Mentioned in SAL (#wikimedia-cloud) [2020-04-13T12:43:00Z] <arturo> adding a new ingress 'default-route-www.toolforge.org' to redirect http://www.toolforge.org to the wikitech front page (T234617)

aborrero added a comment.Apr 13 2020, 12:47 PM
In T234617#6051459, @MBH wrote:

There is interwiki prefix [[toollabs:...]] now, that still leads to old url. Will it be updated or will be introduced a new prefix?

I think @bd808 has been working on this already. See https://wikitech.wikimedia.org/wiki/Tool:Iw

RhinosF1 subscribed.Apr 13 2020, 2:31 PM
bd808 added a comment.Apr 13 2020, 5:29 PM
In T234617#6051591, @aborrero wrote:
In T234617#6051459, @MBH wrote:

There is interwiki prefix [[toollabs:...]] now, that still leads to old url. Will it be updated or will be introduced a new prefix?

I think @bd808 has been working on this already. See https://wikitech.wikimedia.org/wiki/Tool:Iw

Some additional documentation at T247432: Preserve the ability to make interwiki links to Toolforge tools under the host based routing scheme. The short answer is yes we will keep interwiki links working. While we are mixing tools.wmflabs.org/$TOOL and $TOOL.toolforge.org IW links will keep working as a side effect of the proxy redirection. Once we stop letting new tools get a tools.wmflabs.org/$TOOL we will either introduce configuration changes in the IW table to point them all to the iw.toolforge.org tool I created, or we will do something closer to home on the MediaWiki side with hooks.

Abbe98 subscribed.Apr 13 2020, 7:02 PM
aborrero closed subtask T249843: Toolforge.org: k8s canonical temporal redirect uses HTTP 301 as Resolved.Apr 14 2020, 11:37 AM
Urbanecm closed subtask T249815: maps: whitelist/reduce ratelimit from requests with toolforge.org referrer as Resolved.Apr 15 2020, 10:56 PM
bd808 updated the task description. (Show Details)Apr 16 2020, 11:14 PM
bd808 mentioned this in T102367: Migrate tools.wmflabs.org to https only (and set HSTS).
bd808 added a subtask: T245804: Reassign base URLs for toolinfo records' web service links.Apr 17 2020, 2:59 AM
bd808 mentioned this in T250457: Consider adding support for more optional ingress annotations to `webservice` for the Kubernetes backend.Apr 17 2020, 5:06 AM
Hasley subscribed.Apr 27 2020, 12:25 AM
bd808 mentioned this in T125589: Allow each tool to have its own subdomain for browser sandbox/cookie isolation.May 9 2020, 3:45 PM
BEANS-X2 subscribed.May 9 2020, 3:58 PM
Pintoch mentioned this in T254169: Migrate OpenRefine-Wikidata reconciliation service to toolforge.org.Jun 1 2020, 3:38 PM
Pintoch mentioned this in T254170: Migrate OABot service to toolforge.org.
Pintoch mentioned this in T254171: Migrate EditGroups tool to toolforge.org.Jun 1 2020, 3:41 PM
Pintoch mentioned this in T254172: Grace period for toolforge.org migration to tools exposing POST-based APIs?.Jun 1 2020, 4:01 PM
gerritbot added a comment.Jun 3 2020, 11:28 AM

Change 602054 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: legacy_redirector: refresh set of allowed tools

https://gerrit.wikimedia.org/r/602054

Kolossos mentioned this in T254693: Same origin policy access between tools deliberately broken by toolforge.org migration.Jun 7 2020, 3:56 PM
Magnus mentioned this in T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical`.Jun 7 2020, 4:04 PM
TheDJ subscribed.Jun 7 2020, 8:37 PM

i haven't been able to find info on tools-static.wmflabs.org.. can someone provide that info and update https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Static_file_server

bd808 added a subtask: T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical`.Jun 8 2020, 4:50 PM
bd808 added a comment.Jun 8 2020, 9:24 PM
In T234617#6200157, @TheDJ wrote:

i haven't been able to find info on tools-static.wmflabs.org.. can someone provide that info and update https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Static_file_server

No change is being made tools-static.wmflabs.org at this time.

Pasleim subscribed.Jun 15 2020, 8:48 PM
gerritbot added a comment.Jun 17 2020, 10:18 AM

Change 602054 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: legacy_redirector: refresh set of allowed tools

https://gerrit.wikimedia.org/r/602054

Stashbot mentioned this in T247236: Toolforge: introduce a system to preserve old tools.wmflabs.org URLs.Jun 17 2020, 10:40 AM

Mentioned in SAL (#wikimedia-cloud) [2020-06-17T10:40:10Z] <arturo> created VM tools-legacy-redirector, with the corresponding puppet prefix (T247236, T234617)

aborrero changed the status of subtask T247236: Toolforge: introduce a system to preserve old tools.wmflabs.org URLs from Open to Stalled.Jun 17 2020, 11:31 AM
bd808 closed subtask T254640: Default lighttpd config created by `webservice` breaks serving files starting with the same string as the tool's name under `--canonical` as Resolved.Jun 17 2020, 6:31 PM
Magnus subscribed.Jun 22 2020, 3:07 PM

FWIW, I believe all my (active) tools now use the new canonical URL.

aborrero added a comment.Jun 22 2020, 4:16 PM
In T234617#6244839, @Magnus wrote:

FWIW, I believe all my (active) tools now use the new canonical URL.

Thanks you very much!

Ghuron subscribed.Jun 26 2020, 1:38 PM

One of tools I'm maintaining (https://www.mediawiki.org/wiki/MW2SPARQL) is effectively WDQS federated endpoint. WDQS has white list of supported endpoints and http://tools.wmflabs.org/mw2sparql/sparql is there. Who can replace it to http://mw2sparql.toolforge.org/sparql ?

RhinosF1 added a comment.Jun 26 2020, 2:03 PM
In T234617#6260029, @Ghuron wrote:

One of tools I'm maintaining (https://www.mediawiki.org/wiki/MW2SPARQL) is effectively WDQS federated endpoint. WDQS has white list of supported endpoints and http://tools.wmflabs.org/mw2sparql/sparql is there. Who can replace it to http://mw2sparql.toolforge.org/sparql ?

This is currently set in https://github.com/wikimedia/wikidata-query-deploy/blob/master/whitelist.txt#L22

You can update that yourself from gerrit.

gerritbot added a comment.Jun 28 2020, 12:48 PM

Change 608131 had a related patch set uploaded (by Ghuron; owner: Ghuron):
[wikidata/query/deploy@master] Update url to my endpoint because of toolforge.org migration

https://gerrit.wikimedia.org/r/c/wikidata/query/deploy/ /608131

PeterBowman subscribed.Jul 2 2020, 1:55 PM
gerritbot added a comment.Jul 3 2020, 8:30 AM

Change 608131 merged by Gehel:
[wikidata/query/deploy@master] Update url to my endpoint because of toolforge.org migration

https://gerrit.wikimedia.org/r/608131

Ghuron unsubscribed.Jul 3 2020, 5:57 PM
aborrero changed the status of subtask T247236: Toolforge: introduce a system to preserve old tools.wmflabs.org URLs from Stalled to Open.Jul 6 2020, 10:59 AM
gerritbot added a comment.Jul 6 2020, 11:08 PM

Change 609855 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/software/tools-webservice@master] Remove --canonical argument to webservice

https://gerrit.wikimedia.org/r/609855

gerritbot added a comment.Jul 6 2020, 11:08 PM

Change 609856 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/software/tools-webservice@master] kubernetes: remove legacy ingress generation

https://gerrit.wikimedia.org/r/609856

aborrero closed subtask T247236: Toolforge: introduce a system to preserve old tools.wmflabs.org URLs as Resolved.Jul 7 2020, 10:01 AM
gerritbot added a comment.Jul 7 2020, 10:36 AM

Change 610013 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: shinken: refresh URLs with new toolforge.org domain

https://gerrit.wikimedia.org/r/610013

gerritbot added a comment.Jul 7 2020, 10:42 AM

Change 610013 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: shinken: refresh URLs with new toolforge.org domain

https://gerrit.wikimedia.org/r/610013

gerritbot added a comment.Jul 7 2020, 11:16 AM

Change 610024 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: ssh banners: refresh URLs

https://gerrit.wikimedia.org/r/610024

gerritbot added a comment.Jul 7 2020, 11:23 AM

Change 610024 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: ssh banners: refresh URLs

https://gerrit.wikimedia.org/r/610024

gerritbot added a comment.Jul 7 2020, 11:39 AM

Change 610029 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: urlproxy: drop support for the legacy routing scheme

https://gerrit.wikimedia.org/r/610029

Nintendofan885 closed subtask T247432: Preserve the ability to make interwiki links to Toolforge tools under the host based routing scheme as Resolved.Jul 7 2020, 9:15 PM
bd808 mentioned this in T257330: Tools show Puffin Browser Pro 308 Permanent Redirect.Jul 7 2020, 10:35 PM
gerritbot added a comment.Jul 7 2020, 11:00 PM

Change 609855 merged by jenkins-bot:
[operations/software/tools-webservice@master] Remove --canonical argument to webservice

https://gerrit.wikimedia.org/r/609855

gerritbot added a comment.Jul 7 2020, 11:01 PM

Change 609856 merged by jenkins-bot:
[operations/software/tools-webservice@master] kubernetes: remove legacy ingress generation

https://gerrit.wikimedia.org/r/609856

Stashbot added a comment.Jul 7 2020, 11:16 PM

Mentioned in SAL (#wikimedia-cloud) [2020-07-07T23:16:43Z] <bd808> Building webservice v0.73 (T234617, T257229)

Stashbot mentioned this in T257229: webservice: replace .webservicerc with service.template.Jul 7 2020, 11:16 PM
Stashbot added a comment.Jul 7 2020, 11:19 PM

Mentioned in SAL (#wikimedia-cloud) [2020-07-07T23:19:45Z] <bd808> Deploying webservice v0.73 via clush (T234617, T257229)

Stashbot added a comment.Jul 7 2020, 11:22 PM

Mentioned in SAL (#wikimedia-cloud) [2020-07-07T23:22:53Z] <bd808> Rebuilding all Docker images to pick up webservice v0.73 (T234617, T257229)

Jayprakash12345 mentioned this in T257410: Google OCR is not giving any output (due to domain change).Jul 8 2020, 8:58 AM
NicoV mentioned this in T257384: Redirect for digest.txt incorrect.Jul 8 2020, 10:14 AM
Stashbot added a comment.Jul 8 2020, 11:08 AM

Mentioned in SAL (#wikimedia-cloud) [2020-07-08T11:08:46Z] <arturo> live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 (T234617)

Stashbot added a comment.Jul 8 2020, 11:11 AM

Mentioned in SAL (#wikimedia-cloud) [2020-07-08T11:11:49Z] <arturo> live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 (T234617)

gerritbot added a comment.Jul 8 2020, 11:14 AM

Change 610029 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: urlproxy: drop support for the legacy routing scheme

https://gerrit.wikimedia.org/r/610029

Stashbot added a comment.Jul 8 2020, 11:16 AM

Mentioned in SAL (#wikimedia-cloud) [2020-07-08T11:16:39Z] <arturo> merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 -- important change to front-proxy (T234617)

MBH unsubscribed.Jul 8 2020, 11:25 AM
• Bstorm added a subtask: T257481: Refill results page returning '404 - Not Found'.Jul 8 2020, 5:14 PM
NicoV subscribed.EditedJul 8 2020, 6:09 PM

Problem with 308: Permanent Redirect.

With the migration, I see that my tool, WPCleaner , doesn't manage to update itself on user's installation due to the Permanent Redirect. T257495
I'm using getdown tool to have an automatic update when a user launches the tool: I submitted an issue there.

But I'm surprised to see that the tool replaced the content of some files with HTML code telling there's a Permanent Redirect.

<html>
<head><title>308 Permanent Redirect</title></head>
<body bgcolor="white">
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>

Is this normal to have the HTML code, and not be redirected directly?
Is there any way to solve this kind of problem?

NicoV mentioned this in T257495: WPCleaner not updating due to lack of HTTP redirect handling in getdown library.Jul 8 2020, 6:23 PM
• Bstorm added a subtask: T257471: Refill stuck at 'Submitting your task...'.Jul 8 2020, 6:31 PM
bd808 closed subtask T257471: Refill stuck at 'Submitting your task...' as Resolved.Jul 8 2020, 9:40 PM
gerritbot added a comment.Jul 9 2020, 3:16 AM

Change 610548 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] ncredir: Update mappings for pywikibot redirects

https://gerrit.wikimedia.org/r/610548

bd808 added a comment.Jul 9 2020, 3:52 AM
In T234617#6290707, @NicoV wrote:

Problem with 308: Permanent Redirect.

Let's figure this one out over on T257495: WPCleaner not updating due to lack of HTTP redirect handling in getdown library.

gerritbot added a comment.Jul 9 2020, 12:55 PM

Change 610548 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] ncredir: Update mappings for pywikibot redirects

https://gerrit.wikimedia.org/r/610548

Zache added a subtask: T257576: toolserver.org/~para/ coordinate redirection tools return 404 error.Jul 9 2020, 2:07 PM
NicoV unsubscribed.Jul 9 2020, 2:54 PM
bd808 closed subtask T257481: Refill results page returning '404 - Not Found' as Resolved.Jul 9 2020, 11:08 PM
Zache closed subtask T257576: toolserver.org/~para/ coordinate redirection tools return 404 error as Resolved.Jul 10 2020, 9:17 PM
bd808 closed subtask T244473: Toolforge: both domains in parallel and OAuth as Resolved.Jul 10 2020, 10:36 PM
aborrero closed this task as Resolved.Jul 14 2020, 9:13 AM
bd808 awarded a token.Jul 14 2020, 5:39 PM
Nintendofan885 awarded a token.Jul 14 2020, 8:58 PM
Zache mentioned this in T260520: maps.wikilovesmonuments.org returns a HTTP 429 error (let it access varnish maps_domains).Aug 17 2020, 10:01 AM
bd808 closed subtask T245804: Reassign base URLs for toolinfo records' web service links as Resolved.Sep 2 2020, 10:13 PM
Krinkle added a subtask: T262072: Update Cloud Services Terms of use to cover toolforge.org and wmcloud.org and subdomains thereof.Jun 23 2021, 12:23 AM
taavi closed subtask T262072: Update Cloud Services Terms of use to cover toolforge.org and wmcloud.org and subdomains thereof as Resolved.Dec 24 2022, 12:27 PM
taavi closed subtask T257386: Fix the proxy behavior for Kubernetes on toolsbeta as Resolved.May 9 2023, 5:38 PM
bd808 mentioned this in T337897: Uwsgi webservices have strange routing behavior when tool name and first component of route path are exact matches.May 31 2023, 10:54 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL