Page MenuHomePhabricator

Toolforge fourohfour tool (404 error) shows URL twice
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

What happens?:

The URL you have requested, https://no-such-tool.toolforge.orghttps://no-such-tool.toolforge.org/, doesn't seem to actually exist.

What should have happened instead?:

The URL you have requested, https://no-such-tool.toolforge.org/, doesn't seem to actually exist.

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):
commit 5b22d4c3c7

Other information (browser name/version, screenshots, etc.):

Screen Shot 2025-11-02 at 19.30.36.png (2×3 px, 396 KB)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Looking at this code in index():

# ingress-nginx connection error pages
if "X-Original-URI" in flask.request.headers:
    url = (
        flask.request.url_root.removesuffix("/")
        + flask.request.headers["X-Original-URI"]
    )
    ctx = context_for_url(url, flask.request.headers)
    return render_error_template(ctx)

I’m guessing the X-Original-URI is suddenly a full URL (with origin) instead of just the path + search? Which, from a look through Puppet, sounds like fallout from T283948: Merge Toolforge Nginx front proxy into the existing K8s HAProxy setup, but I don’t think I know enough about Ngnix/HAProxy to look further into it.

There was also an ingress update recently (T383516)

Would suspect T383516 to be more likely fwiw

The nginx ingress upgrade has been rolled back for the moment so I can find a fix for T408570: IW ingress config does not work with nginx-ingress-controller:v1.13.3.

Oh, this is interesting – it’s only reproducible if the request URL doesn’t have a “proper” path.

https://no-such-tool.toolforge.org/:

The URL you have requested, https://no-such-tool.toolforge.orghttps://no-such-tool.toolforge.org/, doesn't seem to actually exist.

https://no-such-tool.toolforge.org//:

The URL you have requested, https://no-such-tool.toolforge.orghttps://no-such-tool.toolforge.org//, doesn't seem to actually exist.

https://no-such-tool.toolforge.org///:

The URL you have requested, https://no-such-tool.toolforge.orghttps://no-such-tool.toolforge.org///, doesn't seem to actually exist.

But, https://no-such-tool.toolforge.org/404:

The URL you have requested, https://no-such-tool.toolforge.org/404, doesn't seem to actually exist.

Or, https://no-such-tool.toolforge.org/_:

The URL you have requested, https://no-such-tool.toolforge.org/_, doesn't seem to actually exist.

Or, https://no-such-tool.toolforge.org/​ (sneaky U+200B ZERO WIDTH SPACE):

The URL you have requested, https://no-such-tool.toolforge.org/​, doesn't seem to actually exist.

I’m guessing the X-Original-URI is suddenly a full URL (with origin) instead of just the path + search? Which, from a look through Puppet, sounds like fallout from T283948: Merge Toolforge Nginx front proxy into the existing K8s HAProxy setup, but I don’t think I know enough about Ngnix/HAProxy to look further into it.

The new haproxy config is:

http-request set-header X-Original-URI %[url]

The prior nginx config was:

proxy_set_header X-Original-URI $request_uri;

The $request_uri var in nginx is documented as "full original request URI (with arguments)", but this terminology can be confusing as "request URI" here refers to the Request-URI term from RFC 2616 (HTTP/1.1) and not the general term URI from RFC 3986.

The haproxy %[url] is the full requested URL including protocol and host as show in @LucasWerkmeister's example. I think the equivalent of the nginx config would use %[path]%HQ instead. The funny looking %HQ (HTTP Query) there is the equivalent of ?%[query] with the ? omitted if %[query] is empty.

Change #1201549 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Restore original X-Original-URI value

https://gerrit.wikimedia.org/r/1201549

Change #1201549 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Restore original X-Original-URI value

https://gerrit.wikimedia.org/r/1201549

taavi claimed this task.
taavi updated Other Assignee, added: bd808.
taavi subscribed.

Thanks @bd808 for doing the digging here!