Looking at this code in index():

# ingress-nginx connection error pages
if "X-Original-URI" in flask.request.headers:
    url = (
        flask.request.url_root.removesuffix("/")
        + flask.request.headers["X-Original-URI"]
    )
    ctx = context_for_url(url, flask.request.headers)
    return render_error_template(ctx)

I’m guessing the X-Original-URI is suddenly a full URL (with origin) instead of just the path + search? Which, from a look through Puppet, sounds like fallout from T283948: Merge Toolforge Nginx front proxy into the existing K8s HAProxy setup, but I don’t think I know enough about Ngnix/HAProxy to look further into it.

There was also an ingress update recently (T383516)

In T409008#11337155, @DamianZaremba wrote:

Would suspect T383516 to be more likely fwiw

The nginx ingress upgrade has been rolled back for the moment so I can find a fix for T408570: IW ingress config does not work with nginx-ingress-controller:v1.13.3.

Oh, this is interesting – it’s only reproducible if the request URL doesn’t have a “proper” path.

https://no-such-tool.toolforge.org/:

The URL you have requested, https://no-such-tool.toolforge.orghttps://no-such-tool.toolforge.org/, doesn't seem to actually exist.

https://no-such-tool.toolforge.org//:

The URL you have requested, https://no-such-tool.toolforge.orghttps://no-such-tool.toolforge.org//, doesn't seem to actually exist.

https://no-such-tool.toolforge.org///:

The URL you have requested, https://no-such-tool.toolforge.orghttps://no-such-tool.toolforge.org///, doesn't seem to actually exist.

But, https://no-such-tool.toolforge.org/404:

The URL you have requested, https://no-such-tool.toolforge.org/404, doesn't seem to actually exist.

Or, https://no-such-tool.toolforge.org/_:

The URL you have requested, https://no-such-tool.toolforge.org/_, doesn't seem to actually exist.

Or, https://no-such-tool.toolforge.org/​ (sneaky U+200B ZERO WIDTH SPACE):

The URL you have requested, https://no-such-tool.toolforge.org/​, doesn't seem to actually exist.

In T409008#11333817, @LucasWerkmeister wrote:

I’m guessing the X-Original-URI is suddenly a full URL (with origin) instead of just the path + search? Which, from a look through Puppet, sounds like fallout from T283948: Merge Toolforge Nginx front proxy into the existing K8s HAProxy setup, but I don’t think I know enough about Ngnix/HAProxy to look further into it.

The new haproxy config is:

http-request set-header X-Original-URI %[url]

The prior nginx config was:

proxy_set_header X-Original-URI $request_uri;

The $request_uri var in nginx is documented as "full original request URI (with arguments)", but this terminology can be confusing as "request URI" here refers to the Request-URI term from RFC 2616 (HTTP/1.1) and not the general term URI from RFC 3986.

The haproxy %[url] is the full requested URL including protocol and host as show in @LucasWerkmeister's example. I think the equivalent of the nginx config would use %[path]%HQ instead. The funny looking %HQ (HTTP Query) there is the equivalent of ?%[query] with the ? omitted if %[query] is empty.

Change #1201549 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Restore original X-Original-URI value

https://gerrit.wikimedia.org/r/1201549

Change #1201549 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Restore original X-Original-URI value

https://gerrit.wikimedia.org/r/1201549