Page MenuHomePhabricator

Check for Mediawiki redirect loops via httpbb
Closed, ResolvedPublic

Description

In this incident we saw redirect loops cause wikipedia.org to be inaccessible. Is it worth monitoring for redirect loops? And if so, is httpbb the best place for it?

Action item extracted from 2025-02-28 www.wikipedia.org redirect

Incident document: https://docs.google.com/document/d/1lWztDuL13gGRNzw63OOX9zbZ2QOK652wOH5Cm-HM16I/edit?tab=t.0#heading=h.95p2g5d67t9q

Event Timeline

Is this in order to avoid a recurrence of the issue in the incident? It seems fairly specific, is there risk it will recur? I'm not sure if this is actionable to create a useful alert

I wonder if this would be better phrased as "should we detect redirect loops for wikis in general"? It would probably fit best as a httpbb check in that case.

hnowlan renamed this task from Can we check content/http code of www.wikipedia.org to Check for Mediawiki redirect loops via httpbb.May 11 2026, 2:58 PM
hnowlan added a project: ServiceOps new.
hnowlan updated the task description. (Show Details)
hnowlan moved this task from Inbox to Radar on the SRE Observability board.
RLazarus subscribed.

We actually added a test for that case already (in T387549, for this incident):

https://www.wikipedia.org:
- path: /
  assert_status: 200
  assert_body_contains: The Free Encyclopedia

It doesn't check for a redirect loop; it makes a stronger assertion, that this URL isn't a redirect at all. (If we'd had the test at the time of this incident, it would have failed with "expected status 200, got 301"). So for this specific incident we can call the action item resolved.


In general, I think that's probably the correct way to go, too. httpbb doesn't currently follow redirects; when we test them, we check the headers of the redirect response itself, like this one for wikipedia.org -> www.wikipedia.org:

https://wikipedia.org:
- path: /
  assert_status: 301
  assert_headers:
    Location: https://www.wikipedia.org/

So if that page were redirecting to itself, the test would fail because the Location header didn't have the expected value, and that's fine too.

Any time we'd want to assert "url X doesn't have a redirect loop," we should instead just assert either "url X doesn't redirect" or "url X redirects to the right place," and that covers us.

There are two cases this approach doesn't cover:

  1. Redirect loops in URLs that don't have a test, like this one at the time. httpbb can't detect that -- it can't guess what untested URLs might exist. We need to handle that at code-review time, by asking for tests along with Apache config changes, which we usually do. (A different tool could monitor for redirect loops in production traffic, but it would have to be somewhere else, like on a sample of webrequest logs -- and that could only alert us after the deployment, when it's already user-impacting.)
  1. Longer redirect loops, like A -> B -> A, where not only do both the A->B and B->A redirects exist in the config, but they also both have tests; i.e. the config and tests are both wrong. Then the tests would pass even though there's a loop. I guess httpbb could analyze for that, but it's a pretty niche hypothetical that I don't think is worth writing code for.

@hnowlan I think we should either reframe this task to cover case 1 (detect and alert on redirect loops after deployment, by monitoring webrequest logs, somewhere other than httpbb) or close it as resolved since we already wrote the necessary tests for this incident. My instinct is just to close it. What do you think?

hnowlan assigned this task to RLazarus.

@hnowlan I think we should either reframe this task to cover case 1 (detect and alert on redirect loops after deployment, by monitoring webrequest logs, somewhere other than httpbb) or close it as resolved since we already wrote the necessary tests for this incident. My instinct is just to close it. What do you think?

I think you're right, thanks for the great explanation.