Page MenuHomePhabricator

Cloud proxy redirect from $X.wmflabs.org to $X.wmcloud.org happening without a configured wmcloud.org proxy record
Closed, ResolvedPublic

Description

[17:09]  <zpapierski_> hey, we have slight problem with .wmflabs.org domain. We just released new service - wcqs-beta.wmflabs.org. We have a web proxy set up to this address, but if for some reason service is down, this address is permanently redirected to wcqs-beta.wmfcloud.org. Once the service is up, original address works again, but if some user tried to access it during the outage, user's browser probably cached 301. 
[17:09]  <zpapierski_> Is there a an approved way of dealing with that? (other than creating reverse 301 redirect)
[17:10]  <    bd808> zpapierski_: advertise the wmcloud.org hostname. Problem solved
[17:12]  <zpapierski_> wmflabs.org isn't supported anymore?
[17:14]  <    bd808> the redirect to wmcloud.org should only happen if there is a named proxy for it, but maybe there is a bug in that logic. The redirect is happening because of how we have set things up for the wmcloud.org transition for sure
[17:15]  <    bd808> zpapierski_: so I do see a configured wcqs-beta.wmflabs.org and not a configured wcqs-beta.wmcloud.org proxy so I think the 301 you are seeing when the backend for the wcqs-beta.wmflabs.org proxy is down is a bug
[17:16]  <zpapierski_> if I move to wmfcloud.org, that bug is actually quite useful :)
[17:16]  <    bd808> what we intended to setup was 301 redirect from $X.wmlabs.org to $X.wmcloud.org only when there is no configured proxy for $X.wmflabs.org but there is for $X.wmcloud.org
[17:17]  <    bd808> zpapierski_: so I guess we should open a phab task and figure out why the redirect is being over eager

Event Timeline

I've put in place a fallthrough redirect and a wildcard *.wmflabs.org domain so now any <name>.wmflabs.org domain that is not otherwise defined will automatically redirect to <name>.wmcloud.org.

Maybe it is working as designed? But this report does seem to be an unexpected side effect.

Change 615829 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] dynamicproxy: Only redirect to wmcloud if proxy is registered

https://gerrit.wikimedia.org/r/615829

bd808 moved this task from Done to Needs Review/Feedback on the User-bd808 board.

Unfortunately, I wasn't able to confirm that if by any chance cached interaction happened when proxy was actually down (there was a period of 2-3h when that was true) - but it is a possibility.

Change 615829 merged by Andrew Bogott:
[operations/puppet@production] dynamicproxy: Only redirect to wmcloud if proxy is registered

https://gerrit.wikimedia.org/r/615829

An unregistered proxy (for example https://abcdefg.wmflabs.org/) now returns a 404 page. It is an ugly default nginx 404 though. I'd like to instead serve something a bit more branded.

Change 616586 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] dynamicproxy: Add custom response for missing proxy backend

https://gerrit.wikimedia.org/r/616586

Change 616586 merged by Bstorm:
[operations/puppet@production] dynamicproxy: Add custom response for missing proxy backend

https://gerrit.wikimedia.org/r/616586

Change 622195 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] dynamicproxy: Fix missing proxy handling

https://gerrit.wikimedia.org/r/622195

Change 622195 merged by Bstorm:
[operations/puppet@production] dynamicproxy: Fix missing proxy handling

https://gerrit.wikimedia.org/r/622195

An unregistered proxy (for example https://abcdefg.wmflabs.org/) now returns a 404 page. It is an ugly default nginx 404 though. I'd like to instead serve something a bit more branded.

Screen Shot 2020-08-24 at 4.13.56 PM.png (189×649 px, 35 KB)