Currently the beta cluster "your IP has been blocked" page tells users to contact noc@wikimedia.org to be unblocked. That should point to some other email address instead or that sentence should be removed entirely.
Description
Details
- Other Assignee
- BCornwall
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| varnish: Allow customising "contact noc@" error | operations/puppet | production | +8 -8 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T393487 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) | |||
| Resolved | taavi | T393404 Beta cluster IP block page should not point to noc@wikimedia.org |
Event Timeline
This bit in operations/puppet.git:modules/varnish/templates/wikimedia-frontend.vcl.erb is what would need to change:
// Block requests from IPs in blocked_nets. It is important to do this // early but after recv_fe_ip_processing has been called, as the procedure // takes care of writing X-Client-IP if it the request did not come // through the TLS terminator if (std.ip(req.http.X-Client-IP, "192.0.2.1") ~ blocked_nets) { return (synth(403, "Requests from your IP have been blocked, please contact noc@wikimedia.org")); }
One approach would be to move the existing message into a hiera variable and then override that setting in the deployment-prep deployment.
I don't think we really have a shared email account that is ideal for handling end user issues like this. I wonder if we could point at a page under https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep where we could explain the likely problem (caught in a large network block we banned to slow down crawlers) and give instructions on creating a Phab task requesting a hole in the block?
There are three more 4xx synthetic responses in the same area of the Varnish config. They are all related to user-agent blocks of various kinds and all seem to return the same message: Scripted requests from your IP have been blocked, please see https://meta.wikimedia.org/wiki/User-Agent_policy. In case of further questions, please contact noc@wikimedia.org.
A handful of other noc@wikimedia.org references are present in other vcl files in the ::varnish module.
We got another email to noc@ about this. Is there at least an address it can be forwarded to?
For lack of anything better you can bounce them to my work address. bd808-as-a-service doesn't scale, but maybe I'm better than a blackhole route.
Change #1143602 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] varnish: Allow customising "contact noc@" error
I have created https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Blocked_help and a redirect at https://wikitech.wikimedia.org/wiki/Beta/Blocked that we can use to tell folks how to work around the various error responses they may be getting.
Mentioned in SAL (#wikimedia-releng) [2025-05-22T21:09:11Z] <bd808> Added block_help: "see https://wikitech.wikimedia.org/wiki/Beta/Blocked_help for more information." under profile::cache::varnish::frontend::fe_vcl_config in both deployment-cache-text and deployment-cache-upload Prefix Puppet (T393404)
Mentioned in SAL (#wikimedia-releng) [2025-05-22T21:09:32Z] <bd808> Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1143602 (T393404)
Mentioned in SAL (#wikimedia-releng) [2025-05-22T21:12:53Z] <bd808> Forced Puppet run and restarted varnins-frontend on deployment-cache-text08 to pick up new config (T393404)
Mentioned in SAL (#wikimedia-releng) [2025-05-22T21:15:52Z] <bd808> Forced Puppet run and restarted varnins-frontend on deployment-cache-upload08 to pick up new config (T393404)
With the cherry-pick in place and the hiera customization the block screen looks something like this now:
Assigning to @taavi. The remaining work is merging his https://gerrit.wikimedia.org/r/c/operations/puppet/+/1143602 patch.
Cherry-pick caused T399463: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep. The patch has been updated so that it applies cleanly again.
Change #1143602 merged by BCornwall:
[operations/puppet@production] varnish: Allow customising "contact noc@" error
