Hi folks: Wanted to check the proposed timeline on this. There are no concerns from Traffic's end in keeping the CSP in Varnish but the duplicity may lead to some issues and hence the ask.

In T420604#11814351, @ssingh wrote:

Hi folks: Wanted to check the proposed timeline on this. There are no concerns from Traffic's end in keeping the CSP in Varnish but the duplicity may lead to some issues and hence the ask.

I've been lazy getting a patch up. I basically have the config done locally that satisfies this bug and T420607 and T416863. And includes the current, post-incident domain exceptions. We need to push it up to -labs first to make sure it runs as expected. And then it can get deployed and we can remove the VCL config. This shouldn't really take very long and I'd honestly like to try to get this done this week since it's been lingering.

Thanks for the update @sbassett! Sounds good and let us know when you are ready and we can take care of the removal.

Change #1272895 had a related patch set uploaded (by SBassett; author: SBassett):

[operations/mediawiki-config@master] Set CSP to enforce with currently-allow-listed domains on Beta

https://gerrit.wikimedia.org/r/1272895

Change #1272895 merged by jenkins-bot:

[operations/mediawiki-config@master] Set CSP to enforce with currently-allow-listed domains on Beta

https://gerrit.wikimedia.org/r/1272895

Change #1275536 had a related patch set uploaded (by Reedy; author: Reedy):

[operations/mediawiki-config@master] CommonSettings-labs: Fix up CSP config

https://gerrit.wikimedia.org/r/1275536

Change #1275536 merged by jenkins-bot:

[operations/mediawiki-config@master] CommonSettings-labs: Fix up CSP config

https://gerrit.wikimedia.org/r/1275536

Hey @ssingh - We deployed https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1272895 to beta. And @Catrope verified via shell.php that the actual CSP config seems correct. But we're not seeing it updated in what is being sent to various browsers. And there's still a report-only header being sent there, which should no longer be the case, as that is explicitly disabled within beta's config now. Is there another VCL-based config for the beta cluster? That's maybe getting merged now?

Furthermore, I verified that the correct CSP headers appear at https://en.wikipedia.beta.wmcloud.org/w/?oldid=2 (I remember a bug where the VCL logic doesn't apply to URLs like that), but the old ones appear at https://en.wikipedia.beta.wmcloud.org/w/index.php?oldid=2

Hi folks: note that we are still setting the CSP in Varnish, for production and therefore Beta as well, since the same VCL applies to both places. The reason for the disparity between the paths is because of where the VCL is applied; see below:

if (req.url ~ "^/wiki/" || req.url ~ "^/w/index\.php" || req.url ~ "^/\?title=") {
      // ...but exempt CentralNotice banner special pages
      if (req.url !~ "^/(wiki/|(w/index\.php)?\?title=)Special:Banner") {
          set resp.http.Cache-Control = "private, s-maxage=0, max-age=0, must-revalidate, no-transform";
          // T419265 - Current
          set resp.http.Content-Security-Policy = "default-src ...
          set resp.http.Content-Security-Policy-Report-Only = "script-src ...

So that's why we see the CSP you changed at https://en.wikipedia.beta.wmcloud.org/w/?oldid=2 (set by MW) but not at https://en.wikipedia.beta.wmcloud.org/w/index.php?oldid=2 (set at Varnish), since at /w/index.php, we are still setting it in the VCL, which therefore overrides the CSP you set at the MW layer. Does that make sense or am I confusing it with something else?

In T420604#11844137, @ssingh wrote:

So that's why we see the CSP you changed at https://en.wikipedia.beta.wmcloud.org/w/?oldid=2 (set by MW) but not at https://en.wikipedia.beta.wmcloud.org/w/index.php?oldid=2 (set at Varnish), since at /w/index.php, we are still setting it in the VCL, which therefore overrides the CSP you set at the MW layer. Does that make sense or am I confusing it with something else?

No, that makes sense, thanks. Is there a convenient way to disable the VCL config for beta at this time? Such a config should only be temporary as once the CSP proves stable in beta, we'll want to do this same config change in Wikimedia production. I'm guessing we might have to create some additional regexp along the lines of req.url ~ ".+beta.wmcloud.org\/" in those conditional blocks?

In T420604#11845148, @sbassett wrote:

In T420604#11844137, @ssingh wrote:

So that's why we see the CSP you changed at https://en.wikipedia.beta.wmcloud.org/w/?oldid=2 (set by MW) but not at https://en.wikipedia.beta.wmcloud.org/w/index.php?oldid=2 (set at Varnish), since at /w/index.php, we are still setting it in the VCL, which therefore overrides the CSP you set at the MW layer. Does that make sense or am I confusing it with something else?

No, that makes sense, thanks. Is there a convenient way to disable the VCL config for beta at this time? Such a config should only be temporary as once the CSP proves stable in beta, we'll want to do this same config change in Wikimedia production. I'm guessing we might have to create some additional regexp along the lines of req.url ~ ".+beta.wmcloud.org\/" in those conditional blocks?

We can selectively remove the filter in Beta if desired, yes. Right now there isn't a proper way of doing this other than using the hacky etcd_filters variable. I will prepare a patch.

In T420604#11845454, @ssingh wrote:

We can selectively remove the filter in Beta if desired, yes. Right now there isn't a proper way of doing this other than using the hacky etcd_filters variable. I will prepare a patch.

Ok, sounds great. Happy to help test/confirm once that's deployed.

Change #1276017 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] varnish: do not set CSP policy for beta

https://gerrit.wikimedia.org/r/1276017

Change #1276017 merged by Ssingh:

[operations/puppet@production] varnish: do not set CSP policy for beta

https://gerrit.wikimedia.org/r/1276017

In T420604#11846467, @sbassett wrote:

In T420604#11845454, @ssingh wrote:

We can selectively remove the filter in Beta if desired, yes. Right now there isn't a proper way of doing this other than using the hacky etcd_filters variable. I will prepare a patch.

Ok, sounds great. Happy to help test/confirm once that's deployed.

Please test now.

curl -sI https://en.wikipedia.beta.wmcloud.org/w/index.php?oldid=2 | grep -i ^content-security
content-security-policy: script-src 'unsafe-eval' blob: 'self' meta.wikimedia.beta.wmcloud.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.wikisource.org wikisource.org *.wikiquote.org *.wikidata.org *.wikifunctions.org *.wikivoyage.org *.mediawiki.org mediawiki.org wikimedia.org *.wmflabs.org *.wmcloud.org

We are only sending the CSP in VCL in production, not Beta.

Mentioned in SAL (#wikimedia-operations) [2026-04-23T15:48:35Z] <sukhe> sudo cumin -b31 "A:cp and not P{cp2041* or cp2042*}" "run-puppet-agent --enable 'merging CR 1276017'" T420604. finish rollout of removing CSP in VCL from beta

In T420604#11851974, @ssingh wrote:

Please test now.

We are only sending the CSP in VCL in production, not Beta.

Looks good, thanks! I confirmed that we're seeing the proper header on the beta cluster and that the proper headers (via VCL) still remain within Wikimedia production.

So I think the next steps here are to:

1. Monitor beta and check beta-logstash over the next few days to ensure stability
2. Make these same changes within Wikimedia production

In T420604#11852000, @sbassett wrote:

So I think the next steps here are to:

1. Monitor beta and check beta-logstash over the next few days to ensure stability
2. Make these same changes within Wikimedia production

For 2), we will be removing the VCL altogether I suppose? And you meant "same changes" as in same changes on MW?

In T420604#11852692, @ssingh wrote:

For 2), we will be removing the VCL altogether I suppose? And you meant "same changes" as in same changes on MW?

Yes, remove the VCL CSP for Wikimedia production entirely. Which I guess is just removing it entirely. Except for any odd exceptions for static assets, etc. that may already exist (I think there might be one for PDFs and/or testwiki). And then Wikimedia production will be (mostly) controlled by MediaWiki's CSP implementation. So these operations really shouldn't affect what user's experience now except for the report-only header going away. But in the future we plan to define more CSP directives and further tighten the current CSP allow-list - these steps just get us further down that path.

In T420604#11855938, @sbassett wrote:

In T420604#11852692, @ssingh wrote:

For 2), we will be removing the VCL altogether I suppose? And you meant "same changes" as in same changes on MW?

Yes, remove the VCL CSP for Wikimedia production entirely. Which I guess is just removing it entirely. Except for any odd exceptions for static assets, etc. that may already exist (I think there might be one for PDFs and/or testwiki). And then Wikimedia production will be (mostly) controlled by MediaWiki's CSP implementation. So these operations really shouldn't affect what user's experience now except for the report-only header going away. But in the future we plan to define more CSP directives and further tighten the current CSP allow-list - these steps just get us further down that path.

Yeah thanks, that sounds fine. That means then removing the CSP we are setting for text, which is the above one. The CSP for upload.wm.org will exist in VCL anyway (has to) and Traffic is happy to work together with you on updating or fixing it so that we can maintain it properly.

@ssingh, @Catrope et al -

I think we can likely get this same configuration deployed to Wikimedia production today this Monday, 2026-05-04. @ssingh - not sure if you have a preference, but maybe during the afternoon backport window or puppet window? To be clear, what we'd want to do:

1. Write corresponding config patches for Wikimedia production (ref: c1272895)
2. Remove the rest of the relevant VCL configuration for CSP for Wikimedia production and deploy (ref: c1276017)
3. Deploy CSP via Wikimedia config (from 1 above)
4. Verify

What has been observed on the beta cluster:

1. No reported issues with enforcing CSP, which now matches Wikimedia production and is being set via MediaWiki
2. The CSP report-only header has been removed, #csp-report-only has gone to zero in beta-logstash (link)
3. Other CSP reports seem ok in beta-logstash.

Change #1282979 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] varnish: remove CSP (and Report-Only) from VCL

https://gerrit.wikimedia.org/r/1282979

Change #1283020 had a related patch set uploaded (by SBassett; author: SBassett):

[operations/mediawiki-config@master] Set CSP to enforce with currently-allow-listed domains in Wikimedia production

https://gerrit.wikimedia.org/r/1283020

Change #1283020 merged by jenkins-bot:

[operations/mediawiki-config@master] Set CSP to enforce with allow-listed domains in Wikimedia production

https://gerrit.wikimedia.org/r/1283020

Mentioned in SAL (#wikimedia-operations) [2026-05-05T16:38:53Z] <sbassett@deploy1003> Started scap sync-world: Backport for [[gerrit:1283036|Set $wgReauthenticateTime editsitejs to one hour (T197137)]], [[gerrit:1283020|Set CSP to enforce with allow-listed domains in Wikimedia production (T419612 T420604 T420607)]]

Change #1283049 had a related patch set uploaded (by SBassett; author: SBassett):

[operations/mediawiki-config@master] Remove undefined variable $wmgUseCSPReportOnlyHasSession

https://gerrit.wikimedia.org/r/1283049

Change #1283049 merged by jenkins-bot:

[operations/mediawiki-config@master] Remove undefined variable $wmgUseCSPReportOnlyHasSession

https://gerrit.wikimedia.org/r/1283049

Mentioned in SAL (#wikimedia-operations) [2026-05-05T16:50:36Z] <sbassett@deploy1003> Started scap sync-world: Backport for [[gerrit:1283036|Set $wgReauthenticateTime editsitejs to one hour (T197137)]], [[gerrit:1283020|Set CSP to enforce with allow-listed domains in Wikimedia production (T419612 T420604 T420607)]], [[gerrit:1283049|Remove undefined variable $wmgUseCSPReportOnlyHasSession (T419612 T420604 T420607)]]

Mentioned in SAL (#wikimedia-operations) [2026-05-05T16:52:20Z] <sbassett@deploy1003> mstyles, sbassett: Backport for [[gerrit:1283036|Set $wgReauthenticateTime editsitejs to one hour (T197137)]], [[gerrit:1283020|Set CSP to enforce with allow-listed domains in Wikimedia production (T419612 T420604 T420607)]], [[gerrit:1283049|Remove undefined variable $wmgUseCSPReportOnlyHasSession (T419612 T420604 T420607)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdeb

Mentioned in SAL (#wikimedia-operations) [2026-05-05T16:58:02Z] <sbassett@deploy1003> Finished scap sync-world: Backport for [[gerrit:1283036|Set $wgReauthenticateTime editsitejs to one hour (T197137)]], [[gerrit:1283020|Set CSP to enforce with allow-listed domains in Wikimedia production (T419612 T420604 T420607)]], [[gerrit:1283049|Remove undefined variable $wmgUseCSPReportOnlyHasSession (T419612 T420604 T420607)]] (duration: 07m 25s)

Change #1282979 merged by Ssingh:

[operations/puppet@production] varnish: remove CSP (and Report-Only) from VCL

https://gerrit.wikimedia.org/r/1282979

Sorry to necro but are we interested in deduplication of CSP in upload as well? T117618 included some for testwiki but I wonder if that should also be removed from the CDN.

In T420604#11913458, @BCornwall wrote:

Sorry to necro but are we interested in deduplication of CSP in upload as well? T117618 included some for testwiki but I wonder if that should also be removed from the CDN.

Sadly, no. That particular config will likely change, but there will need to be at least a few CSP header exceptions exclusively defined within VCL config. These will be for things like upload.w.o, static assets that are web-server-served, and some exceptions that will require a tighter CSP. Most of these exceptions are known and are tracked on the CSP board here: https://phabricator.wikimedia.org/maniphest/query/xmCUfgiKYZen/#R.