Change 318490 merged by Ema:
Expand Content-Security-Policy on upload test to fr.

https://gerrit.wikimedia.org/r/318490

Change 341207 had a related patch set uploaded (by bawolff):
[operations/puppet] Extend the upload Content-Security-Policy test to other large wikis

https://gerrit.wikimedia.org/r/341207

Change 341207 merged by BBlack:
[operations/puppet] Extend the upload Content-Security-Policy test to other large wikis

https://gerrit.wikimedia.org/r/341207

Recent logs suggest we may need to add "media-src 'self'" for webm files.

Change 547929 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/puppet@production] Adjust CSP header for pdfs & videos & set enforce on testwiki

https://gerrit.wikimedia.org/r/547929

A small number of browsers seem to want android-webview-video-poster: as a source when viewing videos, but the number of reports are small enough its not clear if we should include it.

In firefox (and edge based on logs, but i haven't tested edge yet), it seems to sometimes complain about lack of the current image in img-src, and lack of chrome: as a script-src for audio/video files. However, testing locally, when in enforce mode, nothing gets blocked, and the warnings go away. This is one of the reasons I'd like to put testwiki in enforce mode

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all tickets that aren't are neither part of our current planned work nor clearly a recent, higher-priority emergent issue. This is simply one step in a larger task cleanup effort. Further triage of these tickets (and especially, organizing future potential project ideas from them into a new medium) will occur afterwards! For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

In T117618#8637047, @Bawolff wrote:

Got deployed to /wikipedia/(el|fr|ru|it|de|uk|ja|id|he|fi|zh|test) but never everywhere. This is definitely something that would really make sense to deploy everywhere in enforce mode, especially for SVGs

So why did this stall ? I mean I understand that CSP on the wikis has some concerns and complexity, but for uploads it should be pretty clear case right ?

Can someone check the csp reports to see if there is any violations that should not be blocked ? I have no idea where we log these things

So why did this stall ?

For reference, this was around the time when i left my job at Wikimedia. I guess nobody took it up after.

If i remember there was some concern about overloading the logging server if we put this on a big wiki. Best solution is probably just not to log once we know it works.

Logging at the time was in log stash. I presume its still there.

https://logstash.wikimedia.org/goto/02dea35a7640ce8ad1babf9674174ed4
Dashboard ContentSecurityPolicy, message:"upload.wikimedia.org"

It seems most violations are related to browser extensions doing script injections and svgs trying to load fonts.

I've done some checkups in the logs and brought @Bawolff 's patches up to date. I think they are still fine to merge and move forward with, at the very least initially towards wikipedia/test resources.

Scheduled this for may 9th puppet window to start testing with test.wikipedia.org

Change #1059423 had a related patch set uploaded (by CDobbins; author: CDobbins):

[operations/puppet@production] varnish: Add restrictive CSP to upload.wikimedia.org and add tests

https://gerrit.wikimedia.org/r/1059423

What's the effect? Shall Wikimedia content be include-able into Commons svg graphics? Are we talking about resources external to a given svg graphic or external to Wikimedia?

In T117618#10043609, @Vollbracht wrote:

What's the effect? Shall Wikimedia content be include-able into Commons svg graphics? Are we talking about resources external to a given svg graphic or external to Wikimedia?

@Vollbracht The experienced effect should be nothing. It will only make it safer to use SVGs, as browsers won't allow the SVG to do unsafe things in your browser. The eventual goal is that at some point this will create the confidence required to allow usage of native SVG files, instead of PNG images and unblock T208578 for Wikimedia.

Following up on the patch by @TheDJ (thanks!) and per https://gerrit.wikimedia.org/r/c/operations/puppet/+/1059423/comment/61d3b1a3_eebb1d4b/, @Vgutierrez's comment:

so we would be sending the header content twice, on both -Report-Only and enforcing flavors, according to turnilo, upload.wm.o gets 47M hits per day for URLs matching ^/wikipedia/(el|fr|ru|it|de|uk|ja|id|he|fi|zh|test). The content of the non-PDF CSP header is 328 bytes long so we would be increasing our outbound traffic in ~ 14GB per day.

We are still discussing this in Traffic but sharing here for posterity and easier discussion.

In T117618#10044260, @TheDJ wrote:

In T117618#10043609, @Vollbracht wrote:

What's the effect? Shall Wikimedia content be include-able into Commons svg graphics? Are we talking about resources external to a given svg graphic or external to Wikimedia?

@Vollbracht The experienced effect should be nothing. It will only make it safer to use SVGs, as browsers won't allow the SVG to do unsafe things in your browser. The eventual goal is that at some point this will create the confidence required to allow usage of native SVG files, instead of PNG images and unblock T208578 for Wikimedia.

But what about my question? Did I get this right? Will Wikimedia content be include-able with this policy?

To clarify, this only affects direct viewing of SVGs, and SVGs included via an <iframe> tag (or in theory <object>). CSP policies do not affect SVGs included via <img> tags. <img> tags already work in secure mode where external resources and scripts are not allowed ( https://svgwg.org/specs/integration/#processing-mode )

The main benefit for using a CSP policy here, is that our svg upload filters are sketchy, and a malicious SVG could be used to compromise other people's accounts.

In T117618#10102911, @Bawolff wrote:

our svg upload filters are sketchy,

Could you describe where our upload filter will accept absolute resource references? wouldn't it be better to avoid this directly?

In T117618#10103308, @Vollbracht wrote:

In T117618#10102911, @Bawolff wrote:

our svg upload filters are sketchy,

Could you describe where our upload filter will accept absolute resource references? wouldn't it be better to avoid this directly?

By sketchy, i mean i dont fully trust them (it is probably off topic to this bug as to why). I do not mean i am aware of a bypass.

In T117618#10103514, @Bawolff wrote:

By sketchy, i mean i dont fully trust them [..]. I do not mean i am aware of a bypass.

That shouldn't be enough. We're not pushing bits in memory blocks that might be affected by any third process. We're checking text files that by all means have to be human readable. Verifying those filters shouldn't be witchcraft. Our decisions shouldn't be given distinction by German angst. Either we have an idea of an attack vector and harden our software or we assume our solution to be secure for the next few months.

In T117618#10106360, @Vollbracht wrote:

In T117618#10103514, @Bawolff wrote:

By sketchy, i mean i dont fully trust them [..]. I do not mean i am aware of a bypass.

That shouldn't be enough. We're not pushing bits in memory blocks that might be affected by any third process. We're checking text files that by all means have to be human readable. Verifying those filters shouldn't be witchcraft. Our decisions shouldn't be given distinction by German angst. Either we have an idea of an attack vector and harden our software or we assume our solution to be secure for the next few months.

Clearly you feel strongly about this, but security decisions are made based on serious concerns, regardless of what you postulate.

Even if (as you imply) our current security scanning was perfect both for known and all possible future threads, which I very much doubt, note that SVG files on Commons have had 20 years worth of differing security approaches. There is no process by which SVGs uploaded in 2005 get re-scanned and deleted.

In T117618#10106487, @Jdforrester-WMF wrote:

security decisions are made based on serious concerns

So name them and meet them!

There is no process by which SVGs uploaded in 2005 get re-scanned and deleted.

Seems as if we need a regular process for handling files that lost confidence. Of course I expect every file to meet the same security standards. On the other hand no file uploaded in 2005 will use an exploit developed in 2024.

So yes! Raise our standards! But do it in a sensible way and scope. Decide whether a new standard is to be met by old files and put some in quarantine or describe why a given new standard needs not be met by old files.

This seems to be a heap of work - way more than just formulating a new CSP. What can I do?

If you're interested in making the existing SVG filter more robust, by all means write a patch

In T117618#10107095, @Bawolff wrote:

If you're interested in making the existing SVG filter more robust, by all means write a patch

I'm not interested in writing a patch without previously made specs and design decisions. But if you'd give me a point where to start with, I'd probably spend some time on it.

In T117618#10081491, @ssingh wrote:

Following up on the patch by @TheDJ (thanks!) and per https://gerrit.wikimedia.org/r/c/operations/puppet/+/1059423/comment/61d3b1a3_eebb1d4b/, @Vgutierrez's comment:

so we would be sending the header content twice, on both -Report-Only and enforcing flavors, according to turnilo, upload.wm.o gets 47M hits per day for URLs matching ^/wikipedia/(el|fr|ru|it|de|uk|ja|id|he|fi|zh|test). The content of the non-PDF CSP header is 328 bytes long so we would be increasing our outbound traffic in ~ 14GB per day.

We are still discussing this in Traffic but sharing here for posterity and easier discussion.

In T117618#10102911, @Bawolff wrote:

To clarify, this only affects direct viewing of SVGs, and SVGs included via an <iframe> tag (or in theory <object>). CSP policies do not affect SVGs included via <img> tags. <img> tags already work in secure mode where external resources and scripts are not allowed ( https://svgwg.org/specs/integration/#processing-mode )

If CSP only affects direct viewing, wouldn’t it be safe to send headers only on direct viewing (if it’s possible to detect it reliably)? I guess <1% of the requests to upload.wikimedia.org are direct views (and even those are much less time-sensitive, since the user loads a single image, as opposed to an article with dozens of images).

In T117618#10081491, @ssingh wrote:

Following up on the patch by @TheDJ (thanks!) and per https://gerrit.wikimedia.org/r/c/operations/puppet/+/1059423/comment/61d3b1a3_eebb1d4b/, @Vgutierrez's comment:

so we would be sending the header content twice, on both -Report-Only and enforcing flavors, according to turnilo, upload.wm.o gets 47M hits per day for URLs matching ^/wikipedia/(el|fr|ru|it|de|uk|ja|id|he|fi|zh|test). The content of the non-PDF CSP header is 328 bytes long so we would be increasing our outbound traffic in ~ 14GB per day.

We are still discussing this in Traffic but sharing here for posterity and easier discussion.

On HTTP/2 (and HTTP/3) HTTP headers are compressed, so

• the second copy of the header content would compress to almost nothing
• the header itself would compress quite well, being less than those 328 bytes

On the other hand, HTTP/1.x clients would fully receive that hit.

If CSP only affects direct viewing, wouldn’t it be safe to send headers only on direct viewing (if it’s possible to detect it reliably)? I guess <1% of the requests to upload.wikimedia.org are direct views (and even those are much less time-sensitive, since the user loads a single image, as opposed to an article with dozens of images).

Its possible (using sec-fetch-dest header) however that's kind of sketchy and I wouldn't particularly reccomend it unless we absolutely had to.

In T117618#10287163, @Platonides wrote:

In T117618#10081491, @ssingh wrote:

Following up on the patch by @TheDJ (thanks!) and per https://gerrit.wikimedia.org/r/c/operations/puppet/+/1059423/comment/61d3b1a3_eebb1d4b/, @Vgutierrez's comment:

so we would be sending the header content twice, on both -Report-Only and enforcing flavors, according to turnilo, upload.wm.o gets 47M hits per day for URLs matching ^/wikipedia/(el|fr|ru|it|de|uk|ja|id|he|fi|zh|test). The content of the non-PDF CSP header is 328 bytes long so we would be increasing our outbound traffic in ~ 14GB per day.

We are still discussing this in Traffic but sharing here for posterity and easier discussion.

On HTTP/2 (and HTTP/3) HTTP headers are compressed, so

• the second copy of the header content would compress to almost nothing
• the header itself would compress quite well, being less than those 328 bytes

On the other hand, HTTP/1.x clients would fully receive that hit.

FWIW, if space is really at a premium here, we can just do (At least for SVG. Would need testing to make sure that it doesn't break direct views of video or PDFs on different browsers):
Content-Security-Policy: script-src 'none'; default-src data: 'unsafe-inline'

Drops the back compat stuff for older browsers, and the reporting probably nobody will read (If this breaks some images they can just file a ticket). Would break favicon though.

Commenting from Traffic's side: this is in some ways, a trivial patch for us because we are simply setting an additional header. The challenge here, though, is understanding the header itself and the associated ramifications of setting it and also keeping it updated. For that, the Security should be/needs to be consulted, so this patch currently blocks on that happening.

In T117618#11224896, @ssingh wrote:

Commenting from Traffic's side: this is in some ways, a trivial patch for us because we are simply setting an additional header. The challenge here, though, is understanding the header itself and the associated ramifications of setting it and also keeping it updated. For that, the Security should be/needs to be consulted, so this patch currently blocks on that happening.

Would you like to have a chat about this? I think we'd be fine with @Bawolff's suggested CSP in T117618#11072208, if we're fine breaking favicons. I'm not sure of what the best way to test this would be.

In T117618#11359861, @sbassett wrote:

In T117618#11224896, @ssingh wrote:

Commenting from Traffic's side: this is in some ways, a trivial patch for us because we are simply setting an additional header. The challenge here, though, is understanding the header itself and the associated ramifications of setting it and also keeping it updated. For that, the Security should be/needs to be consulted, so this patch currently blocks on that happening.

Would you like to have a chat about this? I think we'd be fine with @Bawolff's suggested CSP in T117618#11072208, if we're fine breaking favicons. I'm not sure of what the best way to test this would be.

Hi @sbassett. Yes that sounds good, let's set up a time to talk about this and move it forward.

In T117618#11360663, @ssingh wrote:

Hi @sbassett. Yes that sounds good, let's set up a time to talk about this and move it forward.

Booked a call for 2025-11-18.

Per the 2025-11-18 meeting, next steps are:

1. @ssingh and Traffic to clean up and optimize the most recent, existing vcl patch
2. Traffic to schedule deployment time for new config (likely within about 2 weeks likely the week of January 12th, 2026)
3. @ssingh and Product Safety and Integrity to alert relevant WMF / Community folks (specifically test engineers)
4. Product Safety and Integrity to evaluate relevant CSP log data and determine if a rollback is necessary (this should only be the case if there are extreme disruptions on testwiki or report-only logspam)

Thanks for the update @sbassett. Noting that Traffic requested the move to January.

Hey @ssingh - Any updates on getting this deployed from SRE's end? Thanks.

In T117618#11500741, @sbassett wrote:

Hey @ssingh - Any updates on getting this deployed from SRE's end? Thanks.

Hi @sbassett. Thanks for checking. The plan is to revamp this a bit and then deploy it next week. I will follow up with a specific date shortly.

In T117618#11500836, @ssingh wrote:

Hi @sbassett. Thanks for checking. The plan is to revamp this a bit and then deploy it next week. I will follow up with a specific date shortly.

Ok, this should be fine, but I'll note that nearly all of the Product Safety and Integrity team will be at our offsite in Europe next week.

In T117618#11500863, @sbassett wrote:

In T117618#11500836, @ssingh wrote:

Hi @sbassett. Thanks for checking. The plan is to revamp this a bit and then deploy it next week. I will follow up with a specific date shortly.

Ok, this should be fine, but I'll note that nearly all of the Product Safety and Integrity team will be at our offsite in Europe next week.

Ah that makes it less than ideal in a way. Let's plan for the mid of the week of Jan 19? That gives us some time to prep the patches as well and we can just stack them up and merge later.

In T117618#11500866, @ssingh wrote:

Ah that makes it less than ideal in a way. Let's plan for the mid of the week of Jan 19? That gives us some time to prep the patches as well and we can just stack them up and merge later.

Ok, sounds good. We can follow up then.

Mentioned in SAL (#wikimedia-operations) [2026-01-20T15:38:57Z] <sukhe> sudo cumin "A:cp" "disable-puppet 'merging CR 1059423'": T117618

Change #1059423 merged by Ssingh:

[operations/puppet@production] varnish: Add restrictive CSP to upload.wikimedia.org for testwiki only

https://gerrit.wikimedia.org/r/1059423

Mentioned in SAL (#wikimedia-operations) [2026-01-20T15:47:27Z] <sukhe> enable puppet on cp110[15].eqiad.wmnet: T117618

Mentioned in SAL (#wikimedia-operations) [2026-01-20T15:50:39Z] <sukhe> sudo cumin -b31 "A:cp" "run-puppet-agent --enable 'merging CR 1059423'": T117618

Hey @Bawolff - Wondering if you had any thoughts on some efficient ways to test this within Wikimedia production... Thanks.

Example file:
https://test.wikipedia.org/wiki/File:18KOZ.pdf
https://upload.wikimedia.org/wikipedia/test/b/b5/18KOZ.pdf

Current http response headers:

< content-security-policy: default-src 'none'; style-src 'unsafe-inline' data:; font-src data:; img-src data: https://upload.wikimedia.org/favicon.ico; connect-src https://upload.wikimedia.org/favicon.ico; media-src data: 'self'; object-src 'self'; report-uri https://commons.wikimedia.org/w/api.php?reportonly=0&source=image&action=cspreport&format=json&
< x-content-security-policy: default-src 'none'; style-src 'unsafe-inline' data:; font-src data:; img-src data: https://upload.wikimedia.org/favicon.ico; connect-src https://upload.wikimedia.org/favicon.ico; media-src data: 'self'; object-src 'self'; report-uri https://commons.wikimedia.org/w/api.php?reportonly=0&source=image&action=cspreport&format=json&

Other test wiki files respond with:
https://test.wikipedia.org/wiki/File:Wikitech-2021-blue-large-icon_(copy).png

Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline' data:; font-src data:; img-src data: https://upload.wikimedia.org/favicon.ico; connect-src https://upload.wikimedia.org/favicon.ico; media-src data: 'self'; sandbox; report-uri https://commons.wikimedia.org/w/api.php?reportonly=0&source=image&action=cspreport&format=json&
x-content-security-policy: default-src 'none'; style-src 'unsafe-inline' data:; font-src data:; img-src data: https://upload.wikimedia.org/favicon.ico; connect-src https://upload.wikimedia.org/favicon.ico; media-src data: 'self'; sandbox; report-uri https://commons.wikimedia.org/w/api.php?reportonly=0&source=image&action=cspreport&format=json&

Difference being that pdf's are allowed to be used from within object/embed/applet usage. By opening the https://upload.wikimedia.org/wikipedia/test/b/b5/18KOZ.pdf in Firefox, we can see that the viewer (an embedder) still works with CSP enforcement (this should fail without this exception).

German Wikipedia: https://de.wikipedia.org/wiki/Datei:Betreute_Diplom_Staatsexamensarbeiten_W_Schumacher.pdf
ttps://upload.wikimedia.org/wikipedia/de/thumb/e/ec/Betreute_Diplom_Staatsexamensarbeiten_W_Schumacher.pdf/page1-1086px-Betreute_Diplom_Staatsexamensarbeiten_W_Schumacher.pdf.jpg?20241019105354

Content-Security-Policy-Report-Only: default-src 'none'; style-src 'unsafe-inline' data:; font-src data:; img-src data: https://upload.wikimedia.org/favicon.ico; connect-src https://upload.wikimedia.org/favicon.ico; media-src data: 'self'; report-uri https://commons.wikimedia.org/w/api.php?reportonly=1&source=image&action=cspreport&format=json&
x-content-security-policy-report-only: default-src 'none'; style-src 'unsafe-inline' data:; font-src data:; img-src data: https://upload.wikimedia.org/favicon.ico; connect-src https://upload.wikimedia.org/favicon.ico; media-src data: 'self'; report-uri https://commons.wikimedia.org/w/api.php?reportonly=1&source=image&action=cspreport&format=json&

So German Wikipedia files are still in report-only mode and PDFs also carry the embed-src annotation,, so this too seems correct.

And then for SVG. https://test.wikipedia.org/wiki/File:Cswiki_veryactiveusers_20231212.svg, accessing the SVG directly via https://upload.wikimedia.org/wikipedia/test/f/f1/Cswiki_veryactiveusers_20231212.svg still seems to work directly in the browser.

Testing if bad SVG is effectively blocked by opening a bad SVG in the browser, is a bit more difficult, as we first need an svg with external resources, and then we can test if the console reports that connections to unalloyed resources is effectively blocked. Finding one in the repo's might be a bit of a chore... And uploading a new one might be blocked ?

Note that the version of the header prefixed with an x- is only needed for browsers from before 2013. Its probably not relavent anymore.

Difference being that pdf's are allowed to be used from within object/embed/applet usage. By opening the https://upload.wikimedia.org/wikipedia/test/b/b5/18KOZ.pdf in Firefox, we can see that the viewer (an embedder) still works with CSP enforcement (this should fail without this exception).

Historically it was chrome that refused to direct view pdfs without object-src. However i think that changed somewhere down the line.

Testing if bad SVG is effectively blocked by opening a bad SVG in the browser, is a bit more difficult, as we first need an svg with external resources, and then we can test if the console reports that connections to unalloyed resources is effectively blocked. Finding one in the repo's might be a bit of a chore... And uploading a new one might be blocked ?

I think

https://upload.wikimedia.org/wikipedia/commons/c/c0/Gnome-emblem-web.svg

Is an example of one that should have blocked image loads.

@TheDJ - Thanks for all of those test cases! These make sense to me, especially the first two.

In T117618#11578659, @Bawolff wrote:

https://upload.wikimedia.org/wikipedia/commons/c/c0/Gnome-emblem-web.svg

Is an example of one that should have blocked image loads.

So that definitely has some upload.w.o resource calls, but when I check various console output via Chrome's developer tools, they all 404 for me when I load Gnome-emblem-web.svg. It seems all of those alphanum-named PNGs might not exist anymore? As I would expect to see actual CSP violation reports within Chrome's developer tools console output as opposed to 404 errors.

So that definitely has some upload.w.o resource calls, but when I check various console output via Chrome's developer tools, they all 404 for me when I load Gnome-emblem-web.svg. It seems all of those alphanum-named PNGs might not exist anymore? As I would expect to see actual CSP violation reports within Chrome's developer tools console output as opposed to 404 errors.

They probably never existed (They don't follow mediawiki's file naming convention with the hashed directory and that goes way back to the beginning). It looks like the file was authored loading some local images, and then uploader forgot to delete the local references when they uploaded the file.

When CSP is in enforce mode I would expect only the CSP error, if its in report-only I would expect both. When i tested right now, no CSP header was served with the file.

In T117618#11585450, @Bawolff wrote:

when CSP is in enforce mode I would expect only the CSP error, if its in report-only I would expect both. When i tested right now, no CSP header was served with the file.

The file is not in the /wikipedia/test area of the uploads domain so I wouldn't expect it right now, as the patch doesn't provide the headers for that area yet.. We also can't easily move/copy this file to test.wikipedia.org and test it there, as we no longer accept it for upload. Maybe we can do an import of the image via maintenance script or something ?

I've been looking a bit at logstash, and I can't see any significant change within the noise that is already there. I did notice that upload.wikimedia.org came up quote often as a CSP violation but this was a pre existing situation and I'm not entirely sure what is causing that. It's also for png etc, but the reporting url seems to be a proper wikimedia source that is I open it doesn't report any violation errors whatsoever.

I can't seem to find the log channels with the non-reporting errors. Possibly, the enforcement channel is not handled yet by logstash ? I would expect to see at least some messages, but the channels is either not there, or there truly were 0 reports.

PS. there's quite a few concerning data collection urls that I did see in logstash btw. Data collecting extensions, viruses.. It does seem to me that this is something we should be stricter about.

PPS. This does show a slight rise in reports, but can't really pinpoint it to something that would be rooted in this change: https://logstash.wikimedia.org/goto/d18a764cbcab11ad8b7a45059cb78218

PPPS: I was able to confirm Brian's example with https://upload.wikimedia.org/wikipedia/de/8/8b/Disney_Hannah_Montana_Logo.svg, which shows similar behavior. But this was of course already blocked as before.

The only difference (other than wikipedia tests), should be with PDF. But we don't have people directly viewing PDFs that were uploaded on test.wikipedia.org that ALSO have connect/embed issues I think.

In general, I see no change. I also don't really see blocked traffic that shouldn't have been blocked. It's mostly fonts in svgs, or lots of stuff that has to do with extensions trying to do things. Would be good to verify the channel would be available in logstash or not, because otherwise we will be guessing.

In T117618#11597391, @TheDJ wrote:

In general, I see no change. I also don't really see blocked traffic that shouldn't have been blocked. It's mostly fonts in svgs, or lots of stuff that has to do with extensions trying to do things. Would be good to verify the channel would be available in logstash or not, because otherwise we will be guessing.

I'm not sure this would be logged with the standard x-/content-security-policy: headers as it is currently configured in varnish? SRE may need to add specific logging somewhere else for that?