In T111820 @csteipp rightly pointed out that there is value in setting CSP headers to disallow framing on non-HTML/SVG responses. Unless there is a use case for custom CSP headers on JSON or other non-HTML/SVG responses, I would propose to unconditionally set very restrictive CSP headers on anything but /^(?:text\/html|image\/svg)/i:

Content-Security-Policy: default-src 'none'; frame-ancestors 'none'