The Varnish caching of banners served via Special:BannerLoader is "hit-for-pass", which means cachability of an item isn't known until it's actually served up by the MW backend. Cache header values are set like so, in SpecialBannerLoader.
We should review this setup to see if it's currently optimal. Review should include any relevant vlc (varnish setup) code that comes into play, too.
As part of this update, we might well turn off caching for output for at caught errors.