After further investigation we've noticed that the problem is not reproducible forcing a cache miss by adding some random query parameters. Further, we've tried purging the affected object from a varnish frontend and fetching it again; the issue couldn't be reproduced.

The two varnish backends in the path from esams to eqiad are cp1072 and cp3037, These are the possible scenarios we think might have triggered the problem:

1. cp1072 temporarily emitted CE:gzip with this object then later stopped doing so (for the same cache object)
2. cp3037 temporarily added CE:gzip on reception, then was later evicted/purged and cleaned itself up
3. cp3037 may or may not have ever evicted/purged since, but temporarily added CE:gzip to its response to several frontends
4. several frontends all added CE:gzip to their cache objects on reception from cp3037, and then whatever triggers that went away

Scenario 4) seems unlikely because that would imply multiple temporary issues on all frontends. Scenario 1) also is not the most likely one given that we know for sure (because of Age), that cp1072 is still using the same unevicted/unexpired cache object it was at the time the problem started, and doesn't have bad output now.

The specific repro URL for the Serbia map has been PURGEd now to clear up the issue for users, since we're not getting much debug value out of keeping it broken.

To be clearer about what was debugged on IRC: this wasn't a case of actual bad gzip encoding. The object contents in all affected caches were always the correct, uncompressed PNG data. The issue was just that a Content-Encoding: gzip header was on the objects (and thus the outputs) in the affected frontends for unknown reasons, which caused clients to attempt to interpret the otherwise-ok PNG data as gzipped content (and thus fail to gzip decode what isn't gzipped).