While I was investigating an issue in beta cluster (T360595#9672032) I checked config of a kafka broker in production (kafka-logging1004.eqiad.wmnet) and it had this config:
ssl.truststore.location=/etc/ssl/localcerts/wmf-java-cacerts ssl.truststore.password=changeit
which worried me as password of stuff in production shouldn't be "changeme". I wanted to check where this is coming from and it seems that it's actually coming from sslcert::trusted_ca (and the variable has not been overwritten in private repo except for the presto hosts, hadoop and analytics tests hosts). That puppet module adds these files to every production host that has java included (600 hosts: https://debmonitor.wikimedia.org/packages/ca-certificates-java) .
For example, I logged in to logstash1036.eqiad.wmnet and ran this:
keytool -v -list -keystore /etc/ssl/certs/java/cacerts
and used the aforementioned password and it worked. The problem is not the ability to see content of the truststore (also not to be confused with keystore), it is that anyone with that password and access to the host can add a trusted CA to the list via keytool (I'm pretty basic at java, maybe I'm missing something obvious but I didn't want to test adding a problematic CA in production.) That means any java application running, will automatically accept attacker's CA and all certificates issued by it as valid