@Joe added IPv6 support for production Kafka yesterday (https://gerrit.wikimedia.org/r/c/operations/puppet/+/684814), but WMCS does not have IPv6 so Puppet is failing on deployment-deploy*:
taavi@deployment-deploy01:~$ sudo run-puppet-agent Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, DNS lookup failed for deployment-kafka-jumbo-1.deployment-prep.eqiad1.wikimedia.cloud Resolv::DNS::Resource::IN::AAAA (file: /etc/puppet/modules/profile/manifests/kubernetes/deployment_server/global_config.pp, line: 80, column: 23) on node deployment-deploy01.deployment-prep.eqiad.wmflabs Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run