Splitting off the `kafka-main` Debian Trixie upgrade work from {T419216} for better visibility
and consistency with {T426835} and {T417001}
Having upgraded kafka-main to kafka 3.7, we're implementing the final part of the upgrade plan enacted by the Kafka upgrade WG, which is to upgrade the brokers to Debian Trixie / jdk21.
[x] Upgrade kafka-main to Kafka 3.7 (codfw > eqiad) in {T419216}
Additional migrations/upgrades:
[] Upgrade all brokers to Debian Trixie {T419212}.
[] VLAN migrations for `kafka-main1008-1009` {T421711}
The VLAN migration practically means running the reimage cookbook with the `--move-vlan` flag. This will change the IP of the host which needs additional attention:
- Update `kafka_brokers_main` in `hieradata/common.yaml`, run puppet on `conf100*` and the other brokers in eqiad
- Update [[ https://wikitech.wikimedia.org/wiki/External-services | external-services ]] in k8s.
- Run puppet on deployment server(s)
- Deploy the external service release to update network policies to include the new broker IP (and remove the old)
- [[ https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_restart | Roll restart ]] eventgate-main out of precaution since it's pretty picky with loosing brokers
Post-upgrade cleanup:
[] Clean up per-host hiera configs, move to role level
**Cluster states**:
**kafka-main codfw: **
| Kafka Broker | Confluent distribution 77 | Inter-broker protocol | Host-level override patch | Debian Trixie |
| kafka-main2006.codfw.wmnet | Upgraded ✅ | 3.7 ✅ | [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1288917 | 1288917 ]] | |
| kafka-main2007.codfw.wmnet | Upgraded ✅ | 3.7 ✅ | [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1288918 | 1288918 ]] | |
| kafka-main2008.codfw.wmnet | Upgraded ✅ | 3.7 ✅ | [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1288919 | 1288919 ]] | |
| kafka-main2009.codfw.wmnet | Upgraded ✅ | 3.7 ✅ | [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1288920 | 1288920 ]] | |
| kafka-main2010.codfw.wmnet | Upgraded ✅ | 3.7 ✅ | [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1288921 | 1288921 ]] | |
**kafka-main eqiad: **
| Kafka Broker | Confluent distribution 77 | Inter-broker protocol | Host-level override patch | Debian Trixie |
| kafka-main1006.eqiad.wmnet | Upgraded ✅ | 3.7 ✅ | [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1285474 | 1285474 ]] | |
| kafka-main1007.eqiad.wmnet | Upgraded ✅ | 3.7 ✅ | [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1285475 | 1285475 ]] | |
| kafka-main1008.eqiad.wmnet | Upgraded ✅ | 3.7 ✅ | [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1285476 | 1285476 ]] | |
| kafka-main1009.eqiad.wmnet | Upgraded ✅ | 3.7 ✅ | [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1285477 | 1285477 ]] | |
| kafka-main1010.eqiad.wmnet | Upgraded ✅ | 3.7 ✅ | [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1285478 | 1285478 ]] | |