Page MenuHomePhabricator
Paste P96506

(An Untitled Masterwork)
ActivePublic
Actions

Authored by tappof on Wed, Sep 23, 8:00 AM.
Project Tags
None
Referenced Files
F104436071: raw-paste-data.txt
Wed, Sep 23, 8:00 AM
Subscribers
None
root@deploy1003:~# kube-env admin eqiad
root@deploy1003:~# cd /srv/deployment-charts/helmfile.d/admin_ng
root@deploy1003:/srv/deployment-charts/helmfile.d/admin_ng# helmfile -e eqiad -lname=external-services diff
skipping missing values file matching "external-services/values.yaml"
skipping missing values file matching "values/eqiad/external-services-values.yaml"
skipping missing values file matching "/etc/helmfile-defaults/private/admin/external-services/eqiad.yaml"
Comparing release=external-services, chart=wmf-stable/external-services, namespace=external-services
external-services, kafka-logging-eqiad, Endpoints (v1) has changed:
...
- ip: 10.64.131.13
- ip: 10.64.133.11
- ip: 10.64.135.13
- ip: 10.64.16.176
- ip: 10.64.16.205
+ - ip: 10.64.169.4
+ - ip: 10.64.181.5
- ip: 2620:0:861:102:10:64:16:176
- ip: 2620:0:861:102:10:64:16:205
- ip: 2620:0:861:10a:10:64:131:13
- ip: 2620:0:861:10c:10:64:133:11
- ip: 2620:0:861:10e:10:64:135:13
+ - ip: 2620:0:861:119:10:64:169:4
+ - ip: 2620:0:861:13b:10:64:181:5
ports:
- port: 9092
name: plaintext
protocol: TCP
- port: 9093
...
root@deploy1003:/srv/deployment-charts/helmfile.d/admin_ng# helmfile -e eqiad -lname=external-services sync
skipping missing values file matching "external-services/values.yaml"
skipping missing values file matching "values/eqiad/external-services-values.yaml"
skipping missing values file matching "/etc/helmfile-defaults/private/admin/external-services/eqiad.yaml"
Upgrading release=external-services, chart=wmf-stable/external-services, namespace=external-services
Release "external-services" has been upgraded. Happy Helming!
NAME: external-services
LAST DEPLOYED: Wed Sep 23 07:58:37 2026
NAMESPACE: external-services
STATUS: deployed
REVISION: 39
TEST SUITE: None
Listing releases matching ^external-services$
external-services external-services 39 2026-09-23 07:58:37.031164671 +0000 UTC deployed external-services-0.0.3
UPDATED RELEASES:
NAME NAMESPACE CHART VERSION DURATION
external-services external-services wmf-stable/external-services 0.0.3 14s
root@deploy1003:/srv/deployment-charts/helmfile.d/admin_ng# kube-env admin codfw
root@deploy1003:/srv/deployment-charts/helmfile.d/admin_ng# helmfile -e codfw -lname=external-services diff
skipping missing values file matching "external-services/values.yaml"
skipping missing values file matching "values/codfw/external-services-values.yaml"
skipping missing values file matching "/etc/helmfile-defaults/private/admin/external-services/codfw.yaml"
Comparing release=external-services, chart=wmf-stable/external-services, namespace=external-services
external-services, kafka-logging-eqiad, Endpoints (v1) has changed:
...
- ip: 10.64.131.13
- ip: 10.64.133.11
- ip: 10.64.135.13
- ip: 10.64.16.176
- ip: 10.64.16.205
+ - ip: 10.64.169.4
+ - ip: 10.64.181.5
- ip: 2620:0:861:102:10:64:16:176
- ip: 2620:0:861:102:10:64:16:205
- ip: 2620:0:861:10a:10:64:131:13
- ip: 2620:0:861:10c:10:64:133:11
- ip: 2620:0:861:10e:10:64:135:13
+ - ip: 2620:0:861:119:10:64:169:4
+ - ip: 2620:0:861:13b:10:64:181:5
ports:
- port: 9092
name: plaintext
protocol: TCP
- port: 9093
...
root@deploy1003:/srv/deployment-charts/helmfile.d/admin_ng# helmfile -e codfw -lname=external-services sync
skipping missing values file matching "external-services/values.yaml"
skipping missing values file matching "values/codfw/external-services-values.yaml"
skipping missing values file matching "/etc/helmfile-defaults/private/admin/external-services/codfw.yaml"
Upgrading release=external-services, chart=wmf-stable/external-services, namespace=external-services
Release "external-services" has been upgraded. Happy Helming!
NAME: external-services
LAST DEPLOYED: Wed Sep 23 07:59:33 2026
NAMESPACE: external-services
STATUS: deployed
REVISION: 54
TEST SUITE: None
Listing releases matching ^external-services$
external-services external-services 54 2026-09-23 07:59:33.533999712 +0000 UTC deployed external-services-0.0.3
UPDATED RELEASES:
NAME NAMESPACE CHART VERSION DURATION
external-services external-services wmf-stable/external-services 0.0.3 34s

Event Timeline

tappof created this paste.EditedWed, Sep 23, 8:00 AM
tappof@deploy1003:~$ scap sync-world --k8s-only --k8s-confirm-diff -Dbuild_mw_container_image:False
           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

Log message (press enter for none): T432444 - Provision kafka-logging100[6-8]
08:01:16 Collecting helmfile diffs for review
=== Diff for codfw/mw-debug-next in testservers ===
Comparing release=next, chart=wmf-stable/mediawiki, namespace=mw-debug
mw-debug, mediawiki-next, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-debug, mediawiki-next-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-debug, mw-debug.codfw.next, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: b0e2058c1f34b7b22453767ac5c766accd68ae80cbb65a5e35046d8c48711893

-         checksum/rsyslog: 3af06643e2f18bfe179a512eb3be6c41872ad73243adcbeb4387960b14dc508d
+         checksum/rsyslog: 8e4a701e0672985705b700993070f5f02bdb7569920de6ae4c46381a3b7ef8d6

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: e378cb44fb5caf67d937a8ee992aef6192ce3af01c942c171b64ad07d9fa54d1
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-debug-pinkunicorn in testservers ===
Comparing release=pinkunicorn, chart=wmf-stable/mediawiki, namespace=mw-debug
mw-debug, mediawiki-pinkunicorn, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-debug, mediawiki-pinkunicorn-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-debug, mw-debug.codfw.pinkunicorn, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: a7cf229f0939198b9de78b0039392431e9c12a0bedaabba0fb2e450474610b58

-         checksum/rsyslog: 3af06643e2f18bfe179a512eb3be6c41872ad73243adcbeb4387960b14dc508d
+         checksum/rsyslog: 8e4a701e0672985705b700993070f5f02bdb7569920de6ae4c46381a3b7ef8d6

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 1b7f84451fb220da510460ee4db08d47af93b9ac8a3fcec3838cedc8921b2ef7
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-misc-main in testservers ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-misc
mw-misc, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-misc, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-misc, mw-misc.codfw.main, Deployment (apps) has changed:
...
          checksum/sites: 178e89d4b68c7beefbf2d4c3e1553cd8d82ab319284a2e2b768289f66009b93b
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 5daf2ba1230e762df91c8b91d1410455ec7728c97b18c19b7d60a722a2bbbabd

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 865c8c7b1ae3f3d0ce4cd7e18fbd675a630b680ec450871886f7be773fe19418
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-pretrain-debug in testservers ===
Comparing release=debug, chart=wmf-stable/mediawiki, namespace=mw-pretrain
mw-pretrain, mediawiki-debug, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-pretrain, mediawiki-debug-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-pretrain, mw-pretrain.codfw.debug, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 37bc70908c1dfa826bdb72ca7977c37249477a954b81846fca4f6d28f7612bc5

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: c39f3e5efed899678334b28965b226f1947d6913b465c389459e198c12acaac3
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-debug-next in testservers ===
Comparing release=next, chart=wmf-stable/mediawiki, namespace=mw-debug
mw-debug, mediawiki-next, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-debug, mediawiki-next-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-debug, mw-debug.eqiad.next, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: b0e2058c1f34b7b22453767ac5c766accd68ae80cbb65a5e35046d8c48711893

-         checksum/rsyslog: 631a9bc54502c2743d6f223bdf8a453c8f4929d282b0103c3bb5628761e6274d
+         checksum/rsyslog: b9617f9e1d7cb88c8dda920f903397e38ca2f95986d4389b0fcb9922a0006bcd

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: e378cb44fb5caf67d937a8ee992aef6192ce3af01c942c171b64ad07d9fa54d1
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-debug-pinkunicorn in testservers ===
Comparing release=pinkunicorn, chart=wmf-stable/mediawiki, namespace=mw-debug
mw-debug, mediawiki-pinkunicorn, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-debug, mediawiki-pinkunicorn-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-debug, mw-debug.eqiad.pinkunicorn, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: a7cf229f0939198b9de78b0039392431e9c12a0bedaabba0fb2e450474610b58

-         checksum/rsyslog: 631a9bc54502c2743d6f223bdf8a453c8f4929d282b0103c3bb5628761e6274d
+         checksum/rsyslog: b9617f9e1d7cb88c8dda920f903397e38ca2f95986d4389b0fcb9922a0006bcd

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 1b7f84451fb220da510460ee4db08d47af93b9ac8a3fcec3838cedc8921b2ef7
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-misc-main in testservers ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-misc
mw-misc, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-misc, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-misc, mw-misc.eqiad.main, Deployment (apps) has changed:
...
          checksum/sites: 178e89d4b68c7beefbf2d4c3e1553cd8d82ab319284a2e2b768289f66009b93b
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 5daf2ba1230e762df91c8b91d1410455ec7728c97b18c19b7d60a722a2bbbabd

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 865c8c7b1ae3f3d0ce4cd7e18fbd675a630b680ec450871886f7be773fe19418
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-pretrain-debug in testservers ===
Comparing release=debug, chart=wmf-stable/mediawiki, namespace=mw-pretrain
mw-pretrain, mediawiki-debug, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-pretrain, mediawiki-debug-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-pretrain, mw-pretrain.eqiad.debug, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 37bc70908c1dfa826bdb72ca7977c37249477a954b81846fca4f6d28f7612bc5

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: c39f3e5efed899678334b28965b226f1947d6913b465c389459e198c12acaac3
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-api-ext-canary in canaries ===
Comparing release=canary, chart=wmf-stable/mediawiki, namespace=mw-api-ext
mw-api-ext, mediawiki-canary, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-ext, mediawiki-canary-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-ext, mw-api-ext.codfw.canary, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 1edea43b27b8aa742adc7623b4b7fcbb1b44df3041548745136ae8cef2c1edc5

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 1ae896e7872f769ea2a14b5093ca8c553fd60f728fb97cfe14007739ab87986c
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-api-int-canary in canaries ===
Comparing release=canary, chart=wmf-stable/mediawiki, namespace=mw-api-int
mw-api-int, mediawiki-canary, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-int, mediawiki-canary-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-int, mw-api-int.codfw.canary, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 1edea43b27b8aa742adc7623b4b7fcbb1b44df3041548745136ae8cef2c1edc5

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: e2ab5a81915816cb243af5782e5829d0b861993b67133edfa41d165813ab36ca
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-jobrunner-canary in canaries ===
Comparing release=canary, chart=wmf-stable/mediawiki, namespace=mw-jobrunner
mw-jobrunner, mediawiki-canary, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-jobrunner, mediawiki-canary-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-jobrunner, mw-jobrunner.codfw.canary, Deployment (apps) has changed:
...
          checksum/sites: 24b050862dde81ba6b62639141b095614a65811b5593afa7fca3923d03e98594
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 1edea43b27b8aa742adc7623b4b7fcbb1b44df3041548745136ae8cef2c1edc5

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: e80c3cb55f86c8b1f460ad756e77c113ffacdbc3a3a8e3653b193c1200da3e1e
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-pretrain-canary in canaries ===
Comparing release=canary, chart=wmf-stable/mediawiki, namespace=mw-pretrain
mw-pretrain, mediawiki-canary, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-pretrain, mediawiki-canary-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-pretrain, mw-pretrain.codfw.canary, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 1edea43b27b8aa742adc7623b4b7fcbb1b44df3041548745136ae8cef2c1edc5

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 4ad2401628813bbf615db02037ef6bf3b8fa52c38bfc46699884324d027ccb2f
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-pretrain-jobrunner-canary in canaries ===
Comparing release=jobrunner-canary, chart=wmf-stable/mediawiki, namespace=mw-pretrain
mw-pretrain, mediawiki-jobrunner-canary, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-pretrain, mediawiki-jobrunner-canary-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-pretrain, mw-pretrain.codfw.jobrunner-canary, Deployment (apps) has changed:
...
          checksum/sites: 54a648b1867038e10cadab5506e14ef8b962065f95a5904830be218c8a68a179
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 4a673d8057d8dad30e2c9acc448fb7cff4178ad7e06e58473105ce882daf2506

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 52212e520c01286896fb3fba87e2417b7196094ac1f18d38647741a16eef6cb1
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-web-canary in canaries ===
Comparing release=canary, chart=wmf-stable/mediawiki, namespace=mw-web
mw-web, mediawiki-canary, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-web, mediawiki-canary-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-web, mw-web.codfw.canary, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 1edea43b27b8aa742adc7623b4b7fcbb1b44df3041548745136ae8cef2c1edc5

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 65488efa691aae38603b6b6b08ca4cc28ed52860f0958e359342df09f3183ebd
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-api-ext-canary in canaries ===
Comparing release=canary, chart=wmf-stable/mediawiki, namespace=mw-api-ext
mw-api-ext, mediawiki-canary, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-ext, mediawiki-canary-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-ext, mw-api-ext.eqiad.canary, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 1edea43b27b8aa742adc7623b4b7fcbb1b44df3041548745136ae8cef2c1edc5

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 1ae896e7872f769ea2a14b5093ca8c553fd60f728fb97cfe14007739ab87986c
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-api-int-canary in canaries ===
Comparing release=canary, chart=wmf-stable/mediawiki, namespace=mw-api-int
mw-api-int, mediawiki-canary, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-int, mediawiki-canary-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-int, mw-api-int.eqiad.canary, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 1edea43b27b8aa742adc7623b4b7fcbb1b44df3041548745136ae8cef2c1edc5

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: e2ab5a81915816cb243af5782e5829d0b861993b67133edfa41d165813ab36ca
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-jobrunner-canary in canaries ===
Comparing release=canary, chart=wmf-stable/mediawiki, namespace=mw-jobrunner
mw-jobrunner, mediawiki-canary, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-jobrunner, mediawiki-canary-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-jobrunner, mw-jobrunner.eqiad.canary, Deployment (apps) has changed:
...
          checksum/sites: 24b050862dde81ba6b62639141b095614a65811b5593afa7fca3923d03e98594
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 1edea43b27b8aa742adc7623b4b7fcbb1b44df3041548745136ae8cef2c1edc5

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: e80c3cb55f86c8b1f460ad756e77c113ffacdbc3a3a8e3653b193c1200da3e1e
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-pretrain-canary in canaries ===
Comparing release=canary, chart=wmf-stable/mediawiki, namespace=mw-pretrain
mw-pretrain, mediawiki-canary, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-pretrain, mediawiki-canary-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-pretrain, mw-pretrain.eqiad.canary, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 1edea43b27b8aa742adc7623b4b7fcbb1b44df3041548745136ae8cef2c1edc5

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 4ad2401628813bbf615db02037ef6bf3b8fa52c38bfc46699884324d027ccb2f
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-pretrain-jobrunner-canary in canaries ===
Comparing release=jobrunner-canary, chart=wmf-stable/mediawiki, namespace=mw-pretrain
mw-pretrain, mediawiki-jobrunner-canary, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-pretrain, mediawiki-jobrunner-canary-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-pretrain, mw-pretrain.eqiad.jobrunner-canary, Deployment (apps) has changed:
...
          checksum/sites: 54a648b1867038e10cadab5506e14ef8b962065f95a5904830be218c8a68a179
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 4a673d8057d8dad30e2c9acc448fb7cff4178ad7e06e58473105ce882daf2506

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 52212e520c01286896fb3fba87e2417b7196094ac1f18d38647741a16eef6cb1
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-web-canary in canaries ===
Comparing release=canary, chart=wmf-stable/mediawiki, namespace=mw-web
mw-web, mediawiki-canary, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-web, mediawiki-canary-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-web, mw-web.eqiad.canary, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 1edea43b27b8aa742adc7623b4b7fcbb1b44df3041548745136ae8cef2c1edc5

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 65488efa691aae38603b6b6b08ca4cc28ed52860f0958e359342df09f3183ebd
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-api-ext-main in production ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-api-ext
mw-api-ext, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-ext, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-ext, mw-api-ext.codfw.main, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 5daf2ba1230e762df91c8b91d1410455ec7728c97b18c19b7d60a722a2bbbabd

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: db1573aa2d6670cf78b58eaf6e064d3bac489a1d21cdebd47f16733dca9653f2
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-api-ext-migration in production ===
Comparing release=migration, chart=wmf-stable/mediawiki, namespace=mw-api-ext
mw-api-ext, mediawiki-migration, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-ext, mediawiki-migration-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-ext, mw-api-ext.codfw.migration, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 650a3ee70dec691ce505048c2ca81e7accc67dae073dff1b1c6d219261974487

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: c4be563c6ca26dae6fe7c8eb0aaf4620d6519b511762f1318fabf242aa24f8a8
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-api-ext-next in production ===
Comparing release=next, chart=wmf-stable/mediawiki, namespace=mw-api-ext
mw-api-ext, mediawiki-next, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-ext, mediawiki-next-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-ext, mw-api-ext.codfw.next, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: b0e2058c1f34b7b22453767ac5c766accd68ae80cbb65a5e35046d8c48711893

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: acde6b5e7a2dbb05664dd7990c15aafd69fc97b5824db40f46b3ee01cae3d92f
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-api-int-main in production ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-api-int
mw-api-int, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-int, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-int, mw-api-int.codfw.main, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 5daf2ba1230e762df91c8b91d1410455ec7728c97b18c19b7d60a722a2bbbabd

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: a81f23ecd9e0766df48fd8a6e384a3640c89bc434b8e182b0324c3aebfefb047
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-api-int-migration in production ===
Comparing release=migration, chart=wmf-stable/mediawiki, namespace=mw-api-int
mw-api-int, mediawiki-migration, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-int, mediawiki-migration-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-int, mw-api-int.codfw.migration, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 650a3ee70dec691ce505048c2ca81e7accc67dae073dff1b1c6d219261974487

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: d84153d4a7000ca1008bb3e155c1fe8ce0fb1c11919170940f8056684412347f
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== No changes for codfw/mw-cron-main in production ===
=== Diff for codfw/mw-jobrunner-main in production ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-jobrunner
mw-jobrunner, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-jobrunner, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-jobrunner, mw-jobrunner.codfw.main, Deployment (apps) has changed:
...
          checksum/sites: 24b050862dde81ba6b62639141b095614a65811b5593afa7fca3923d03e98594
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 5daf2ba1230e762df91c8b91d1410455ec7728c97b18c19b7d60a722a2bbbabd

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 0786b6793f77d45eb30cd3a0d59d1b75f024dbfec1cf0d6f551661d0a04fca22
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-jobrunner-migration in production ===
Comparing release=migration, chart=wmf-stable/mediawiki, namespace=mw-jobrunner
mw-jobrunner, mediawiki-migration, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-jobrunner, mediawiki-migration-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-jobrunner, mw-jobrunner.codfw.migration, Deployment (apps) has changed:
...
          checksum/sites: 24b050862dde81ba6b62639141b095614a65811b5593afa7fca3923d03e98594
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 650a3ee70dec691ce505048c2ca81e7accc67dae073dff1b1c6d219261974487

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 700f260d8ff69d087cc1530ba96355fe95fe0a255a04c870e30b24239dd5d4db
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-pretrain-jobrunner in production ===
Comparing release=jobrunner, chart=wmf-stable/mediawiki, namespace=mw-pretrain
mw-pretrain, mediawiki-jobrunner, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-pretrain, mediawiki-jobrunner-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-pretrain, mw-pretrain.codfw.jobrunner, Deployment (apps) has changed:
...
          checksum/sites: 54a648b1867038e10cadab5506e14ef8b962065f95a5904830be218c8a68a179
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 8c7d7fadbc064dc56f0d7f60ba7a4fb3c5d876586490300a5a47f52ae8808558

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 3ddb9c46ef540ac75b7ca33591d47143b9c7f09d79d40962ee7f460a69cedc9c
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-pretrain-main in production ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-pretrain
mw-pretrain, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-pretrain, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-pretrain, mw-pretrain.codfw.main, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 5daf2ba1230e762df91c8b91d1410455ec7728c97b18c19b7d60a722a2bbbabd

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: b760d5cab89ebaac0bb81cf75d82631568ca6f601a69f0a108e06702d0f058cf
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-videoscaler-main in production ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-videoscaler
mw-videoscaler, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-videoscaler, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...


=== Diff for codfw/mw-web-main in production ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-web
mw-web, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-web, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-web, mw-web.codfw.main, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 5daf2ba1230e762df91c8b91d1410455ec7728c97b18c19b7d60a722a2bbbabd

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: bfcb78c0fff835977559f96bc26dfe6dfc845b99674c3aba1a05d4c1da2d503d
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-web-migration in production ===
Comparing release=migration, chart=wmf-stable/mediawiki, namespace=mw-web
mw-web, mediawiki-migration, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-web, mediawiki-migration-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-web, mw-web.codfw.migration, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 650a3ee70dec691ce505048c2ca81e7accc67dae073dff1b1c6d219261974487

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 9c8e8dbd358b600b1617a8dde80b35103f7232b0492a48f791793f4e3c26006f
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-web-next in production ===
Comparing release=next, chart=wmf-stable/mediawiki, namespace=mw-web
mw-web, mediawiki-next, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-web, mediawiki-next-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-web, mw-web.codfw.next, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: b0e2058c1f34b7b22453767ac5c766accd68ae80cbb65a5e35046d8c48711893

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 598218831b90ea1b9831d10febb3c1dc22516adf8b8fe423869abc8c16658d49
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-wikifunctions-group0 in production ===
Comparing release=group0, chart=wmf-stable/mediawiki, namespace=mw-wikifunctions
mw-wikifunctions, mediawiki-group0, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-wikifunctions, mediawiki-group0-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-wikifunctions, mw-wikifunctions.codfw.group0, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 845f0d82e0845fbf631a7ca344b1d552023285a297c077bd3baf5e88451ea8a9

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 764577aac83b0610870f0341b4bfa8cad2caef27b8e323ff0c87683dec50fc23
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-wikifunctions-group1 in production ===
Comparing release=group1, chart=wmf-stable/mediawiki, namespace=mw-wikifunctions
mw-wikifunctions, mediawiki-group1, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-wikifunctions, mediawiki-group1-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-wikifunctions, mw-wikifunctions.codfw.group1, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 56da13f9b514e2ac8f0340bdfc3bc79f3b08a37f21a16dcd93417592c6af7f47

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: bdac9f1a0206d04a444a4d75ce16aedee3f1a77f94c7d01ced2209a8282733b2
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for codfw/mw-wikifunctions-group2 in production ===
Comparing release=group2, chart=wmf-stable/mediawiki, namespace=mw-wikifunctions
mw-wikifunctions, mediawiki-group2, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-wikifunctions, mediawiki-group2-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-wikifunctions, mw-wikifunctions.codfw.group2, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 22225e7b3acc07474f6d989066bd021e4996b34cac99745f5032fa6b5dfb1076

-         checksum/rsyslog: f51320bbd54515e0c0fb3bc9b77a0a829e687e3b6adc80d1ce49fee85803fb31
+         checksum/rsyslog: 6497ed63811f97ffa41a3081847e0ce9aba627573d8b2db6b7760bdf37566721

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 67f9ea7b34ed08c0bbbe853d55c19f9fc76eb4b2366fade0a87c0fb6713e5e63
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for dse-k8s-eqiad/mediawiki-dumps-legacy-production in production ===
Comparing release=production, chart=wmf-stable/mediawiki, namespace=mediawiki-dumps-legacy
mediawiki-dumps-legacy, mediawiki-production-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...


=== Diff for eqiad/mw-api-ext-main in production ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-api-ext
mw-api-ext, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-ext, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-ext, mw-api-ext.eqiad.main, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 5daf2ba1230e762df91c8b91d1410455ec7728c97b18c19b7d60a722a2bbbabd

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: db1573aa2d6670cf78b58eaf6e064d3bac489a1d21cdebd47f16733dca9653f2
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-api-ext-migration in production ===
Comparing release=migration, chart=wmf-stable/mediawiki, namespace=mw-api-ext
mw-api-ext, mediawiki-migration, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-ext, mediawiki-migration-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-ext, mw-api-ext.eqiad.migration, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 650a3ee70dec691ce505048c2ca81e7accc67dae073dff1b1c6d219261974487

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: c4be563c6ca26dae6fe7c8eb0aaf4620d6519b511762f1318fabf242aa24f8a8
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-api-ext-next in production ===
Comparing release=next, chart=wmf-stable/mediawiki, namespace=mw-api-ext
mw-api-ext, mediawiki-next, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-ext, mediawiki-next-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-ext, mw-api-ext.eqiad.next, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: b0e2058c1f34b7b22453767ac5c766accd68ae80cbb65a5e35046d8c48711893

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: acde6b5e7a2dbb05664dd7990c15aafd69fc97b5824db40f46b3ee01cae3d92f
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-api-int-main in production ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-api-int
mw-api-int, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-int, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-int, mw-api-int.eqiad.main, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 5daf2ba1230e762df91c8b91d1410455ec7728c97b18c19b7d60a722a2bbbabd

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: a81f23ecd9e0766df48fd8a6e384a3640c89bc434b8e182b0324c3aebfefb047
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-api-int-migration in production ===
Comparing release=migration, chart=wmf-stable/mediawiki, namespace=mw-api-int
mw-api-int, mediawiki-migration, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-api-int, mediawiki-migration-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-api-int, mw-api-int.eqiad.migration, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 650a3ee70dec691ce505048c2ca81e7accc67dae073dff1b1c6d219261974487

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: d84153d4a7000ca1008bb3e155c1fe8ce0fb1c11919170940f8056684412347f
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-cron-main in production ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-cron
mw-cron, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...


=== Diff for eqiad/mw-jobrunner-main in production ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-jobrunner
mw-jobrunner, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-jobrunner, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-jobrunner, mw-jobrunner.eqiad.main, Deployment (apps) has changed:
...
          checksum/sites: 24b050862dde81ba6b62639141b095614a65811b5593afa7fca3923d03e98594
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 5daf2ba1230e762df91c8b91d1410455ec7728c97b18c19b7d60a722a2bbbabd

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 0786b6793f77d45eb30cd3a0d59d1b75f024dbfec1cf0d6f551661d0a04fca22
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-jobrunner-migration in production ===
Comparing release=migration, chart=wmf-stable/mediawiki, namespace=mw-jobrunner
mw-jobrunner, mediawiki-migration, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-jobrunner, mediawiki-migration-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-jobrunner, mw-jobrunner.eqiad.migration, Deployment (apps) has changed:
...
          checksum/sites: 24b050862dde81ba6b62639141b095614a65811b5593afa7fca3923d03e98594
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 650a3ee70dec691ce505048c2ca81e7accc67dae073dff1b1c6d219261974487

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 700f260d8ff69d087cc1530ba96355fe95fe0a255a04c870e30b24239dd5d4db
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-pretrain-jobrunner in production ===
Comparing release=jobrunner, chart=wmf-stable/mediawiki, namespace=mw-pretrain
mw-pretrain, mediawiki-jobrunner, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-pretrain, mediawiki-jobrunner-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-pretrain, mw-pretrain.eqiad.jobrunner, Deployment (apps) has changed:
...
          checksum/sites: 54a648b1867038e10cadab5506e14ef8b962065f95a5904830be218c8a68a179
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 8c7d7fadbc064dc56f0d7f60ba7a4fb3c5d876586490300a5a47f52ae8808558

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 3ddb9c46ef540ac75b7ca33591d47143b9c7f09d79d40962ee7f460a69cedc9c
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-pretrain-main in production ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-pretrain
mw-pretrain, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-pretrain, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-pretrain, mw-pretrain.eqiad.main, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 5daf2ba1230e762df91c8b91d1410455ec7728c97b18c19b7d60a722a2bbbabd

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: b760d5cab89ebaac0bb81cf75d82631568ca6f601a69f0a108e06702d0f058cf
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-videoscaler-main in production ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-videoscaler
mw-videoscaler, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-videoscaler, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...


=== Diff for eqiad/mw-web-main in production ===
Comparing release=main, chart=wmf-stable/mediawiki, namespace=mw-web
mw-web, mediawiki-main, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-web, mediawiki-main-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-web, mw-web.eqiad.main, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 5daf2ba1230e762df91c8b91d1410455ec7728c97b18c19b7d60a722a2bbbabd

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: bfcb78c0fff835977559f96bc26dfe6dfc845b99674c3aba1a05d4c1da2d503d
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-web-migration in production ===
Comparing release=migration, chart=wmf-stable/mediawiki, namespace=mw-web
mw-web, mediawiki-migration, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-web, mediawiki-migration-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-web, mw-web.eqiad.migration, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 650a3ee70dec691ce505048c2ca81e7accc67dae073dff1b1c6d219261974487

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 9c8e8dbd358b600b1617a8dde80b35103f7232b0492a48f791793f4e3c26006f
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-web-next in production ===
Comparing release=next, chart=wmf-stable/mediawiki, namespace=mw-web
mw-web, mediawiki-next, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-web, mediawiki-next-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-web, mw-web.eqiad.next, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: b0e2058c1f34b7b22453767ac5c766accd68ae80cbb65a5e35046d8c48711893

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 598218831b90ea1b9831d10febb3c1dc22516adf8b8fe423869abc8c16658d49
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-wikifunctions-group0 in production ===
Comparing release=group0, chart=wmf-stable/mediawiki, namespace=mw-wikifunctions
mw-wikifunctions, mediawiki-group0, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-wikifunctions, mediawiki-group0-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-wikifunctions, mw-wikifunctions.eqiad.group0, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 845f0d82e0845fbf631a7ca344b1d552023285a297c077bd3baf5e88451ea8a9

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 764577aac83b0610870f0341b4bfa8cad2caef27b8e323ff0c87683dec50fc23
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-wikifunctions-group1 in production ===
Comparing release=group1, chart=wmf-stable/mediawiki, namespace=mw-wikifunctions
mw-wikifunctions, mediawiki-group1, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-wikifunctions, mediawiki-group1-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-wikifunctions, mw-wikifunctions.eqiad.group1, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 56da13f9b514e2ac8f0340bdfc3bc79f3b08a37f21a16dcd93417592c6af7f47

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: bdac9f1a0206d04a444a4d75ce16aedee3f1a77f94c7d01ced2209a8282733b2
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...


=== Diff for eqiad/mw-wikifunctions-group2 in production ===
Comparing release=group2, chart=wmf-stable/mediawiki, namespace=mw-wikifunctions
mw-wikifunctions, mediawiki-group2, NetworkPolicy (networking.k8s.io) has changed:
...
          port: 9092
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
+           cidr: 10.64.169.4/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 10.64.181.5/32
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
            cidr: 2620:0:861:102:10:64:16:176/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
...
        - protocol: TCP
          port: 9093
      - to:
        - ipBlock:
            cidr: 2620:0:861:10e:10:64:135:13/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:119:10:64:169:4/128
+       ports:
+       - protocol: TCP
+         port: 9092
+       - protocol: TCP
+         port: 9093
+     - to:
+       - ipBlock:
+           cidr: 2620:0:861:13b:10:64:181:5/128
        ports:
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
...
mw-wikifunctions, mediawiki-group2-rsyslog-config, ConfigMap (v1) has changed:
...
      # These logs are received via udp
      template(name="access-log-topic" type="string" string="mediawiki.httpd.accesslog")
      ruleset(name="accesslog_to_kafka") {
        action(type="mmjsonparse" name="mmjsonparse_accesslog" cookie="")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="access-log-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="ecs_1110_k8s"
...
      # Slowlogs are sent to a specialized topic, mediawiki.php-fpm.slowlog
      template(name="php-slowlog-topic" type="string" string="mediawiki.php-fpm.slowlog")
      ruleset(name="slowlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-slowlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-slowlog-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="slowlog"
...
      # PHP-FPM logs
      template(name="php-fpm-topic" type="string" string="rsyslog-%syslogseverity-text::lowercase%")
      ruleset(name="errorlog_to_kafka") {
        action(type="mmnormalize" rulebase="/etc/rsyslog.d/php-errorlog.rb")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="php-fpm-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...

      # Use a separate (in memory) queue to limit message processing to this ruleset only.
      ruleset(name="udp_localhost_to_kafka" queue.type="LinkedList") {
        action(type="mmjsonparse" name="mmjsonparse_udp_localhost")
        action(type="omkafka"
-               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093"]
+               broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093","kafka-logging1006.eqiad.wmnet:9093","kafka-logging1007.eqiad.wmnet:9093","kafka-logging1008.eqiad.wmnet:9093"]
                topic="mw-kafka-topic"
                dynatopic="on"
                dynatopic.cachesize="1000"
                partitions.auto="on"
                template="syslog_cee"
...
mw-wikifunctions, mw-wikifunctions.eqiad.group2, Deployment (apps) has changed:
...
          checksum/sites: b16477bd9fc9f62612c7494631ef64896cf35e673fce3a853e1e66dd287ec3f7
          # we need to restart the deployment when we add/remove external
          # env variables, not just the ones we explicitly declare here.
          checksum/environment: 22225e7b3acc07474f6d989066bd021e4996b34cac99745f5032fa6b5dfb1076

-         checksum/rsyslog: 2faa5e34136556f402a15c757a37d82a027701c9f0804341184f66c5bb6a5d36
+         checksum/rsyslog: 6b53bab1c3a1a16ccefce69759ea01e9ebe20d0802e97433ddcbdc3f2f408c0a

          prometheus.io/scrape_by_name: "true"
          checksum/tls-config: 67f9ea7b34ed08c0bbbe853d55c19f9fc76eb4b2366fade0a87c0fb6713e5e63
          envoyproxy.io/scrape: "true"
          envoyproxy.io/port: "9361"
...







Note: Diffs are relative to the current helm charts and helmfile values. These may become outdated if new changes are merged.
Continue with the deployment? [y/N]: y
08:04:18 Started scap sync-world: T432444 - Provision kafka-logging100[6-8]
08:04:19 Started cache_git_info
08:04:21 Finished cache_git_info (duration: 00m 01s)
08:04:21 Started l10n-update
08:04:22 Updating ExtensionMessages-1.47.0-wmf.20.php
08:04:27 Updating LocalisationCache for 1.47.0-wmf.20 using 22 thread(s)
08:04:27 Running rebuildLocalisationCache.php (cdb)
08:04:30 0 languages rebuilt out of 549
08:04:30 Use --force to rebuild the caches which are still fresh.
08:04:31 Generating JSON versions and md5 files (as www-data)
08:04:36 Updating ExtensionMessages-1.47.0-wmf.21.php
08:04:41 Updating LocalisationCache for 1.47.0-wmf.21 using 22 thread(s)
08:04:41 Running rebuildLocalisationCache.php (cdb)
08:04:43 0 languages rebuilt out of 549
08:04:43 Use --force to rebuild the caches which are still fresh.
08:04:45 Generating JSON versions and md5 files (as www-data)
08:04:48 Finished l10n-update (duration: 00m 27s)
08:04:48 Checking for new runtime errors locally
08:04:52 Started sync-masters
08:04:53 ['/usr/bin/scap', 'pull-master', 'deploy1003.eqiad.wmnet'] (ran as mwdeploy@deploy2002.codfw.wmnet) returned [1]: Traceback (most recent call last):
  File "/usr/bin/scap", line 3, in <module>
    from scap import cli
ImportError: cannot import name 'cli' from 'scap' (unknown location)

08:05:14 sync-masters: 100% (in-flight: 0; ok: 1; fail: 1; left: 0)
08:05:14 1 masters had sync errors
08:05:14 Finished sync-masters (duration: 00m 21s)
08:05:14 Started Collecting rollback information
08:05:19 Finished Collecting rollback information (duration: 00m 05s)
08:05:19 Started sync-testservers-k8s
08:07:57 K8s deployment progress: 100% (ok: 16; fail: 0; left: 0)
08:07:57 Finished sync-testservers-k8s (duration: 02m 37s)
08:07:57 Started testserver-check-commands
08:07:57 Executing check 'httpbb /srv/deployment/httpbb-tests/appserver/* --hosts=mwdebug.discovery.wmnet --https_port=4444 --retry_on_timeout'
08:07:57 Executing check 'httpbb /srv/deployment/httpbb-tests/appserver/* --hosts=mwdebug-next.discovery.wmnet --https_port=4453 --retry_on_timeout'
08:07:57 Executing check 'httpbb /srv/deployment/httpbb-tests/pretrain/* --hosts=mw-pretrain.discovery.wmnet --https_port=30443 --request_header 'x-wikimedia-debug: backend=k8s-mw-pretrain' --retry_on_timeout'
08:08:55 Finished testserver-check-commands (duration: 00m 57s)
08:08:55 Started sync-canaries-k8s
08:09:41 K8s deployment progress: 100% (ok: 64; fail: 0; left: 0)
08:09:41 Finished sync-canaries-k8s (duration: 00m 46s)
08:09:41 Waiting 20 seconds for canary traffic...
08:10:01 Logstash checker counted 0 error(s) in the last 20 seconds. OK.
08:10:01 Logstash checker counted 0 error(s) in the last 20 seconds. OK.
08:10:01 Started sync-prod-k8s
08:13:25 K8s deployment progress:  99% (ok: 2370; fail: 0; left: 20)
08:13:25 K8s deployment continues in the background.
08:13:25 Finished sync-prod-k8s (duration: 03m 23s)
08:13:25 Started production-check-commands
08:13:25 Executing check 'httpbb /srv/deployment/httpbb-tests/pretrain/* --hosts=mw-pretrain.discovery.wmnet --https_port=30443 --retry_on_timeout'
08:13:31 Finished production-check-commands (duration: 00m 06s)
08:13:31 Waiting 20 seconds for production traffic...
08:13:52 Logstash checker counted 27 error(s) in the last 20 seconds. OK.
08:13:52 Logstash checker counted 29 error(s) in the last 20 seconds. OK.
08:13:52 Running purgeMessageBlobStore.php
08:13:53 Finished scap sync-world: T432444 - Provision kafka-logging100[6-8] (duration: 12m 52s)
tappof mentioned this in T432444: Provision kafka-logging100[6-8] and kafka-logging200[6-8].Wed, Sep 23, 8:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits