We need to allow the $DSE_KUBEPODS_NETWORKS ferm srange in the firewall rules of a variety of services, such as mariadb1001, druid, wikireplicas, etc.
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
Related Changes in GitLab:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Add missing deb/pip packages to the Superset image | repos/data-engineering/superset!25 | brouberol | add-missing-packages | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | brouberol | T353782 Decommission an-tool1010 | |||
| Resolved | brouberol | T347710 Migrate the Analytics Superset instances to our DSE Kubernetes cluster | |||
| Resolved | brouberol | T356623 Ensure necessary firewall rules are open between the DSE worker nodes and external services |
Event Timeline
Comment Actions
Change 997337 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Enable dse k8s workers -> an-mariadb1001:3306 traffic
Comment Actions
Change 997337 merged by Brouberol:
[operations/puppet@production] Enable dse k8s workers -> an-mariadb1001:3306 traffic
Comment Actions
brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/25
Add missing deb/pip packages to the Superset image
Comment Actions
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/25
Add missing deb/pip packages to the Superset image
Comment Actions
Superset -> dbstore1009: ✅
brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.135.22 3320 Trying 10.64.135.22... Connected to 10.64.135.22. Escape character is '^]'. Y 5.5.5-10.6.16-MariaDB��yO\c\"4|�:F4oR/3R"OKgmysql_native_password^CConnection closed by foreign host. brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.135.22 3350 Trying 10.64.135.22... Connected to 10.64.135.22. Escape character is '^]'. Y 5.5.5-10.6.16-MariaDBcG;0!QW?w�Y9M[`OLolO'smysql_native_password^CConnection closed by foreign host. brouberol@dse-k8s-worker1001:~$
Superset -> Kerberos: ✅
brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.0.112 88 Trying 10.64.0.112... Connected to 10.64.0.112. Escape character is '^]'. brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 2620:0:861:101:10:64:0:112 88 Trying 2620:0:861:101:10:64:0:112... Connected to 2620:0:861:101:10:64:0:112. Escape character is '^]'.
Superset -> an-mariadb1001: ✅
brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.138.8 3306 Trying 10.64.138.8... Connected to 10.64.138.8. Escape character is '^]'. ] 5.5.5-10.4.28-MariaDB-log�k)SFGp|Cu�E:0!R\Y*c4x"mysql_native_password brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 2620:0:861:100:10:64:138:8 3306 Trying 2620:0:861:100:10:64:138:8... Connected to 2620:0:861:100:10:64:138:8. Escape character is '^]'. ] 5.5.5-10.4.28-MariaDB-log�kV8U#K3]Z�+gIr&a7/%kkXmysql_native_password
Superset -> an-coord1001 ❌
brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.21.104 8281 Trying 10.64.21.104... ^C brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 2620:0:861:105:10:64:21:104 8281 Trying 2620:0:861:105:10:64:21:104...
Superset -> public druid ✅
brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.16.171 8082 Trying 10.64.16.171... Connected to 10.64.16.171. Escape character is '^]'. brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 2620:0:861:102:10:64:16:171 8082 Trying 2620:0:861:102:10:64:16:171... Connected to 2620:0:861:102:10:64:16:171. Escape character is '^]'.
Superset -> analytics druid ❌
brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.5.17 8082 Trying 10.64.5.17... brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 2620:0:861:104:10:64:5:17 8082 Trying 2620:0:861:104:10:64:5:17...
Superset -> staging-db-analytics ✅
brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.135.22 3350 Trying 10.64.135.22... Connected to 10.64.135.22. Escape character is '^]'. Y 5.5.5-10.6.16-MariaDBsrg-IiUNg�:+gGmm_q=wdnmysql_native_password.
Superset -> Hadoop master ❌
brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.36.15 8020 Trying 10.64.36.15... brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 2620:0:861:106:10:64:36:15 8020 Trying 2620:0:861:106:10:64:36:15...
Superset -> Hadoop worker ❌
brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.21.121 50010 Trying 10.64.21.121... ^C brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.21.121 50020 Trying 10.64.21.121... ^C brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 2620:0:861:105:10:64:21:121 50010 Trying 2620:0:861:105:10:64:21:121... ^C brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 2620:0:861:105:10:64:21:121 50020 Trying 2620:0:861:105:10:64:21:121...
Comment Actions
Change 997792 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Allow pods in the dse k8s cluster to reach an-coord1001
Comment Actions
Change 997793 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Allow pods in the dse k8s cluster to reach an-druid
Comment Actions
Change 997794 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Allow pods in the dse k8s cluster to reach public-druid
Comment Actions
Hadoop masters and workers are now working (they were missing a network policy)
brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.21.121 50010 Trying 10.64.21.121... Connected to 10.64.21.121. Escape character is '^]'. brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.21.121 50020 Trying 10.64.21.121... Connected to 10.64.21.121. Escape character is '^]'. brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 2091818 --net telnet 10.64.36.15 8020 Trying 10.64.36.15... Connected to 10.64.36.15. Escape character is '^]'.
Comment Actions
Change 997792 merged by Brouberol:
[operations/puppet@production] Allow pods in the dse k8s cluster to reach an-coord1001
Comment Actions
Change 997793 merged by Brouberol:
[operations/puppet@production] Allow pods in the dse k8s cluster to reach an-druid
Comment Actions
Superset -> analytics druid ✅
brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 3300144 --net telnet 10.64.5.17 8082 Trying 10.64.5.17... Connected to 10.64.5.17. Escape character is '^]'. ^]^CConnection closed by foreign host. brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 3300144 --net telnet 2620:0:861:104:10:64:5:17 8082 Trying 2620:0:861:104:10:64:5:17... Connected to 2620:0:861:104:10:64:5:17. Escape character is '^]'.
Comment Actions
Change 997794 merged by Brouberol:
[operations/puppet@production] Allow pods in the dse k8s cluster to reach public-druid
Comment Actions
Change 997846 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Enable dse k8s workers -> presto https traffic
Comment Actions
Change 997846 merged by Brouberol:
[operations/puppet@production] Enable dse k8s workers -> presto https traffic
Comment Actions
Superset -> an-coord ✅
brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 3300144 --net telnet 10.64.21.104 8281 Trying 10.64.21.104... Connected to 10.64.21.104. Escape character is '^]'. ^CConnection closed by foreign host. brouberol@dse-k8s-worker1001:~$ sudo nsenter --target 3300144 --net telnet 2620:0:861:105:10:64:21:104 8281 Trying 2620:0:861:105:10:64:21:104... Connected to 2620:0:861:105:10:64:21:104. Escape character is '^]'.