Per T331283 and related tickets, Data Engineering (in consultation with Event Platform and Service Ops) has settled on Zookeeper to implement Flink HA, as described at Flink's website .
AC:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | gmodena | T307959 [Event Platform] Design and Implement realtime enrichment pipeline for MW page change with content | |||
| Resolved | Ottomata | T325303 Deploy mediawiki-page-content-change-enrichment to wikikube k8s | |||
| Open | None | T331283 [Event Platform] [NEEDS GROOMING] Store Flink HA metadata in Zookeeper | |||
| Resolved | bking | T326409 Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model | |||
| Resolved | bking | T341792 Provision Zookeeper Cluster for storing Flink HA data | |||
| Resolved | bking | T341705 eqiad: 3 VMs requested for Zookeeper | |||
| Resolved | None | T343715 codfw: 3 VMs requested for Zookeeper |
Change 938000 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] zookeeper: prepare for new zk cluster
Mentioned in SAL (#wikimedia-operations) [2023-07-13T20:59:43Z] <inflatador> bking@cumin1001 'disable puppet on hosts using zookeeper class T341792'
Change 938000 merged by Bking:
[operations/puppet@production] zookeeper: prepare for new zk cluster
Change 940243 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] flink-zk: Initiate new flink::zookeeper role
Change 940243 merged by Bking:
[operations/puppet@production] flink-zk: Initiate new flink::zookeeper role
Change 942428 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] flink-zk: use correct variable for firewall defs
Change 942428 merged by Bking:
[operations/puppet@production] flink-zk: use correct variable for firewall defs
The cluster is up and all nodes appear to have joined correctly; my compliments to whoever wrote the puppet code.
The next step is to get metrics from the flink-zk cluster into the Zookeeper dashboard.
I've confirmed the zookeeper exporter is up and listening on 12181. I'm going to check the software firewall rules next.
Change 942457 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] flink-zk: allow analytics network
Change 942494 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] flink-zk: Enable prometheus scrapes
Change 942457 abandoned by Bking:
[operations/puppet@production] flink-zk: allow analytics network
Reason:
Confirmed that firewall is not the issue.
Change 942494 merged by Bking:
[operations/puppet@production] flink-zk: Enable prometheus scrapes
Change 945640 had a related patch set uploaded (by Bking; author: Bking):
[operations/alerts@master] prometheus-analytics: create alerts for new ZK cluster
Per today's Data Platform meeting, Ben provided an example of existing zookeeper alerts , we should be able to use this for inspiration.
It might be the case that the alerts are already in place, since we have the metrics (for the eqiad cluster) in Grafana now.
You can also see that flink-eqiad appears here.
What about stopping a couple of the zookeeper servers and seeing if we get an alert raised. Might want to announce the test first :-)
I suppose that it might also depend a bit on how for we should go with: T342578: Ensure Data Platform SREs have a contact group in puppet/alerting
Perhaps we want to have this zookeeper cluster notify our ourselves, rather than the core SRE team.
Oh! I see that you've already done this in https://gerrit.wikimedia.org/r/c/operations/alerts/+/945640 but we haven't onboarded team-data-platform-sre to Alertmanager yet, have we?
Oh! I see that you've already done this in https://gerrit.wikimedia.org/r/c/operations/alerts/+/945640 but we haven't onboarded team-data-platform-sre to Alertmanager yet, have we?
Good catch, I updated the AC of that ticket and will work on it as time permits.
Change 948615 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] flink-zk: add cluster info for codfw
Change 948615 merged by Bking:
[operations/puppet@production] flink-zk: add cluster info for codfw
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk2001.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk2001.codfw.wmnet with OS bookworm executed with errors:
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk2002.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk2002.codfw.wmnet with OS bookworm executed with errors:
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk2002.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk2002.codfw.wmnet with OS bookworm executed with errors:
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk2003.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk2003.codfw.wmnet with OS bookworm executed with errors:
cookbooks.sre.hosts.decommission executed by bking@cumin1001 for hosts: flink-zk2002.codfw.wmnet
Mentioned in SAL (#wikimedia-operations) [2023-08-18T17:18:10Z] <inflatador> bking@cumin1001 temporarily enabling alerts for flink-zk hosts to see if they work T341792
Mentioned in SAL (#wikimedia-operations) [2023-08-18T17:25:06Z] <inflatador> bking@ganeti1024 shutting off flink-zk1001 to check alerting T341792
To check alerting, I removed suppressions and shut off flink-zk1001 via the ganeti master. I saw flink-zk1001 turn red in Icinga, failing all of its checks, but I didn't see any alerts appear in AlertManager.
That suggests to me we're missing some important config in Puppet and/or the alerts repo. Will re-read Wikidata docs on alerting today and escalate Monday if I can't make progress.
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk2001.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk2001.codfw.wmnet with OS bookworm executed with errors:
cookbooks.sre.hosts.decommission executed by bking@cumin1001 for hosts: flink-zk2001.codfw.wmnet
cookbooks.sre.hosts.decommission executed by bking@cumin1001 for hosts: flink-zk2003.codfw.wmnet
Change 954134 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] flink-zk: Move codfw hosts back to insetup
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk2001.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk2001.codfw.wmnet with OS bookworm executed with errors:
Change 954134 merged by Bking:
[operations/puppet@production] flink-zk: Move codfw hosts back to insetup
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk2001.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk2001.codfw.wmnet with OS bookworm executed with errors:
cookbooks.sre.hosts.decommission executed by bking@cumin1001 for hosts: flink-zk2001.codfw.wmnet
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk2001.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk2001.codfw.wmnet with OS bookworm executed with errors:
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk2002.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk2002.codfw.wmnet with OS bookworm completed:
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk2003.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk2003.codfw.wmnet with OS bookworm completed:
OK, let's look at this again together, because I was under the impression that this check should fail with one zookeeper server down:
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/refs/heads/master/team-sre/zookeeper.yaml#19
Change 958991 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Add the analytics and search-pltform teams to flink zk contacts
Change 958991 merged by Btullis:
[operations/puppet@production] Add the analytics and search-platform teams to flink zk contacts
The patch above ensures that Data Platform SREs will be alerted if there's a problem with the flink-zk cluster. Thus, I'm happy to close this one out. Thanks to everyone who contributed to this effort.
Change was reverted due to nonexistent search-platform icinga contactgroup leading to a broken icinga config: https://gerrit.wikimedia.org/r/c/operations/puppet/+/959015
Change 962660 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] flink: Add correct contactgroups
Change 962660 merged by Bking:
[operations/puppet@production] flink: Add correct contactgroups
Change 945640 abandoned by Bking:
[operations/alerts@master] prometheus-analytics: create alerts for new ZK cluster
Reason:
WIP patch is blocking this repo's gitlab migration