1.2.4 is out and is bringing a useful feature: Native Prometheus support
If it fits this doc: https://fastnetmon.com/monitoring-fastnetmon-via-prometheus/ using it might help us monitor its health better as well as more efficiently monitor DDoS attacks.
We're moving these to Bookworm which has 1.2.4 natively:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | MoritzMuehlenhoff | T330884 Upgrade Fastnetmon to 1.2.4 |
I tried to backport FNM 1.2.4, it does some tricky things with Boost and so far I couldn't force cmake to accept Bullseye's Boost libs. Given that Bookworm is close (unstable has 1.2.4, testing 1.2.3 not sure whether it will still migrate), I'm inclined to move forward with Bookworm netflow VMs (even if 1.2.4 isn't accepted into Bookworm any more, the backport should be simple on Bookworm).
It'll take a bit to sort out the setup, my proposal would be to add an additional netflow* VM in one of the sites, get the stack ready and then reimage the netflow* VMs in the other sites.
Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm executed with errors:
Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm executed with errors:
Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm executed with errors:
Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm executed with errors:
Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm executed with errors:
Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm executed with errors:
Cookbook cookbooks.sre.ganeti.reimage was started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm
Cookbook cookbooks.sre.ganeti.reimage started by jmm@cumin2002 for host netflow2003.codfw.wmnet with OS bookworm executed with errors:
Mentioned in SAL (#wikimedia-operations) [2023-05-19T08:09:33Z] <moritzm> copy samplicator from bullseye-wikimedia to bookworm-wikimedia T330884
@ayounsi There's now netflow2003 running Bookworm with FNM 1.2.4. If that works fine, we can reimage the other netflow* VMs in-place once Bookworm is stable.
I copied over samplicator from bullseye-wikimedia to bookworm-wikimedia (the only dependency is glibc itself), but there wasn't a source package on apt.wikimedia.org, do you by chance still have it on your laptop or the build host so that we can import it?
Change 921375 had a related patch set uploaded (by Ayounsi; author: Ayounsi):
[operations/homer/public@master] codfw: use new netflow server
Change 921375 merged by jenkins-bot:
[operations/homer/public@master] codfw: use new netflow server
Change 921384 had a related patch set uploaded (by Ayounsi; author: Ayounsi):
[operations/puppet@production] Kafka: add netflow2003 to the allowed sources
Change 921384 merged by Ayounsi:
[operations/puppet@production] Kafka: add netflow2003 to the allowed sources
I pointed the DFW routers to the new instance and both fastnetmon and nfacctd are working fine!
Example Prometheus metrics (After enabling it briefly) are visible on {P48400}
I copied over samplicator from bullseye-wikimedia to bookworm-wikimedia (the only dependency is glibc itself), but there wasn't a source package on apt.wikimedia.org, do you by chance still have it on your laptop or the build host so that we can import it?
Unfortunately, no, if it can help I think I got it from https://github.com/sleinen/samplicator
Some people tried to generate deb files as well, see https://github.com/hfeeki/samplicator-debian
Change 921390 had a related patch set uploaded (by Ayounsi; author: Ayounsi):
[operations/puppet@production] Fastnetmon: enable Prometheus exporter
Change 921394 had a related patch set uploaded (by Ayounsi; author: Ayounsi):
[operations/puppet@production] Prometheus: fetch FastNetMon metrics
Ok, the current package will work fine for now (given that libc6 is the only depdendency), I'm filing a low prio task to eventually also import a source package (we might also simply upload it to Debian at some point)
Bookworm is released, so these are good to go. Is it acceptable to reimage these in place (meaning a loss of newflow data for like an hour each) or do we need the create-VMs-and-failover dance?
Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host netflow4002.ulsfo.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host netflow4002.ulsfo.wmnet with OS bookworm completed:
Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host netflow5002.eqsin.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host netflow5002.eqsin.wmnet with OS bookworm completed:
Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host netflow6001.drmrs.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host netflow6001.drmrs.wmnet with OS bookworm completed:
Change 929340 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove netflow2002 from Kafka config
Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host netflow3002.esams.wmnet with OS bookworm
Change 929340 merged by Muehlenhoff:
[operations/puppet@production] Remove netflow2002 from Kafka config
Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host netflow3002.esams.wmnet with OS bookworm completed:
Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host netflow1002.eqiad.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host netflow1002.eqiad.wmnet with OS bookworm completed:
Change 921390 merged by Ayounsi:
[operations/puppet@production] Fastnetmon: enable Prometheus exporter
cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: netflow2002.codfw.wmnet
Change 921394 merged by Ayounsi:
[operations/puppet@production] Prometheus: fetch FastNetMon metrics
Mentioned in SAL (#wikimedia-operations) [2023-07-10T11:14:32Z] <moritzm> remove unused VM netflow6002 T330884