As per title, all hosts running this role should be on Buster. Ideally by FQ2 FY20-21.
root@cumin1001:~# cumin 'P{O:netmon} and not P{F:lsbdistcodename = buster}'
2 hosts will be targeted:
netmon[1002,2001].wikimedia.org
DRY-RUN mode enabled, abortingCurrent plan:
- Lower TTL for librenms/smokeping DNS records https://gerrit.wikimedia.org/r/c/operations/dns/+/615417
- Reimage netmon2001 with Buster, verify basic puppet runs are working as intended
- Flip netmon2001 to be the active server
- Sync RRDs in /srv/librenms/rrd/
- Sync rancid in /var/lib/rancid/core and /var/lib/rancid/GIT
- Set netmon2001 as netmon_server in puppet and netmon1002 as netmon_server_failover run puppet on both netmon1002 and netmon2001 https://gerrit.wikimedia.org/r/c/operations/puppet/+/616710
- Flip DNS for librenms and smokeping to netmon2001 https://gerrit.wikimedia.org/r/c/operations/dns/+/616709
- Verify librenms is working as intended: users can login, data is collected, alerts can be sent, etc
- Verify rancid/smokeping are working as intended
- Verify snmp-exporter is working as intended
- Reimage netmon1002 with Buster
- Flip back from netmon2001 to netmon1002
- Restore smokeping data in /var/lib/smokeping from backups
- Sync RRDs in /srv/librenms/rrd/
- Sync rancid in /var/lib/rancid/core and /var/lib/rancid/GIT
- Set netmon1002 as netmon_server in puppet and netmon2001 as netmon_server_failover run puppet on both netmon1002 and netmon2001 https://gerrit.wikimedia.org/r/c/operations/puppet/+/617395
- Flip DNS for librenms and smokeping to netmon1002 https://gerrit.wikimedia.org/r/c/operations/dns/+/617393
Issues identified:
- Polling time for eqiad devices increased significantly due to the added latency. For the most populated rows (eqiad B and D) this means that occasionally poll times are >5 min, resulting in alerts and potentially missed data
- librenms web ui got significantly slower (from Europe at least) in part because of the added latency to reach codfw, in part because the database is still in eqiad