Change Details

When working on the Anycast setup for the ceph swift service I noticed that we get no notification when anycast-healthchecker's check fails and it withdraws a BGP route. I know in most scenarios we have other checks on the service it is looking at, and would alert, but I think it might be worth adding some alerting on this, so things don't look "normal" if the host is not announcing anything for an odd reason? One way to check for it would be to trigger birdc and count the number of routes announced? Something like this might work: ``` cmooney@cephosd1003:~$ sudo birdc show route all export bgp1 | grep -c unicast 0 ``` ``` cmooney@cephosd1003:~$ sudo birdc show route all export bgp1 | grep -c unicast 1 ``` We could also maybe look at using someting like this prometheus [[ https://github.com/czerwonk/bird_exporter | bird exporter ]], which should give us a count of the prefixes exported for each neighbor.

When working on the Anycast setup for the ceph swift service I noticed that we get no notification when anycast-healthchecker's check fails and it withdraws a BGP route. I know in most scenarios we have other checks on the service it is looking at, and would alert, but I think it might be worth adding some alerting on this, so things don't look "normal" if the host is not announcing anything for an odd reason? One way to capture it might be to monitor the system logs and look for when the anycast-healthchecker marks something as DOWN? Alternately we could count how many BGP rotues are announced, using something like this: ``` cmooney@cephosd1003:~$ sudo birdc show route all export bgp1 | grep -c unicast 0 ``` ``` cmooney@cephosd1003:~$ sudo birdc show route all export bgp1 | grep -c unicast 1 ``` We could also maybe look at using someting like this prometheus [[ https://github.com/czerwonk/bird_exporter | bird exporter ]], which should give us a count of the prefixes exported for each neighbor.

When working on the Anycast setup for the ceph swift service I noticed that we get no notification when anycast-healthchecker's check fails and it withdraws a BGP route. I know in most scenarios we have other checks on the service it is looking at, and would alert, but I think it might be worth adding some alerting on this, so things don't look "normal" if the host is not announcing anything for an odd reason? One way to check for it would be to trigger birdc and count the number of routes announced? Something like this might workapture it might be to monitor the system logs and look for when the anycast-healthchecker marks something as DOWN? Alternately we could count how many BGP rotues are announced, using something like this: ``` cmooney@cephosd1003:~$ sudo birdc show route all export bgp1 | grep -c unicast 0 ``` ``` cmooney@cephosd1003:~$ sudo birdc show route all export bgp1 | grep -c unicast 1 ``` We could also maybe look at using someting like this prometheus [[ https://github.com/czerwonk/bird_exporter | bird exporter ]], which should give us a count of the prefixes exported for each neighbor.