We've several possible failure modes for kafkatee, :
# local filtering or log handling bug
# unanticipated kafka-jumbo cluster changes {T254257}
# packet loss due to local bottleneck {T239564}
# packet loss due to local firewall (locally logged)
# packet loss due to network issue
# incomplete topic data due to bug/issue in the kafka cluster {T73056}
# incomplete data due tue eventlogging issue
We already have some monitoring in place:
* check_impression_logs - nagios check that is integrated with the log rotation script, alerts if log rotation stop running or sees gaps in data, this is very limited because there's no way to adjust alert thresholds to suit to campaign activity
* notification on packets dropped by the kernel
We have some tasks defined for improvements:
* {T239564}
Other ideas:
* alert on mismatch between configured kafka broker list and fetched list (note this is an unresolved issue in production too, see T213561, and their proposed approach of a discovery URL doesn't help us as long as we use IP-based firewalls)
* scrape kafkatee statistics files for alert-worthy stats [possibly same as T239564]
* alert on changes to relevant production puppet classes or hiera configuration