Change Details

While looking into T321547 I realized we're not logging which silences get POST'ed to the AM API. The spicerack silences are logged by other means, however we should have a generic mechanism to audit/log all silences (e.g. those set by humans via alerts.w.o). To this end, the simplest approach seems to be to instruct apache to log POST bodies. Somewhat surprisingly to me, this is easier said than done. There are three approaches that emerged after a chat with @elukey (in no particular order): * `mod_dumpio` https://httpd.apache.org/docs/current/mod/mod_dumpio.html * `mod_security` e.g. https://serverfault.com/questions/728575/what-rule-can-i-use-in-modsecurity-to-log-post-payload-for-a-specific-site * `mod_ext_filter` https://httpd.apache.org/docs/current/mod/mod_ext_filter.html Each have their pros and cons, to be evaluated here (and then implemented)

While looking into T321547 I realized we're not logging which silences get POST'ed to the AM API. The spicerack silences are logged by other means, however we should have a generic mechanism to audit/log all silences (e.g. those set by humans via alerts.w.o). To this end, the simplest approach seems to be to instruct apache to log POST bodies. Somewhat surprisingly to me, this is easier said than done. There are three approaches that emerged after a chat with @elukey (in no particular order): * `mod_dumpio` https://httpd.apache.org/docs/current/mod/mod_dumpio.html * `mod_security` e.g. https://serverfault.com/questions/728575/what-rule-can-i-use-in-modsecurity-to-log-post-payload-for-a-specific-site * `mod_ext_filter` https://httpd.apache.org/docs/current/mod/mod_ext_filter.html Each have their pros and cons, to be evaluated here (and then implemented). Next steps are to make sure AM clients go through apache: * `amtool` via `/etc/prometheus/amtool.yml` * `karma` via `/etc/karma.yml`

While looking into T321547 I realized we're not logging which silences get POST'ed to the AM API. The spicerack silences are logged by other means, however we should have a generic mechanism to audit/log all silences (e.g. those set by humans via alerts.w.o). To this end, the simplest approach seems to be to instruct apache to log POST bodies. Somewhat surprisingly to me, this is easier said than done. There are three approaches that emerged after a chat with @elukey (in no particular order): * `mod_dumpio` https://httpd.apache.org/docs/current/mod/mod_dumpio.html * `mod_security` e.g. https://serverfault.com/questions/728575/what-rule-can-i-use-in-modsecurity-to-log-post-payload-for-a-specific-site * `mod_ext_filter` https://httpd.apache.org/docs/current/mod/mod_ext_filter.html Each have their pros and cons, to be evaluated here (and then implemented). Next steps are to make sure AM clients go through apache: * `amtool` via `/etc/prometheus/amtool.yml` * `karma` via `/etc/karma.yml`