While looking into T321547 I realized we're not logging which silences get POST'ed to the AM API. The spicerack silences are logged by other means, however we should have a generic mechanism to audit/log all silences (e.g. those set by humans via alerts.w.o).
To this end, the simplest approach seems to be to instruct apache to log POST bodies. Somewhat surprisingly to me, this is easier said than done. There are three approaches that emerged after a chat with @elukey (in no particular order):
- mod_dumpio https://httpd.apache.org/docs/current/mod/mod_dumpio.html
- mod_security e.g. https://serverfault.com/questions/728575/what-rule-can-i-use-in-modsecurity-to-log-post-payload-for-a-specific-site
- mod_ext_filter https://httpd.apache.org/docs/current/mod/mod_ext_filter.html
Each have their pros and cons, to be evaluated here (and then implemented).
Next steps are to make sure AM clients go through apache:
- amtool via /etc/prometheus/amtool.yml
- karma via /etc/karma.yml