Summary of a conversation that ori, joe, and I had on IRC today:

• You get some of this "for free" once the appservers are on k8s -- you can add labels to your pods that will be automatically propagated to logstash/prometheus
• However, it would be valuable to have a framework like this beyond just the appservers or k8s services
  • For instance, Traffic has done a lot of that kind of experimentation on cp nodes with ad-hoc mechanisms in the past, same for some other teams
• Any Puppet+Prometheus plumbing should be reusable, at least
  • in prometheus you can have those same puppet facts exported by node-exporter after having puppet generate a textfile for it, and then, you can join metrics together at query time
• Logstash might be more difficult that Prometheus (although I don't know for sure, maybe there's an easy mechanism with a filter script)
  • Perhaps those tags could be injected via rsyslog (as configured via puppet)?

Thank you all involved so far! Agreed sth like this would be useful for the non-k8s parts of the infra too. I'm not exactly sure about the logstash bits and how we'd attach tags/attributes (cc @colewhite)

Creating a rsyslog template or amending syslog_cee seems the path of least resistance to injecting a Puppet-defined value into the log stream at the host level.

In the case of amending the syslog_cee template, we'd need to define a default value because this would affect all logs that use that template.

For reference:

...
constant(outname="my_label_name" value="my_custom_value" format="jsonf")
constant(value=", ")
...

Just pinging this task as OKR season is upon us and this might be a useful and fun thing to sneak in