Currently, rsyslog's mmkubernetes data is in the kubernetes top-level key. For ECS messages, this top-level key is invalid and is dropped.
Create a logstash filter that moves the mmkubernetes data into the appropriate ECS fields. Define fields in the ECS schema as needed.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | colewhite | T292585 Evaluate long-term viability of feature gating on ECS major version | |||
| Resolved | BUG REPORT | colewhite | T292099 Logs from Toolhub's Django container not reaching logstash from Kubernetes | ||
| Open | None | T292881 Mutate mmkubernetes k8s fields into ECS fields | |||
| Resolved | colewhite | T294581 Upgrade ECS to 1.11.0 |
I think the general schema created by mmkubernetes is actually a valid starting point for expanding the schema. What is currently offered by ECS for the container and orchestrator namespaces doesn't cover our needs.
Reading the mmkubernetes documentation, here is the list of fields, the corresponding ECS equivalent, and in case of need, my proposed extension to the default orchestrator ECS schema.
| mmkubernetes | ecs equivalent |
| kubernetes.namespace_name | orchestrator.namespace |
| kubernetes.pod_name | orchestrator.resource.name |
| kubernetes.container_name | container.name |
| docker.id | container.id |
| kubernetes.master_url | orchestrator.cluster.url |
| kubernetes.namespace_id | - |
| kubernetes.pod_id | orchestrator.resource.id |
| kubernetes.creation_timestamp | - |
| kubernetes.host | host.hostname |
| kubernetes.labels | orchestrator.metadata.labels |
| kubernetes.annotations | orchestrator.metadata.annotations |
| kubernetes.namespace_labels | orchestrator.metadata.namespace_labels |
| kubernetes.namespace_annotations | orchestrator.metadata.namespace_annotations |
Substantially, I'm proposing to add a metadata extension that would include 4 objects:
And to add a orchestrator.resource.id field, as suggested by Janis.
I would second your proposal apart from what I think is a typo. kubernetes.namespace_name should be orchestrator.namespace. Also we might want to think about carrying kubernetes.pod_id around. With our current workload the field is not of much use (as kubernetes.pod_name is unique as well). But there are workloads that generate non-unique pod names (StatefulSet) where we would be unable to distinguish different incarnations without the ID.
Unfortunately orchestrator.resource.id is not defined in https://www.elastic.co/guide/en/ecs/current/ecs-orchestrator.html so we would need to add that to orchestrator.metadata I guess.
Setting priority to "high" as this would be a blocker for T288851 given we've adopted an ECS schema there.
Change 930597 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/software/ecs@master] backport orchestrator fields from ECS 8.8
Change #930597 merged by jenkins-bot:
[operations/software/ecs@master] add orchestrator and user.extra fields
Change #1032737 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: update ecs patch version to 7