Here's an update on my exploration thus far,

• we can change where k3s stores all of it's data including log data
  • we can modify our exiting configuration in /etc/rancher/k3s/config.yaml
    • this will wipe all data unless we stop the server, move the data, and restart it
  • if we want them to be on separate volumes, the guidance seems to be to move the logs manually and symlink [data-dir]/agent/containerd/logs to another path on another volume
• the kubernetes agent manages log rotation
  • we can configure it to have larger log files (default 10mb) and more log files (default 5)
  • this allows us to keep using kubectl get logs and we can also get earlier logs from the [data-dir]/agent/containerd/logs location
• we will still need sidecar containers to pull out logs from files
  • essentially each sidecar container is responsible for tail-ing one log file to stdout
  • this makes these logs get picked up by kubernets and writes them in the log directory on the host
  • it also allows you to get these logs through the kubectl command
    • e.g. if we had a sidecar container for patchdemo that tailed the apache access.log, patchdemo-access-log
      • we could either do kubectl logs patchdemo and the access log would show up in the pod logs
      • we can also get just those logs with kubectl logs patchdemo -c patchdemo-access-log to see just that log
      • we can also go to [data-dir]/agent/containerd/logs/patchdemo/patchdemo-access-log to see all the logs including rotated logs

1. add 40GiB persistent volume, catalyst-k3s-data
2. attach catalyst-k3s-data to k3s instance (/dev/sdb)
3. add 40GiB persistent volume, catalyst-k3s-logs
4. attach catalyst-k3s-logs to k3s instance (/dev/sdc)
5. mkdir /mnt/k3s-data and /mnt/k3s-logs
6. make the filesystems
   1. sudo mkfs.ext4 /dev/sdb
   2. sudo mkfs.ext4 /dev/sdc
7. add to fstab
   1. UUID=913cb98f-a4fe-4320-803d-8d4e4aa4a0ec /mnt/k3s-data ext4 defaults 0 2
   2. UUID=36f911a5-db56-4d92-9880-17bb8d2988cc /mnt/k3s-logs ext4 defaults 0 2
8. stop the k3s server
   1. sudo systemctl stop k3s
9. move the data dir
   1. sudo mv /var/lib/rancher/k3s /mnt/k3s-data/
10. symlink the data dir
    1. sudo ln -s /mnt/k3s-data/k3s /var/lib/rancher/k3s
11. move the logs dir
    1. sudo mv /var/log/pods /mnt/k3s-logs/pods
12. symlink the logs dir
    1. sudo ln -s /mnt/k3s-logs/pods /var/log/pods
13. restart k3s
    1. sudo systemctl start k3s

kindrobot opened https://gitlab.wikimedia.org/repos/qte/catalyst/patchdemo/-/merge_requests/16

Adds sidecar containers for log collection

kindrobot merged https://gitlab.wikimedia.org/repos/qte/catalyst/patchdemo/-/merge_requests/16

Adds sidecar containers for log collection