[1]: for example the rsyslog solution mentioned in T97861#10957908 still requires access to the Kubernetes API to function properly

I'm not sure it does, it pulls the logs directly from the container log files in the filesystem, it does not talk to k8s. It does not depend (as is at least) on S3, or any other k8s pieces either (ex. if loki deployment can't restart pods because it can't pull images, or because a webhook messes it up, nodes unhealthy, or essentially any other k8s related issue).

Can we add another option, E?

Prioritize persisting the logs of the most critical k8s components (that is just piping to journal on the workers, not centralizing yet), and do the infra logs centralization later. That will give us a quick fix for now for the most critical logs (not awesome, but good enough imo), and learn from the loki setup for tools to take a better decision for the infra centralized setup later.

In T398285#10962459, @dcaro wrote:

[1]: for example the rsyslog solution mentioned in T97861#10957908 still requires access to the Kubernetes API to function properly

I'm not sure it does, it pulls the logs directly from the container log files in the filesystem, it does not talk to k8s. It does not depend (as is at least) on S3, or any other k8s pieces either (ex. if loki deployment can't restart pods because it can't pull images, or because a webhook messes it up, nodes unhealthy, or essentially any other k8s related issue).

The documentation I linked says that the rsyslog Kubernetes metadata module seems to require connection details to the Kubernetes API. Is that documentation wrong? Or is that metadata somehow not necessary?

Can we add another option, E?

Prioritize persisting the logs of the most critical k8s components (that is just piping to journal on the workers, not centralizing yet), and do the infra logs centralization later. That will give us a quick fix for now for the most critical logs (not awesome, but good enough imo), and learn from the loki setup for tools to take a better decision for the infra centralized setup later.

So by this do you mean merging the patches in T383081 to persist maintain-harbor logs? (As far as I'm aware, that's the only infrastructure cronjob we have.) Or do you see value in duplicating all infrastructure container logs to the journal regardless of whether they're kept by the container runtime already?

The documentation I linked says that the rsyslog Kubernetes metadata module seems to require connection details to the Kubernetes API. Is that documentation wrong? Or is that metadata somehow not necessary?

Missed that link, In my comment on the other task I was not specifically suggesting that solution though, just a generic solution. On those patches you can see that the namespace and the pod can be easily extracted from the path to the log itself, so as a start I think it might be good enough. The rest of the metadata we can investigate how to get it, if we want to get anything more. Might also be secondary, as in "get it if you can, but fallback to just adding the namespace+pod", to avoid depending on the k8s api at all.

So by this do you mean merging the patches in T383081 to persist maintain-harbor logs? (As far as I'm aware, that's the only infrastructure cronjob we have.) Or do you see value in duplicating all infrastructure container logs to the journal regardless of whether they're kept by the container runtime already?

Those for sure, and any others we might want to persist too if we fear losing them. Usually they are stored already for some time, so running deployments and such should not be critical imo (they also stay in the filesystem for a bit), but I'm happy to review which ones we want persisted if you have other concerns.

Currently I don't see any value on duplicating them no. It would be quite convenient though if we end up pulling the logs from the journal to whichever central place we store them in, whenever we have that central place (though we could also instead send them directly from the container logs).

In T398285#10962733, @dcaro wrote:

The documentation I linked says that the rsyslog Kubernetes metadata module seems to require connection details to the Kubernetes API. Is that documentation wrong? Or is that metadata somehow not necessary?

Missed that link, In my comment on the other task I was not specifically suggesting that solution though, just a generic solution. On those patches you can see that the namespace and the pod can be easily extracted from the path to the log itself, so as a start I think it might be good enough. The rest of the metadata we can investigate how to get it, if we want to get anything more. Might also be secondary, as in "get it if you can, but fallback to just adding the namespace+pod", to avoid depending on the k8s api at all.

Thanks, fixed in T398285#10963016.

So by this do you mean merging the patches in T383081 to persist maintain-harbor logs? (As far as I'm aware, that's the only infrastructure cronjob we have.) Or do you see value in duplicating all infrastructure container logs to the journal regardless of whether they're kept by the container runtime already?

Those for sure, and any others we might want to persist too if we fear losing them. Usually they are stored already for some time, so running deployments and such should not be critical imo (they also stay in the filesystem for a bit), but I'm happy to review which ones we want persisted if you have other concerns.

Currently I don't see any value on duplicating them no. It would be quite convenient though if we end up pulling the logs from the journal to whichever central place we store them in, whenever we have that central place (though we could also instead send them directly from the container logs).

Ack. I don't see much value in persisting logs of existed deployments. Please do feel free to add option E to the description, or just fold that to A if the stopgap solution for maintain-harbor is already being worked on.

Updated the task, reworded some stuff too to reflect the new option A differences and details

I don't have a strong opinion on this, I would probably go for Option A so that we can use Loki only for tools for a while, and later consider extending its usage to infrastructure logs.

Like option B, using Loki as the central store for the logs.

I'm not sure I understand option D: a single Loki instance for both tools logs and infra logs, whereas option B would have two separate Loki instances?

Given finite Taavi availability, A seems like the best option since it's the easiest (and pretty much done.) If there's time to do more then B seems like the obvious next step since it's nice and incremental.

Having separate log solutions for 'prod' and 'user' logs seems pretty normal to me, more likely a feature than a bug. But I may be missing something; does that imply that we eventually have N log solutions, or just 2?

There's no decision record for this?
Do we have the meeting minutes? (can't see them in google calendar)