Need to ask @hnowlan whether our example API need to be "productionized" in order to view the logs aka for this task to be completed

In this context is the example API one offered by the API gateway? Logs from the gateway are already in Logstash and we can easily set up a filter to capture the correct logs for the API in question

Thanks @hnowlan . Can you elaborate on "logs from the gateway"?

I'm guessing you mean traffic logs, while logs related to business logic within the service would be handled in a different way. But maybe I'm wrong.

For example, suppose an SQL error occurred within an service. The API endpoint might respond with a 500 and a high-level message suitable for consumers, but the service might also log a more detailed error message including things like table/column information that would make less sense in the HTTP response.

Both types of logging, of course, are of interest.

In T288132#7301439, @BPirkle wrote:

Thanks @hnowlan . Can you elaborate on "logs from the gateway"?

I'm guessing you mean traffic logs, while logs related to business logic within the service would be handled in a different way. But maybe I'm wrong.

For example, suppose an SQL error occurred within an service. The API endpoint might respond with a 500 and a high-level message suitable for consumers, but the service might also log a more detailed error message including things like table/column information that would make less sense in the HTTP response.

Both types of logging, of course, are of interest.

Sorry, I misconstrued in this case - I mean traffic logs, access logs and response codes from requests to a service - actual logs coming from the service itself would be somewhere else. If the service ends up in k8s, the logs will go to logstash for free

What I have learned thus far about logging from speaking with observability team:

• If your service is not on k8s, there is no ready-made solution to configure logging (does not matter if hosted on cloudVPS or toolforge or elsewhere) . But if your service is not on k8s, you can follow this guide to forward logs to ryslog
• You can no longer directly send logs to logstash, it must be forwarded to rsyslog
• The existing service-template-node config sends logs via gelf stream > to logstash, but this is long deprecated and the service-template-node needs updating (see open issue)
• An example of this situation happening with citoid https://phabricator.wikimedia.org/T219919

Change 716461 had a related patch set uploaded (by Nikki Nikkhoui; author: Nikki Nikkhoui):

[operations/puppet@production] Add image suggestion api to lookup table

https://gerrit.wikimedia.org/r/716461

Change 716461 abandoned by Nikki Nikkhoui:

[operations/puppet@production] Add image suggestion api to lookup table

Reason:

https://gerrit.wikimedia.org/r/716461

Change 716461 restored by Nikki Nikkhoui:

[operations/puppet@production] Add image suggestion api to lookup table

https://gerrit.wikimedia.org/r/716461

Change 716461 merged by Cwhite:

[operations/puppet@production] Add image suggestion api to lookup table

https://gerrit.wikimedia.org/r/716461

Update to the previous comment: there is no way to get logging from a WMCS node that is not under deployment-prep project onto logstash. So we will have to rely on services being in k8s for the logging to happen.

Change 719366 had a related patch set uploaded (by Nikki Nikkhoui; author: Nikki Nikkhoui):

[operations/puppet@production] Remove image suggestion api from lookup table

https://gerrit.wikimedia.org/r/719366

Change 719366 merged by Cwhite:

[operations/puppet@production] Remove image suggestion api from lookup table

https://gerrit.wikimedia.org/r/719366

I've added comments to the API notes section on why logging is not possible from cloud VPS or toolforge.
Logging comes for free from k8s, we just need to document that API producers need to adhere to logging schema, which I have added in the API notes section.