Add meaningful access logs to KServe's pods
Open, MediumPublic
Actions

Assigned To

None

Authored By

	elukey
	Apr 3 2023, 9:28 AM

Project Tags

Referenced Files

None

Subscribers

Aklapper

elukey

isarantopoulos

Description

The issue is described in https://github.com/kserve/kserve/issues/2778, and a solution is provided in https://github.com/kserve/kserve/pull/2782

The end goal is to have a way to customize access logs for KServe pods, to print headers like User Agent etc.. and have them collected by logstash and displayed by Kibana. This will allow us to create a dashboard to inspect the Lift Wing's traffic when needed.

High level work to do:

Work with upstream to add the code needed to customize access logs.
Wait for the new Kserve release, and update all model-servers to it.
Upgrade the K8s control plane to the new release as well.

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	revscoring: customize kserve logs	Ilias Sarantopoulos	machinelearning/liftwing/inference-services	main	+4 -1

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T333453 Lift Wing improvements to get out of MVP state
		Open		None	T333804 Add meaningful access logs to KServe's pods

Event Timeline

elukey created this task.Apr 3 2023, 9:28 AM

elukey updated the task description. (Show Details)

elukey claimed this task.Apr 3 2023, 9:31 AM

elukey moved this task from Unsorted to In Progress on the Machine-Learning-Team board.

elukey moved this task from In Progress to Blocked on the Machine-Learning-Team board.Apr 13 2023, 3:42 PM

https://github.com/kserve/kserve/pull/2782 got merged, it will be released in a few weeks with KServe 0.11

From https://github.com/benoitc/gunicorn/issues/2457 I created:

'{"remote_address": "%(h)s", "user_name": "%(u)s", "date": "%(t)s", "status": "%(s)s", "method": "%(m)s", "url_path": "%(U)s", "query_string": "%(q)s", "protocol": "%(H)s", "response_length": "%(B)s", "referer": "%(f)s", "user_agent": "%(a)s", "request_time_seconds": "%(L)s"}'

In my test model I just added:

if __name__ == "__main__":
    model = CustomPredictor("test")
    kserve.ModelServer(
        workers=1, enable_latency_logging=False,
        access_log_format='{"remote_address": "%(h)s", "user_name": "%(u)s", "date": "%(t)s", "status": "%(s)s", "method": "%(m)s", "url_path": "%(U)s", "query_string": "%(q)s", "protocol": "%(H)s", "response_length": "%(B)s", "referer": "%(f)s", "user_agent": "%(a)s", "request_time_seconds": "%(L)s"}'
    ).start([model])

And I got valid JSON responses:

{"remote_address": "127.0.0.1:49886", "user_name": "-", "date": "[11/May/2023:10:06:24 +0000]", "status": "200", "method": "POST", "url_path": "/v1/models/test:predict", "query_string": "q=blabla", "protocol": "HTTP/1.1", "response_length": "20", "referer": "-", "user_agent": "test", "request_time_seconds": "0.001030"}

Not sure if we can go as far as setting ECS as format, but the above should be sufficient to have a good Kibana dashboard :)

Found a corner case (not a big blocker): https://github.com/Kludex/asgi-logger/issues/40

elukey mentioned this in T337213: Update to KServe 0.11.May 22 2023, 7:45 AM

This is blocked until we upgrade to KServe 0.11 :)

elukey mentioned this in T346446: Upgrade revscoring Docker images to KServe 0.11.Sep 22 2023, 6:47 AM

Shall we proceed with the format mentioned above?

'{"remote_address": "%(h)s", "user_name": "%(a)s", "date": "%(t)s", "status": "%(s)s", "method": "%(m)s", "url_path": "%(U)s", "query_string": "%(q)s", "protocol": "%(H)s", "response_length": "%(B)s", "referer": "%(f)s", "user_agent": "%(a)s", "request_time_seconds": "%(L)s"}'

So far i just haven't managed to get the user agent to show up by changing "user_name": "%(u)s" to "user_name": "%(a)s" according to asgi-logger

Currently I'm debugging some logging errors related to our decorators used to log time elapsed while fetching featues etc. More info if anyone is interested in this paste.

isarantopoulos reopened this task as Open.Sep 22 2023, 2:59 PM

isarantopoulos moved this task from Blocked to In Progress on the Machine-Learning-Team board.Sep 22 2023, 3:02 PM

elukey removed elukey as the assignee of this task.Oct 9 2023, 3:31 PM

elukey moved this task from In Progress to Blocked on the Machine-Learning-Team board.

Change 964568 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] revscoring: kserve logs

https://gerrit.wikimedia.org/r/964568

gerritbot added a project: Patch-For-Review.Oct 9 2023, 4:44 PM

Since asgi-logger can only be used if we specify the access_log_format we defined the environment variable LOGGING_FORMAT in the above patch to allow logging customization in revscoring inference services.
At the moment only inference services updated to kserve 0.11 can access thing functionality.

achou triaged this task as Medium priority.Nov 2 2023, 7:29 PM

Change #964568 abandoned by Ilias Sarantopoulos:

[machinelearning/liftwing/inference-services@main] revscoring: customize kserve logs

Reason:

cleaning up

https://gerrit.wikimedia.org/r/964568

isarantopoulos moved this task from Blocked to Backlog/Lift Wing on the Machine-Learning-Team board.Jun 4 2024, 2:36 PM

Add meaningful access logs to KServe's podsOpen, MediumPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Add meaningful access logs to KServe's pods
Open, MediumPublic
Actions

Related Objects
Search...