Improve ml-serve's Istio logs
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	elukey
	Feb 2 2022, 8:23 AM

Description

We should improve the way that we log requests in Istio, to have something like the ORES dashboard with access logs etc..

Moreover, we should clear some log-spam. For some reason, the istio gateways on ml-serve clusters emit the following warning over and over:

2022-02-02T07:54:39.796130Z	warning	envoy misc	Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.http2_protocol_options' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.

I don't see the same warning for the ServiceOps' staging codfw cluster, so it may be something specific to the ML configuration. It would be nice to get rid of these logs, they are very annoying to read when debugging issues.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		None	T272917 Lift Wing proof of concept
		Resolved		elukey	T300707 Improve ml-serve's Istio logs

Event Timeline

elukey created this task.Feb 2 2022, 8:23 AM

The issue seems to be https://github.com/istio/istio/pull/32387

Commit: https://github.com/istio/istio/commit/7bb5275ecdbef28b6b1c18a28b582233b669baa8

We copy tools/packaging/common/envoy_bootstrap.json to the proxyv2 docker image, and the http2_protocol_options option needs to be moved elsewhere. In theory with istio 1.13 the issue should go away, or we could simply patch the envoy_bootstrap file manually in our production-images repo.