I don’t think webservice necessarily has to support patching in raw YAML snippets as each probe type, at least not as the main mode of this feature. We can probably assume that HTTP probes are the common case, and standardize on a common default path (e.g. /health or /healthz), which would then also be filtered out of access logs by default (T127367? though that talks about error logs).

See also T314053: Allow automatically restarting tool web services if non OK error code (which I think is closer to the automatic restart use case than the zero-downtime restart use case).

and standardize on a common default path (e.g. /health or /healthz), which would then also be filtered out of access logs by default

And if you have some good reason to deviate from the default path (my preference is /healthz since it doesn't conflict with a real word), it could be configurable via service.template.

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/23

add probes

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/24

k8s: allow passing the http probe path

In T341919#9017404, @LucasWerkmeister wrote:

standardize on a common default path (e.g. /health or /healthz), which would then also be filtered out of access logs by default (T127367? though that talks about error logs).

Liveness and other lifecycle probes would happen from within Kubernetes and thus show on grafana dashboards via the Kubernetes stats import to Prometheus, but they would not route through the front proxy where we collect access log data for https://toolviews.toolforge.org/.

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/23

k8s: add default tcp probes

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/24

k8s: allow passing the http probe path

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/25

d/changelog: bump to 0.103.2

Mentioned in SAL (#wikimedia-cloud) [2024-02-28T11:57:21Z] <dcaro> deploy tools-webservice 0.103.2 with probes (T341919)

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/25

d/changelog: bump to 0.103.2

This is available for use now, I'll leave the task open to do a bit of following for the next few days monitoring https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview?orgId=1

Current state:

@dcaro, this is awesome and I think also completely worthy of a message to cloud-announce. :)

Mentioned in SAL (#wikimedia-cloud) [2024-02-28T18:50:17Z] <wmbot~lucaswerkmeister@tools-sgebastion-10> deployed 3030faaa3c (health-check-path, T341919)

Seems to work like a charm, thanks a lot! The “only terminate old pod once new one is ready” seems to behave as expected:

In T341919#9017404, @LucasWerkmeister wrote:

We can probably assume that HTTP probes are the common case, and standardize on a common default path (e.g. /health or /healthz), which would then also be filtered out of access logs by default (T127367? though that talks about error logs).

FWIW, as long as we don’t have the common logging solution yet, filtering out /healthz from the uWSGI logs turns out to be easy enough for Python webservices: see uwsgi.ini in the commit in my tool.

In T341919#9584300, @bd808 wrote:

@dcaro, this is awesome and I think also completely worthy of a message to cloud-announce. :)

+1 :)

FYI, I added this to the cookiecutter-toolforge template / boilerplate for new tools now: https://github.com/lucaswerkmeister/cookiecutter-toolforge/commit/621944e6a7aa7944cafa42efef03b9b2fccbbf01

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/27

d/changelog: bump to 0.103.3

If HTTP probes are configurable in service.template, can that please be documented on Wikitech? If it is not configurable, can that feature be added?

In T341919#9589241, @tstarling wrote:

If HTTP probes are configurable in service.template, can that please be documented on Wikitech? If it is not configurable, can that feature be added?

Added something there. (It’s health-check-path: PATH, i.e. the service.template YAML and the command line option use the same key, similar to mem:/--mem or cpu:/--cpu.)

In T341919#9589279, @LucasWerkmeister wrote:

In T341919#9589241, @tstarling wrote:

If HTTP probes are configurable in service.template, can that please be documented on Wikitech? If it is not configurable, can that feature be added?

Added something there. (It’s health-check-path: PATH, i.e. the service.template YAML and the command line option use the same key, similar to mem:/--mem or cpu:/--cpu.)

Thanks, yes that's the key, you can see all the options that are set in the service.manifest file after you run toolforge webservice start ...options..., and copy that file as template.
We should add a subcommand to generate the template file with all the defaults too as example, that would be helpful.

Could someone point me to how to make the probe not appear in access.log for perl5.36 webservice?

In T341919#9617675, @Dvorapa wrote:

Could someone point me to how to make the probe not appear in access.log for perl5.36 webservice?

Based on https://redmine.lighttpd.net/projects/lighttpd/wiki/Mod_accesslog#Disable-logging, this might work:

$HTTP["url"] == "/healthz" { accesslog.filename = "" }