Hello @taavi , thanks for helping reverse this. This was a oversight on my path. I thought the cli changes has already been deployed

yes docs. Will add that

In T364204#9852951, @Multichill wrote:

@Raymond_Ndibe what is the status? I see you merged something. I just tested and the problem still exists.

also I am working on unifying the parameters

then we rename this task to move jobs load logic to the jobs-api? that way it will still be valid for component-api? because we need to resolve this or else we can't say we've conclusively resolved the parent task https://phabricator.wikimedia.org/T364204

made some attempt to define somethings and answer some important questions on the task description, based on our discussion @dcaro . Input and possible modifications are welcome

Thanks @bd808

this is done right @dcaro? we should mark it as resolved if so

In T341066#9692931, @dcaro wrote:

In T341066#9691067, @Raymond_Ndibe wrote:

how will this affect the current 3 continuous jobs limit? does 2 replicas of a continuous job count as 1 or 2 when considering limits?

I would consider it as one, that uses 2x the amount of resources.

I have a thing against reuse-from. It is not immediately clear what it means by just looking at it. depends-on is a more descriptive name if I understand the supposed meaning of reuse-from correctly. In contrast reuse-from sounds like we are somehow reusing the configuration of a particular component in another. Obviously this is not already set in stone but it's important to point it out

Very valid point Bryan. I can't think of a reason too. The most important quotas to put in place imo are hardware resource quotas like ram and cpu. Maybe @dcaro and @taavi has an opinion?

In T362520#9796110, @bd808 wrote:

to some TBD higher value.

How about 16 to match the number of pods in the default quota? We can certainly pick any other arbitrary number >1 as well, but this at least has some rationale.

I find it pretty unlikely that a typical tool will exhaust either the Pod or Service quota at 16; I think the most typical tool will continue to be a webservice that consumes one Pod and one Service. In the spirit of T306324: Consider improving quota workflow I think we should set the default limits quite high relative to expected use when possible so that we don't end up discouraging folks from innovating by making them stop to ask for permission to use the platform.

one of the resources we need to be aware of here https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes

Can we link any resources we already have (automations, cookbooks, instructions, etc) on how we handle k8s upgrade here too? k8s upgrade is easy on paper but I assume it'll probably be more hairy for our particular implementation. Btw for the decision request I'm going with Option 2

marking as resolved. We can open it again if anyone disagrees

I think we can mark this as resolved now @taavi

In T335592#9691103, @bd808 wrote:

@Raymond_Ndibe I think this feature deserves a section on https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework and an email to cloud@ letting folks know it is possible now and any special things that they should look out for as they try to write their own check script.

In T335592#9699275, @bd808 wrote:

In T335592#9699268, @Raymond_Ndibe wrote:

@bd808 the script will be executed inside the pod. you can either provide an inline script (--health-check-script "echo this-is-a-script") or create a script file and do --health-check-script ./script.sh (goes without saying but you need to make the script executable). In both cases the running script can view the state of things inside the pod (processes, envvars, etc).

Thanks for these details @Raymond_Ndibe. If you use a script file, can that script file be a part of the job's custom build service manged image or does the file need to be readable by the toolforge jobs script on the bastion when the job is configured?

In T335592#9692471, @taavi wrote:

Re-opening since I think documentation needs to be added to https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework for this to be considered complete.

Also T348755: [jobs-api,webservice] Run webservices via the jobs framework will need a HTTP probe, should I file a separate task for that?

In T335592#9691106, @bd808 wrote:

In T335592#9691103, @bd808 wrote:

@Raymond_Ndibe I think this feature deserves a section on https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework and an email to cloud@ letting folks know it is possible now and any special things that they should look out for as they try to write their own check script.

Questions I have right now are where does the script actually end up running (inside the live Pod or elsewhere?) and can the script somehow see the state of things inside the live Pod including processes, envvars, and file system things?

we now have a --health-check-script argument that allows you to provide a custom health script that kubernetes uses to decide when your workload becomes unhealthy so it can be restarted. This only works for continuous jobs

how will this affect the current 3 continuous jobs limit? does 2 replicas of a continuous job count as 1 or 2 when considering limits?

@dcaro do you have any idea on how to reproduce this issue?

I think I need help reproducing this. So far multiple runs on tools and toolsbeta failed to trigger this error. Is it likely that this is a combination of more than 1 factor, maybe a result of multiple users trying to upload at the same time?

In T354116#9580068, @dcaro wrote:

There's documentation about the buffering being disabled if the client uses HTTP 1.1 with chunked transfer:
"""
When HTTP/1.1 chunked transfer encoding is used to send the original request body, the request body will be buffered regardless of the directive value unless HTTP/1.1 is enabled for proxying.
""
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering

But it seems the client is using 2.0, I have not found yet any mention of how http 2.0 is handled.