[components-api,buildsa-api] When building and deploying, if none of the settings changed, the jobs are not restarted
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dcaro
	Mar 17 2025, 9:17 AM

Project Tags

Referenced Files

None

Subscribers

Aklapper

dcaro

Raymond_Ndibe

Description

This is because the image name and tag is the same, so jobs-api considers the job to be the same though the actual contents of the image did change.

This might be solved partially the same way as T389043: [builds-api] Store the commit hash that was used for the build, though a first simple approach is to restart all continuous jobs (scheduled ones will just pull the new image on the next run)

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	toolforge.component.deploy: run only component tests by default	cloud/wmcs-cookbooks	main	+51 -17

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	LucasWerkmeister	T320140 Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes
Resolved	matmarex	T319707 Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes
Resolved	Legoktm	T320062 Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes
Resolved	Legoktm	T320011 Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes
In Progress	dcaro	T194332 [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs
Resolved	dcaro	T393564 [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features
Resolved	Raymond_Ndibe	T389044 [components-api,buildsa-api] When building and deploying, if none of the settings changed, the jobs are not restarted
Resolved	dcaro	T389043 [builds-api] Store the commit hash that was used for the build

Event Timeline

dcaro created this task.Mar 17 2025, 9:17 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 17 2025, 9:17 AM

dcaro updated the task description. (Show Details)Mar 17 2025, 9:18 AM

dcaro triaged this task as High priority.Mar 18 2025, 10:00 AM

dcaro edited projects, added: Toolforge (Toolforge iteration 19); removed: Toolforge (Toolforge iteration 18).Mar 18 2025, 3:56 PM

dcaro added a parent task: T393564: [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features.May 12 2025, 12:35 PM

dcaro added a subtask: T389043: [builds-api] Store the commit hash that was used for the build.May 14 2025, 8:43 AM

dcaro mentioned this in T393564: [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features.May 14 2025, 9:02 AM

dcaro edited projects, added: Toolforge (Toolforge iteration 20); removed: Toolforge (Toolforge iteration 19).May 14 2025, 2:30 PM

dcaro closed subtask T389043: [builds-api] Store the commit hash that was used for the build as Resolved.May 19 2025, 12:33 PM

Now that we have the resolved_ref property returned by the builds-api for each build, we can compare the commit of the last build with the commit from the new build and decide if it's the same or not.

@dcaro this seems like something that should be done on the jobs-api side. What's your view on how we should approach this?
I think since the jobs api already compares jobs and decides if it should recreate or not, it should also be able to do deeper level comparison by checking if the there is a change in the image (using the resolved_ref thing. wdyt?
Would you rather we do it on the components api side?

Raymond_Ndibe claimed this task.May 20 2025, 2:22 AM

In T389044#10837733, @Raymond_Ndibe wrote:

@dcaro this seems like something that should be done on the jobs-api side. What's your view on how we should approach this?
I think since the jobs api already compares jobs and decides if it should recreate or not, it should also be able to do deeper level comparison by checking if the there is a change in the image (using the resolved_ref thing. wdyt?
Would you rather we do it on the components api side?

The jobs-api does not have access to the resolved_ref property of the build, it just knows about the image name.

The problem comes also whenever the image changes, but the commit does now (ex. when it's rebuilt later from the same commit, maybe pulling in different dependencies).

I propose the following behavior of the deployment flow on the components-api:

Adding two new boolean parameters to the deploy edpoint, force-rebuild and force-restart
If both are true:
- Always build the images
- Always restart the jobs, even if they did not change
If force-rebuild is false (default):
- if the resolved_ref of the last build for the component in the previous deployment is the same as the current hash -> skip build for that component
- if there's no previous deployment, or the component did not exist in the previous deployment, or the destination image does not exist anymore, or the build reference in the previous deployment does not exist anymore, rebuild
If force-restart is false (default):
- if the build did not change, don't restart the job
If force-restart is true
- if the job spec is the same (so jobs-api did not restart it), do an explicit restart of the job

If/when we start exposing information about available runtimes on the builds-api (ex. to list the pre-built images that are available, and the bulidservice images available for the tool), then we can add the information of the commit hash the buildservice one is built from too, so the jobs-api can also retain that information on the job datastructure itself (once we store it), and we can compare more in detail which image was used for that job, and if the new job would change that.

Other options are to add to the builds-api a way to expose/configure the tags an image has, so we can use that with the commit hash as the image name, so the newer image will have a different commit hash, though that has to be thought a bit more clearly.

This might also be influenced by whichever way we do the rollbacks. For now, I'd start with the above flow, that is doable and does not prevent any of the future solutions.

dcaro changed the task status from Open to In Progress.May 20 2025, 9:38 AM

dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 20) board.

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/77

[components-api] skip build if refs are same

Raymond_Ndibe moved this task from In Progress to In Review on the Toolforge (Toolforge iteration 20) board.May 23 2025, 1:57 AM

maybe we should add this "force" field to the config? that way it's possible to pick and choose the components we want to re-build/re-start

if instead we add this to the cli, then we should also add a way to deploy decide the components to deploy

In T389044#10851153, @Raymond_Ndibe wrote:

maybe we should add this "force" field to the config? that way it's possible to pick and choose the components we want to re-build/re-start

if instead we add this to the cli, then we should also add a way to deploy decide the components to deploy

We are intentionally not supporting per-component actions yet, we will deal with that later, for now making it simple and deploying everything is more than enough.

Note also that you deploy without passing any config, so it has to be there even if we do per-component deploy eventually.

It should be a parameter of the POST /deployment endpoint, that is component agnostic.

Once we start supporting per-component deploys, then we'll extend whatever method we use to deploy specific components to support that option (be that a path variable like POST /components/<component_name>/deploy, or something else).

Feel free to open a task for the per-component deployment and start adding ideas/notes on what's needed and how things could work. But for this task, let's keep it simple, and focus on what we support now.

In T389044#10851295, @dcaro wrote:

In T389044#10851153, @Raymond_Ndibe wrote:

maybe we should add this "force" field to the config? that way it's possible to pick and choose the components we want to re-build/re-start

if instead we add this to the cli, then we should also add a way to deploy decide the components to deploy

We are intentionally not supporting per-component actions yet, we will deal with that later, for now making it simple and deploying everything is more than enough.

Note also that you deploy without passing any config, so it has to be there even if we do per-component deploy eventually.

It should be a parameter of the POST /deployment endpoint, that is component agnostic.

Once we start supporting per-component deploys, then we'll extend whatever method we use to deploy specific components to support that option (be that a path variable like POST /components/<component_name>/deploy, or something else).

Feel free to open a task for the per-component deployment and start adding ideas/notes on what's needed and how things could work. But for this task, let's keep it simple, and focus on what we support now.

What I was trying to suggest was adding the force field to the config instead of passing it as a parameter to the deploy endpoint. It is not exactly clear what option you prefer reading your comment above. I am also aware the config is not separate thing and not provided during the deploy. What I suggest is a config that looks like this:

build:
    ...
    force: True
deploy:
    command: <whatever command>
    force: True

Instead of passing it as part of the deploy endpoint

Hey, if you put it in the config, there's no way to control per-deployment (you'd have to reconfigure, then create deploy, then reconfigure to restore), I propose passing it as a parameter of the deploy create endpoint, so you can do things like "soft-deploy" every hour to pull code changes, and force deploy every week to rebuild with the latest dependencies (as an example):