Just imagine this situation:

For completeness, drbd after the operation:

Done. No issues detected with the connection test.

Done. No packet drop detected on this test.

I'll leave this test running during the operation:

I'll leave this ping test running for the time of the operation:

I think I'll go the configmap route. Let me write a patch.

For the record, before the operation:

In T287107#7229504, @Majavah wrote:

In T287107#7229499, @aborrero wrote:

So a simple solution here is to connect to the neutron port VIP directly, instead of the floating IP address. Didn't we have a trick in the resolver to workaround this? Or did we drop it?

We do (labsaliaser), but this address is not directly assigned to a VM (it's managed with keepalived), so it does not find the address when looping thru all instances and their addresses from Nova.

I see at least one thing I can explain.

In T286784#7227761, @Bstorm wrote:

@aborrero if all that sounds reasonable, I can try to start implementing it quick before you are done with the beta so we get feedback.

The current approach as of this writing to avoid the harcoded container list is to fetch them from the registry at API startup time.

In T287036#7225527, @Bstorm wrote:

@aborrero do you think we can expect a patched backport from the official repo eventually?

oh, I actually know what happened here. I changed the way we store the command in the job definition. Now we wrap the command on a /bin/sh exec to support file logging (T286485), and that's preventing the jobs API parser from working properly.

I just uploaded new versions of both the API and the CLI.

I just checked this: labstore1007 is already standby. labstore1006 is active. No actions are required for this server.

In T286108#7217298, @Majavah wrote:

In T286108#7217292, @aborrero wrote:

I suspect we'll be on k8s 1.21 before we leave the beta phase for this.

How long do you expect the beta phase to last?

I wasn't uncomfortable with having to delete each job individually after completion. It allowed to review execution results and status.
But I totally understand the desire for them to be auto-cleaned up.

I'm currently evaluating this: https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/ i.e, simply executing a shell oneliner to send an email from within the job pod.

I added the package back!

Sharing a bit our experience @ WMCS with ingress-nginx:

I have more information on this.

In T285944#7199809, @Legoktm wrote:

Is there a way to dump/export the toolforge-jobs config so it can be committed in Git? And then a way to import that config back. I tend to commit crontabs for documentation and transparency purposes.

I don't see any other option here but add additional parameters to our REST API. Will do!

Is this something the grid supports today?

For the record, I identified the problem early in the development of the framework. It was a conscious decision to leave it out in the first iteration because I wanted to collect clear use cases and expectations.
I was hoping that the wrapper approach that is suggested in wikitech was enough to cover basically all cases.

In T286065#7194569, @Bstorm wrote:

> @aborrero does cloudgw require manual failover?

In T285944#7191072, @LucasWerkmeister wrote:

If toolforge-jobs is only available on dev-buster.toolforge.org – is the SSH fingerprint for that host available somewhere? (I don’t see it on Help:SSH Fingerprints yet, nor in the draft announcement.)

The initial build out is completed. Closing the ticket now.