For production the pod seems to get evicted because of ephemeral-storage not being specified in the resources we request when the job is running.

The site now has some data so it's already up to 18GB it seems.

https://github.com/wbstack/charts/pull/84 bumps it to 60GB by default

On staging they seem to run fine

https://github.com/wmde/wbaas-deploy/pull/262 for deploying

This morning the failed because 0/4 nodes are available: 4 Insufficient cpu. seems we should alter this too.

Updated https://github.com/wbstack/charts/pull/84 to specify a request as the docs say

If you specify a limit for a resource, but do not specify any request, and no admission-time mechanism has applied a default request for that resource, then Kubernetes copies the limit you specified and uses it as the requested value for the resource.

This thing still needs a +2 https://github.com/wmde/wbaas-deploy/pull/262

So a few new patches.

https://github.com/wbstack/charts/pull/87 mounts a volume to be used as scratch space for taking backups
https://github.com/wmde/wbaas-backup/pull/15 moves the temp work dir and output to be under /backups

This is however based on the validation added here and that should get reviewed first https://phabricator.wikimedia.org/T306493

Ok to summarize what the problem was and what is proposed.

As the migration started the disk usage increased and the ephemeral storage provided by the nodes weren't enough for a long term solution.

Therefore we need to either give it more space or reduce the disk usage of the logical backups (could be done by just writing straight to the bucket but the already existing validation tasks are going in the opposite direction T306493: 🟦️ Add validation of backups to wbaas-backup, validate existance of temp files etc.)

Some initial attempts were done using the ephemeral storage provided by the nodes but this produced some different behavior if the restorePod was running or if the CronJob was using the same configuration. For the restore pod a new PV was created for the time with the requested size however for the CronJob this was not the case and the size was limited to what the Nodes could offer. Relying on the storage of the nodes felt overall like a less than ideal solution as we could be a bit more explicit with where this storage comes from.

The proposed solution is,

1. Move all temporary writing to happen under /backups and add a cleanup script that removes temporary files after a backup is taken. https://github.com/wmde/wbaas-backup/pull/15
2. Add a terraform managed compute disk that gets mounted by the job/restore-pod under /backups and do all the temp storage there. https://github.com/wmde/wbaas-deploy/pull/283
3. Cut a new chart for wbaas-backup that uses this new version https://github.com/wbstack/charts/pull/87 (this still requires image bump + new image tag)
4. Use the new chart on staging https://github.com/wmde/wbaas-deploy/pull/284

So, after review @Tarrow fortunately wanted to give the ephemeral storage approach another go and found that this difference in behavior seems to be coming from the way manual jobs are scheduled. And seems after all it might work but depends on running the manual jobs from kubectl rather than the google ui.

The left-most yaml is created by running a manual job from within the google UI, and this seems to just ignore the ephemeral storage part and instead use the emptyDir type

The right-most yaml is created when running a manual job using kubectl kubectl create job --from=cronjob/sql-logic-backup sql-logic-backup-manual-01

So, lets skip the terraform managed disk and use this + update the docs not to use the ui, might also be worth confirming that actual CronJobs in GKE also gets created correctly when scheduled.

might also be worth confirming that actual CronJobs in GKE also gets created correctly when scheduled.

Cronjob triggered by the schedule seems to also work correctly, this behavior is only from the UI. boo!

The proposed solution is,

1. Move all temporary writing to happen under /backups and add a cleanup script that removes temporary files after a backup is taken. merged!
2. Cut a new chart for wbaas-backup that uses this new version with generic ephemeral volume merged
3. Use the new chart on staging/local https://github.com/wmde/wbaas-deploy/pull/284

This was just deployed to production, and took a manual backup. The output can be validated by looking in wikibase-cloud-sql-backup bucket under cloud storage.

We might need to size up the scratch disk space pretty soon again though depending on how big the data from next migration batches would be.