Initially I was on the side of Option 4. But maybe that's too extreme and won't be worth sweating over? idk.
Option 2 is till a good middle ground so I have no problem with that too. It's something we can start immediately we close this decision request too. If Option 2 get's more vote we can go with that.

In T374875#10150343, @Raymond_Ndibe wrote:

some pointers that might help to reproduce this:

log into cloud cumins:

ssh user@cloudcumin1001.eqiad.wmnet

run tmux

tmux

inside tmux, run sudo su:

sudo su

run cookbook with task-id specified

cookbook <cookbook.path> --task-id <task id>

The above is the way I'm currently running the cookbook. Hopefully it'll help in reproducing this

some pointers that might help to reproduce this:

oooh this https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/511 ?
Thought the idea is to be done with toolsbeta before deploying anything on tools?

What failed upgrade are we talking about @dcaro ? The upgrade to 1.26 or did we attempt doing anything directly on tools yesterday while planning for 1.27 upgrade?

Toolsbeta:

• create harborstorage object storage on horizon
• figure out authentication for the harborstorage object storage
• create a new compute instance toolsbeta-harbor-2
• setup harbor on the new instance and configure it to use the harborstorage object storage (In progress)
  • setup harbor
  • configure harbor to use object storage (s3)
  • test push and pull locally using limakilo (In progress)
• configure toolsbeta-harbor-1 to replicate to toolsbeta-harbor-2 through the harbor ui
• announce toolsbeta harbor down time to read-only mode
• reconfigure toolsbeta-harbor-1 harbor to use the harborstorage object storage
• any necessary cleanup (delete toolsbeta-harbor-2, etc)

Tools:

• create harborstorage object storage on horizon
• figure out authentication for the harborstorage object storage
• create a new compute instance tools-harbor-2
• setup harbor on the new instance and configure it to use the harborstorage object storage
  • setup harbor
  • configure harbor to use object storage (s3)
  • test push and pull locally using limakilo
• configure tools-harbor-1 to replicate to tools-harbor-2 through the harbor ui
• announce tools harbor down time to read-only mode
• reconfigure tools-harbor-1 harbor to use the harborstorage object storage
• any necessary cleanup (delete tools-harbor-2, etc)

If this becomes a toolforge component, I'm thinking about discarding the toolforge-jobs abstraction that we use here and scheduling maintain-harbor jobs directly on k8s through deployment templates like we do for other repos. what do you think about that @dcaro?

closing because this has already been implemented and deployed. See the above linked task by Taavi for more context.

In T372720#10071887, @JJMC89 wrote:

The current quota is fine. 50 cronjobs can be scheduled but only 15 jobs can run concurrently.

In T353701#9887644, @Slst2020 wrote:

This is unlikely to be fixed even with new robot permissions. Only admin accounts can access system-wide configurations (via api/v2.0/configurations), which is the only place the default project quota can be retrieved from. Similarly, only admin users can set the default quota which prevents us from defining it in code.

I'm tempted to leave things as they are right now:

local.tf-test@lima-kilo:~$ toolforge build quota
BuildClientError: Error getting quota from Harbor: Quota cannot be displayed because no builds have run yet

this looks like something we can handle with harbor @dcaro

what could be the reason for this? any pointers? logs and servers to look at? etc
It'd be nice to see what you @Andrew and @fnegri are already doing and looking at, and results so far

Documentation can be found here https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Harbor/maintain-harbor

In T358203#10068690, @Raymond_Ndibe wrote:

Tested on lima-kilo:
https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/46#note_100236