HTTP 401, is that the openstack API?

This is a duplicate similar to T346453: [cumin] [openstack] Openstack backend fails when project is not set, but I thought it only failed when a project was not specified?

It's now definitely failing when a project is set, but I think they are two separate issues:

• when running cumin with O{project:some-project} it fails without an error message (but seemingly only if the project contains many hosts)
• when running cumin with O{*} or A:all it fails with HTTP 401 because of T346453, that should be fixable with the workaround listed there

The failure when a project is specified is a concurrency issue, running cumin with -b 10 works fine. This is probably related to T340241: cloudcumin is slow when targeting many cloud VMs.

Increasing connect_timeout from 20 to 40 in /etc/cumin/config.yaml does indeed fix the issue. I have no idea why 20 was enough when I set it in T323484, I'm sure I tested it in large projects with hundreds of hosts and it worked fine.

Instead of increasing the value to 40, I will first try to assign more CPUs to the restricted bastion, as discussed in T340241.

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-27T10:52:38Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'bastion-restricted-eqiad1' (T347428)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-27T10:55:10Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'bastion-restricted-eqiad1' (T347428)

After replacing the bastion in T340241, it's no longer necessary to add -b 10, though Cumin is still not as fast as I'd like (more details in that task).

Fixing the 401 issue is tracked in T346453, so I'll mark this one as "Stalled" until that one is resolved.