This is completed

That's fair re: user confusion concerns. From my SRE POV I was surprised to find that the CSP report url we announce filters the feed of legitimate, albeit confusing to tool maintainers, reports. I am thinking of a middle ground where we collect all reports and present the report firehose unfiltered only on demand. The known-domains retention of course can be short as we don't really care for it except for operational problems. What do you think ?

Thinking about this problem a little more: we would be lowering the default memory request, while leaving limit untouched, therefore I think it should be safe to do: many tools are already exceeding their requests though not hitting the limit today. I'm for testing a 128mb memory default request and take it from there, what do you think ?

The two failures I can identify are: oslo.messaging not failing over (T422820) and tooz lamenting memcached unavailable. I'm going to rename this task to address the latter

This is done, rabbit/openstack are now able to survive a rabbit host or server process going down (i.e. all durable queues) and automatically reload certs without a restart

This is done in eqiad and codfw

Done in codfw too. I left the security groups in place also in light of T422801

This is completed in eqiad1, codfw next

I have finished applying the configuration change to all eqiad1 trove instances, this time around more or less manually. Next up is deleting the exchanges and restart guest-agent

There isn't very much more info though I opened T422830: Openstack uwsgi logging to '<frozen importlib._bootstrap>.log'

I went back through the cloudcontrol1007 logs to see how extensive this problem is, P90364 contains logs across openstack components sorted by time and filtered for when they reconnected to rabbit. It looks like some components did reconnect as expected when cloudrabbit1001 went down, I haven't looked deep into why/how though

I saved all openstack components logs from /var/log to /root/filippo-T417393 on cloudcontrol1007 and cloudcontrol1011 to save them from rotation temporarily and further investigation.

In T417393#11799875, @Andrew wrote:

cloudcontrol nodes not in C8 (i.e. 1006/1007) though didn't seem to give up trying to connect to rabbitmq01.eqiad1.wikimediacloud.org:5671 whereas cloudcontrol1011 stopped trying to talk to rabbitmq01 as expected.

I tuned oslo settings for rabbitmq timeout &c but it was a long time ago, probably before our current rabbitmq setup. So we should do some new testing and reviewing of those settings. This blog post is very old but somewhat relevant to the topic: https://medium.com/@george.shuklin/rabbit-heartbeat-timeouts-in-openstack-fa5875e0309a

The other aspect to consider, which was the culprit in this case, is OS_PROJECT_ID vs OS_PROJECT_NAME usage (+OS_PROJECT_DOMAIN_NAME, which is default AFAICT).

Alternatively we can ask unattended-upgrades to not do anything until cloud-init has finished, though I'd rather avoid fixing up the fix up

Indeed under normal circumstances cloud-init will try to bring back puppet to 7 after the first puppet run (from modules/openstack/templates/nova/vendordata.txt.erb)

Following up from IRC: stopping memcached on all cloudcontrols, together with all designate servers, then restarting memcached and designate seems to have brought things back

FWIW the oslo timeout issue looks like to me a whole lot like https://bugs.launchpad.net/oslo.messaging/+bug/2096926

Today cloudrabbit1001 and cloudcontrol1011 were tested:

For context / more info

FS utilization has stabilized as segments are reclaimed

opentofu-infra-diff.service is failing on cloudcontrol1007 wrt this:

Update from network sync meeting: hosting read-only NFS behind LVS, like we're going to do for http/rsync (T306550) should be explored as a solution, which is going to be easier and simpler to maintain as opposed to the shared IP. I need to investigate/test more though at least conceptually read only NFS should be fine from the client's POV

I did some digging and the space is used by raft quorum logs via shared wal -> per-queue segments. The segments are single files on the filesystem and not actually deleted until the segment is full: https://www.rabbitmq.com/docs/quorum-queues#resource-use

Unfortunately I was too hasty: we do still see this, though it is unclear to me whether what the impact (if any) is

Plan LGTM, I haven't tested the code changes though Pontoon can help with that for sure.

In T421054#11771226, @dcaro wrote:

Note that despite the graph we have ~300G free on the VG, plus the vg0/srv LV is not really used and we can reclaim its space in case it is needed

Is it a leftover from an old partman recipe or something?

No longer observed

Ok now all queues but trove-guestagent are using quorum/durable, I'll be looking into deploying that too next.

The final bits of flipping neutron-l3-agent to quorum queues will be done tomorrow at 7 UTC within a scheduled window. The actual work to be performed:

This took a bunch of tries today and despite my best attempts to mess with rabbit and oslo, openstack reacted reasonably well IMHO.

This is fixed now, both rabbitmq server and CLI use ipv6 for erlang distribution protocol and its ports for CLI tools are open between rabbitmq servers

In T420565#11727731, @dcaro wrote:

Of immediate note the fact that 50% of namespaces/tools use less than 20%, i.e. we could be reducing their requests by 4-5x

Can we check how many of the ones that use more memory are actually setting the limits themselves vs using the default values?
It would be great if we can reduce the defaults without needing any user to add extra config to their tools.

The oslo setting I mentioned rabbit_transient_quorum_queue refers to the transient queues (reply, fanout) that openstack manages on rabbit, as opposed to the "service" queues which are indeed already quorum.

After some investigation I found the following:

1. rabbitmq cli utils use tcp ports 35672-35682 to communicate with nodes. In "reverse" in the sense that the host running e.g. rabbitmqctl acts as a temporary server to which other nodes connect to. These ports will need to be open on the firewall.
2. erlang doesn't implement happy eyeballs for dualstack hosts thus even if the ports are open they need to bind to v6, thus we'll need at least RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp" to make cli tools DTRT

Change deployed and rabbit roll-restarted:

FWIW I found some prior art / ideas here T367592: hadoop rolling reboot cookbook: add start-datetime flag

This is done, however 32GB barely made a dent into the % requests vs available. Resolving and will followup in parent task.

Today during T417393: Carry out controlled network switch down tests in cloud the same failure happened, namely cloudrabbit1001 was disconnected from the network and a partition was formed. When the host came back stopping and starting rabbit on the host eventually made things recover.

Tests today went significantly better: cloud vps networking stayed intact, I did start with failing over cloudgw which meant hosts using anycast addresses already failed over: lb, services. Rabbit suffered from network partition (i.e. T418444) though stopping and starting rabbit on cloudrabbit1001 eventually made things recover. cloudcontrol1011 was the last host and not tested yet

I also was wondering about a resumable rolling reboot feature for cookbooks and found this task, and of course I'm +1! The way I understand the feature currently is the following:

This is fixed! Thank you all for your help, and will follow up with another task to get http proxy support for spicerack openstack backend (and other related can of worms!)

We discussed this in the team meeting today: to restore functionality I have https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1253574 out. I'll be following up with a specific spicerack task for the openstack backend to be able to use an http proxy.

Confirmed that rabbitmq reloads certs without a restart:

I agree cloudcumin talking via prod http proxy like any other client is the right fix here. @Volans what do you think of the above idea? namely get cumin O backend to talk through prod proxies to the openstack api? from a quick look to wmcs-cookbooks spicerack shouldn't be affected in the sense that openstack interaction happens through CLI anyways and thus works

@taavi mentioned that https://gerrit.wikimedia.org/r/c/operations/homer/public/+/970275 might have broken this communication, which seems likely. Short of reverting that change what's the right approach here to make sure cloudcumin can talk to the openstack api? cc @ayounsi @cmooney

In T417393#11702131, @Andrew wrote:

In T417393#11701437, @fgiunchedi wrote:

Thank you, I looked at cloudvirt.drain though I couldn't find an option specifically to make sure the destination host is not in the rack we are draining. Maybe not a huge issue though? The scenario I'm thinking about is we're draining a cloudvirt and all/most VMs migrate to another cloudvirt in the same rack, of course things would converge eventually at the risk of moving VMs a bunch of times.

Correct, those cookbooks are not rack aware at all. The most efficient process would be to run the set_maintenance script on all cloudvirts in a given rack and then drain the individual cloudvirts; that would avoid moving any VMs more than once.

I'm also assuming that the most requests come from nfs workers above, to be verified once a nfs worker is added and how it changes memory requests %

Following the docs at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Add_a_worker this is what I'm planning on running, then wait for completion, observe memory requests percentage at https://grafana.wmcloud.org/goto/cffsdds3j2juod?orgId=1 and repeat as needed to bring reservation % to say 70%

In T417393#11698352, @Andrew wrote:

In T417393#11696203, @fgiunchedi wrote:

Plan is to grab another announced maint window on Tues March 17th to resume the testing.

I have also opened subtasks for the remaining racks, one notable difference is that those do contain cloudvirt hosts. @Andrew what's the recommended procedure to temporarily drain a rack of VMs and then put them back? So far I found wmcs.openstack.cloudvirt.drain cookbook mentioned on wikitech

wmcs.openstack.cloudvirt.drain should be what you need -- it will mark migrate VMs off the host and also mark the host as in maintenance. Then to repool you'll use wmcs.openstack.cloudvirt.unset_maintenance

Following up from T419674: ToolforgeKubernetesCapacity alert actionability

Thank you @dcaro for the pointer to T404726 ! I went through it again and it was a good read; I'm resolving this one in favor of T414513: Add new alerts for Toolforge cluster high load and will follow up there

Plan is to grab another announced maint window on Tues March 17th to resume the testing.

In T419508#11691123, @cmooney wrote:

Thanks @fgiunchedi. The other aspects we should monitor are the keepalived operations on both the cloudnet and cloudgw nodes, to make sure they are failing over. I think if we test each element one at a time we can be set up and logged on to all hosts and monitoring events, so hopefully we can isolate exactly what parts aren't working as expected.