Page MenuHomePhabricator

Move trove DB instances to rabbitmq transient quorum queues
Closed, ResolvedPublic

Description

With https://gerrit.wikimedia.org/r/c/operations/puppet/+/1264557 trove-guestagent got rabbit transient queue durable enabled. This task tracks deploying the change to DB instances

The per-host procedure I (Filippo) worked out as of April 2026 is as follows:

  1. Identify the instance name and trove ID from wmcs-openstack database instance list --all-projects
  2. Enable ssh to the host: wmcs-openstack --os-project-id trove server add security group NAME ssh-from-anywhere
  3. Login with the trove ssh key as user ubuntu
  4. Apply the following patch to /etc/trove/conf.d/trove-guestagent.conf
@@ -95,8 +95,13 @@ log_dir = /var/log/trove/
 log_file = logfile.txt
 # log_config_append = /etc/trove/trove-logging-guestagent.conf
 
+[oslo_concurrency]
+lock_path = /var/lib/trove/tmp
+
 [oslo_messaging_rabbit]
+use_queue_manager=true
 rabbit_quorum_queue=true
+rabbit_transient_quorum_queue=true
 rabbit_retry_interval=1
 rabbit_retry_backoff=2
 
  1. Delete the exchange and fanout queue from one of the cloudrabbit hosts:
rabbitmqctl list_exchanges | grep TROVE-ID
rabbitmqadmin delete exchange name=EXCHANGE-NAME
rabbitmqctl list_queues | grep TROVE-ID_fanout
rabbitmqctl delete_queue QUEUE-NAME
  1. Restart guestagent on the host: systemctl restart guest-agent
  2. The queue will be back under a different name: rabbitmqctl list_queues | grep TROVE-ID_fanout
  3. Remove the ssh security group: wmcs-openstack --os-project-id trove server remove security group NAME ssh-from-anywhere

Event Timeline

Change #1265425 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] openstack: add lock_path for trove-guestagent

https://gerrit.wikimedia.org/r/1265425

taavi triaged this task as High priority.Apr 1 2026, 1:57 PM

Change #1265425 merged by Filippo Giunchedi:

[operations/puppet@production] openstack: add lock_path for trove-guestagent

https://gerrit.wikimedia.org/r/1265425

Mentioned in SAL (#wikimedia-cloud) [2026-04-07T10:23:31Z] <godog> switch mariatest1 to transient quorum queues - T421857

I have finished applying the configuration change to all eqiad1 trove instances, this time around more or less manually. Next up is deleting the exchanges and restart guest-agent

Mentioned in SAL (#wikimedia-cloud) [2026-04-13T09:42:23Z] <godog> delete non-durable guestagent rabbit exchanges and roll-restart guest-agent - T421857

This is completed in eqiad1, codfw next

Done in codfw too. I left the security groups in place also in light of T422801

fgiunchedi claimed this task.