Thanks for reporting @Xaosflux This is what seems to be the issue.

Apr 07 14:43:57 vrts1003 OTRS-CGI-10[4134825]: [Error][Kernel::Modules::AgentTicketQueue::Run][Line:271]: Invalid Filter: 'Unlocked'!
Apr 07 14:44:33 vrts1003 OTRS-CGI-10[4134825]: [Error][Kernel::Modules::AgentTicketQueue::Run][Line:271]: Invalid Filter: 'Unlocked'!

For the average number of requests for each planet sub-domain in the last 30 days: https://w.wiki/HywH

Expected due to T419712.

Test instance is back up. I did the following as recommended here https://gitlab.com/gitlab-org/gitlab/-/issues/590848#note_3100211428

aokoth@gitlab-1002:~$  sudo gitlab-psql -t -c "UPDATE pool_repositories
SET organization_id = (
  SELECT organization_id
  FROM projects
  WHERE projects.id = pool_repositories.source_project_id
)
WHERE organization_id IS NULL;
"

then:

sudo gitlab-ctl upgrade
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart

Started this on the test instance but it failed due to a migration error:

Manually deleted the timer and unit files from the host. So hopefully this does not fire again.

Resolving this since we already followed steps shared by Znuny for resolution.

@elukey Ack. Thank you.

Thanks @elukey

@Johannnes89 Np. Do we need to amend anything or this can be resolved?

@Ottomata / @Milimetric / @Ahoelzl Kindly approve.

@Johannnes89 Could you clear your cache and cookies and retry?

@Ottomata / @Milimetric / @Ahoelzl Kindly approve.

Similar to previous incidents of this (a bug in the exporter seems to cause this). I've restarted the service and the alert has recovered.

I think we can resolve this. Any follow up can be tracked on T411452: No space left on device on VRTS host.

@Dzahn Yes, I disabled that timer because it conflicted with another one run by the built-in VRTS daemon. Running both of them resulted in a bunch of noise on the emails from the daemon. I can enable it to run maybe daily and see how that goes.

lists2001 is currently setup as the mailman passive host in hieradata/common.yaml with the key lists_standby_host. From what I can tell, only the mailman3 service is masked, the rest of the services are running.

Will resolve this as it's pretty similar to other instances e.g. T402510

@Xaosflux Perhaps this is a bug... From what I can see, some counts are accurate while others aren't. Also, is it possible that some queues are configured differently? Maybe the count you see is influenced by the permissions you have in relation to the queue?

@Krd From what I've gathered from the docs, that's configured via the AdminEmail in the System Configuration. I don't have the historical context as to why it's current value is the admins mailing list. My assumption would be it's used in other places aside from being the source of password reset emails. The description on the setting is "Defines the system administrator's email address. It will be displayed in the error screens of the application."

@Dzahn The error seems to have resolved. I think maybe someone altered the setting and made it valid. I'll resolve this. I just saw 6.5.18 was released today. :)

Upgrade is complete though there is a new error related to one of the settings. Will close this once a resolution to the issue has been found.

Currently working on a the VRTS dashboard... I added a panel with the logs. Still tinkering with other panels so still work in progress.

I tried this but I think it might be a bit tricky to deploy in production. Firstly, it's not officially supported by Znuny so I'm doubtful they will be able to help if we run into any challenges. Secondly, I haven't found any useful docs on the same and most forum responses on the same are "reach out to the maintainer" which I doubt will be sustainable if we need a quick response on an issue.

From vrts-1003 on devtools. I'll test it out.

aokoth@vrts-1003:~$ apt search --names-only "^znuny"
Sorting... Done
Full Text Search... Done
znuny/stable-backports 6.5.15-2~bpo12+1 all
  flexible web-based ticketing system

Thanks @MoritzMuehlenhoff We'll consider that... But I'm doubtful we "strictly" need to test this on hardware. A VM will just work as well me thinks. @jhathaway is out on vacation so we can wait and hear is thoughts when he's back.

@Dzahn We used to run it on VMs but we kept running into resource issues (especially with clamav) even after resizing the VM a few times. Additionally, it was also suggested that we could migrate the attachments to the disk though this is still on ice as we wait for object storage support in newer Znuny versions and compare (last I checked this was still in development).

@Dzahn Or we could repurpose a spare server (if available)? miscweb comes to mind... Or were those VMs?

This coincides with an apache restart (though unclear what triggered it). Doubtful it has anything to do with decommissioning doc2002.