Similar to previous incidents of this (a bug in the exporter seems to cause this). I've restarted the service and the alert has recovered.

I think we can resolve this. Any follow up can be tracked on T411452: No space left on device on VRTS host.

@Dzahn Yes, I disabled that timer because it conflicted with another one run by the built-in VRTS daemon. Running both of them resulted in a bunch of noise on the emails from the daemon. I can enable it to run maybe daily and see how that goes.

lists2001 is currently setup as the mailman passive host in hieradata/common.yaml with the key lists_standby_host. From what I can tell, only the mailman3 service is masked, the rest of the services are running.

Will resolve this as it's pretty similar to other instances e.g. T402510

@Xaosflux Perhaps this is a bug... From what I can see, some counts are accurate while others aren't. Also, is it possible that some queues are configured differently? Maybe the count you see is influenced by the permissions you have in relation to the queue?

@Krd From what I've gathered from the docs, that's configured via the AdminEmail in the System Configuration. I don't have the historical context as to why it's current value is the admins mailing list. My assumption would be it's used in other places aside from being the source of password reset emails. The description on the setting is "Defines the system administrator's email address. It will be displayed in the error screens of the application."

@Dzahn The error seems to have resolved. I think maybe someone altered the setting and made it valid. I'll resolve this. I just saw 6.5.18 was released today. :)

Upgrade is complete though there is a new error related to one of the settings. Will close this once a resolution to the issue has been found.

Currently working on a the VRTS dashboard... I added a panel with the logs. Still tinkering with other panels so still work in progress.

I tried this but I think it might be a bit tricky to deploy in production. Firstly, it's not officially supported by Znuny so I'm doubtful they will be able to help if we run into any challenges. Secondly, I haven't found any useful docs on the same and most forum responses on the same are "reach out to the maintainer" which I doubt will be sustainable if we need a quick response on an issue.

From vrts-1003 on devtools. I'll test it out.

aokoth@vrts-1003:~$ apt search --names-only "^znuny"
Sorting... Done
Full Text Search... Done
znuny/stable-backports 6.5.15-2~bpo12+1 all
  flexible web-based ticketing system

Thanks @MoritzMuehlenhoff We'll consider that... But I'm doubtful we "strictly" need to test this on hardware. A VM will just work as well me thinks. @jhathaway is out on vacation so we can wait and hear is thoughts when he's back.

@Dzahn We used to run it on VMs but we kept running into resource issues (especially with clamav) even after resizing the VM a few times. Additionally, it was also suggested that we could migrate the attachments to the disk though this is still on ice as we wait for object storage support in newer Znuny versions and compare (last I checked this was still in development).

@Dzahn Or we could repurpose a spare server (if available)? miscweb comes to mind... Or were those VMs?

This coincides with an apache restart (though unclear what triggered it). Doubtful it has anything to do with decommissioning doc2002.

T397899

This is fully running on the aux cluster but the last update that allowed the rsync to run daily broke the CSS. Working on a fix for this.

doc1003 is decom'd now... Resolving this.

I've switched out the active doc host to doc2003 running Bookworm with PHP 8.2. cc: @matmarex

@JMeybohm Sure. I'll move it over there.

@jcrespo Yes, it's under setup. Apologies for the noise. I'll add it to the ignore list.

This failed because I removed doc1003 from the list of all_hosts which is what sets up the firewall on the primary so the rsync was blocked as a result. I'll be switching over the active doc host doc2003 and decom'ing doc1003 next week. So I'll resolve this for now.

Thank you @KFrancis

Hello @Kappakayala Do you approve this request?

Hi @Milimetric Do you approve of this request?

@DMburugu Kindly approve.

@SKivlehan-WMF This was merged and deployed... Kindly test and confirm if it works as expected.

@jcrespo Yes, it's a new host meant to replace doc1003. From looking at the manifest there seems to be a section that was added to prevent alerting so I'm not entirely sure the backup is supposed to work. cc: @hashar @Dzahn

# This is to prevent monitoring from alerting. The directory is empty until
# we have completed the migration of generated documentations. Once done,
# we should remove the file.
file { '/srv/doc/BACKMEUP':
    ensure  => 'file',
    owner   => 'root',
    group   => 'root',
    mode    => '0444',
    content => 'Placeholder for Bacula',
}

@KFrancis Please work with @Corvus on the NDA.

Ack @MoritzMuehlenhoff Thank you.

I restarted apache. The alerts recovered too.