Without having looked into anything specifically, I highly suspect issues are similar to {T430888} (apart from running out of disk space). Specifically, I recommend:

• Upgrading Anubis – this was the main thing that helped XTools come back up.
• Add PHP-FPM, which should address the flood of Apache processes that prevent new connections from being made (T430888#12080489).

please tune up the system so that failed connections get closed quickly; that will reduce the likelihood of wsexports breaking other cloud-vps services.

The proxy, in an ideal world, should just be able to handle any traffic the backends can throw it without breaking at least for other tenants. My patch in the subtask should make it at least not hit this particular issue again, so I've started the apache service back up. The service is still broken with the disk full and so I'll leave this task open for you to handle that.

Thanks for working on this!

I've now:

• Upgraded Anubis to 1.27, and it seems to be working correctly.
• Removed the 5GB of access logs. Haven't updated logrotate for apache though (I think adding maxsize 300M might be all that's needed).
• Switched to PHP-FPM, status page is now active again at https://ws-export.wmcloud.org/fpm-status

The site is back up anyway.

Thanks for fixing this before – unfortunately PDF exports seem to have stopped working again on enws and frws (haven't checked other languages), as reported at Wikisource:Scriptorium#WSExport, with the error message:

503 Service Unavailable

Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Apache/2.4.68 (Debian) Server at ws-export.wmcloud.org Port 80

The discussion says OTRS pdf export is also affected (unsure if this is the same service; Wikipedia pdf export seems to be working as normal). Wikisource epub exports seem to be unaffected at the moment.

In T437349#12410429, @Yodin wrote:

The discussion says OTRS pdf export is also affected (unsure if this is the same service; Wikipedia pdf export seems to be working as normal). Wikisource epub exports seem to be unaffected at the moment.

This is a misunderstanding. There is no such thing as an OTRS pdf export (OTRS has been renamed to VRT years ago, BTW) but we have an open ticket at VRT, complaining about Wikisource PDF function not working.

In T437349#12410429, @Yodin wrote:

[..] unfortunately PDF exports seem to have stopped working again on enws and frws (haven't checked other languages), [..]

This effects other projects as well, have just checked de.wikisource.org. There is the same problem.