Two other clusters are also down:

• https://wikistats.wmcloud.org/display.php?t=et
• https://wikistats.wmcloud.org/display.php?t=si

I'm very confused, the logs are full of undefined array keys but this started just a few hours ago. The code hasn't changed in months.

I restarted apache2 and it fixed it but it doesn't make much sense as to why it was failing

In T435959#12252172, @RhinosF1 wrote:

I'm very confused, the logs are full of undefined array keys but this started just a few hours ago. The code hasn't changed in months.

As I know is that Pywikibot test jobs passed 6 hours ago but failed 4 hours ago:
https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1328306 (Refer "Experimental build")

root@wikistats-bookworm:/var/log/apache2# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.0G     0  2.0G   0% /dev
tmpfs           393M  588K  393M   1% /run
/dev/sdb1        20G   19G     0 100% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sdb15      124M   12M  112M  10% /boot/efi
tmpfs           393M     0  393M   0% /run/user/0
tmpfs           393M     0  393M   0% /run/user/2075
tmpfs           393M     0  393M   0% /run/user/21170
/dev/sda        974M  122M  786M  14% /srv/wikistats-backup
root@wikistats-bookworm:/var/log/apache2#

I'm not sure if this could be a cause or an effect

I cleaned some space up but I'll chat to @Dzahn about further investgating

I don't know anything since I was completely away from work for weeks. I see this now and it was working before and it's working now. :)