Page MenuHomePhabricator

jembot error logs growing too fast
Closed, ResolvedPublic

Description

The jembot tool is generating a lot of error messages that have consumed 260 gigabytes in the past 10 days. We're running short on available NFS capacity (T247315) and had to truncate the error.log file.

@-jem- can you please take a look at these errors?

log snippet
2020-03-20 14:59:02: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Notice:  Undefined variable: Index in /mnt/nfs/labstore-secondary-tools-project/jembot/public_html/ti/index.php on line 1010
2020-03-20 14:59:02: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Stack trace:
2020-03-20 14:59:02: (mod_fastcgi.c.2543) FastCGI-stderr: PHP   1. {main}() /mnt/nfs/labstore-secondary-tools-project/jembot/public_html/ti/index.php:0
2020-03-20 14:59:02: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Notice:  Undefined variable: Main in /mnt/nfs/labstore-secondary-tools-project/jembot/public_html/ti/index.php on line 1012
2020-03-20 14:59:02: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Stack trace:
2020-03-20 14:59:02: (mod_fastcgi.c.2543) FastCGI-stderr: PHP   1. {main}() /mnt/nfs/labstore-secondary-tools-project/jembot/public_html/ti/index.php:0
2020-03-20 14:59:02: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Notice:  Undefined variable: Log in /mnt/nfs/labstore-secondary-tools-project/jembot/public_html/ti/index.php on line 1015
2020-03-20 14:59:02: (mod_fastcgi.c.2543) FastCGI-stderr: PHP Stack trace:
2020-03-20 14:59:02: (mod_fastcgi.c.2543) FastCGI-stderr: PHP   1. {main}() /mnt/nfs/labstore-secondary-tools-project/jembot/public_html/ti/index.php:0

Event Timeline

I'm very sorry for this. I fixed it some weeks ago. I will add automatic checks to prevent this from happening again.