The videoconvert tool's data usage on tools is spiking up rapidly over the last few days. We are at pretty high tools usage right now. Please clean up your data soon, and let us know if we can help with long term strategies for any clean up. Our NFS store (which is underlying tools) is not meant to be used to store large amounts of data, and if you need to do this, do reach out to us on Cloud-Services so we can help with alternate strategies if possible.
Description
Event Timeline
I truncated jerr.log (https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL) since it was 967 GB, and since I did that it has grown to 700 MB, it seems like this file is growing rapidly
It's now at 2G. I've owned the file to root:root, and will revert when this is fixed.
I cant login to the server right now, What does the logfile say? I can check it in the evening.
@Prolineserver Repeated PHP stacktraces that look like this:
PHP Stack trace: PHP 1. {main}() /mnt/nfs/labstore-secondary-tools-project/videoconvert/public_html/server/php/shell.php:0 PHP 2. UploadHandler->commonsupload() /mnt/nfs/labstore-secondary-tools-project/videoconvert/public_html/server/php/shell.php:38 PHP 3. readdir() /mnt/nfs/labstore-secondary-tools-project/videoconvert/public_html/server/php/UploadHandler.php:1657 PHP Warning: readdir() expects parameter 1 to be resource, boolean given in /mnt/nfs/labstore-secondary-tools-project/videoconvert/public_html/server/php/UploadHandler.php on line 1657
You can ping me (madhuvishy) on Cloud-Services when you work on this so I can revert permissions when this is fixed. Thanks!
Fixed it for now, only a single place caused the error. I added a check for a false directory handle though I can't reproduce where it actually originated from. Deleted the error log file.
Also cleaned the actual video data by running "php cleanup.php >> cleanup170520.txt" in /data/project/videoconvert/public_html/server/php, reduced the data from 226GB to 20GB.