Page MenuHomePhabricator

Clean up huge logs on toollabs
Closed, ResolvedPublic

Description

So we have about 1.3T of logs on toollabs. P628 lists the biggest offender. Almost 1T is just one log file, filled with php warnings and notices. There's a NFS backup going on, so we should clean these up after.

Event Timeline

yuvipanda raised the priority of this task from to Needs Triage.
yuvipanda updated the task description. (Show Details)
yuvipanda added a project: Toolforge.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 8 2015, 11:26 PM
valhallasw triaged this task as High priority.May 10 2015, 7:50 PM
valhallasw moved this task from Triage to Backlog on the Toolforge board.May 10 2015, 8:42 PM

NFS backup still going on, will be at least 3 days before it's done, I think.

Don't attempt to rm these from the labs instances tho - should be removed on the server directly or we'll kill NFS :)

coren added a comment.May 13 2015, 7:47 AM

Actually, unlinking a file via NFS should not be any harder on the server than doing it directly - the real thing to watch out for is to make sure it is not currently opened by any process. Something that /can/ be done from the server which might be a better idea is to simply truncate the file.

scfc added a comment.May 28 2015, 12:27 AM

With regard to fiwiki-tools, I have asked the maintainer here.

yuvipanda moved this task from To Do to Doing on the Labs-Sprint-100 board.Jun 2 2015, 5:03 PM

cluebot's error log grew to 1.5T again, deleted and notifying maintainer (https://en.wikipedia.org/wiki/User_talk:Rich_Smith#Cluebot.27s_error_logs)

valhallasw lowered the priority of this task from High to Low.Aug 2 2015, 1:35 PM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptAug 2 2015, 1:35 PM
chasemp closed this task as Resolved.Mar 2 2016, 11:39 PM
chasemp added a subscriber: chasemp.

This has happened a few times and we will have to continue to clean things up. I'm hopeful T126623 will make it easier to identify and easier for users to understand in teh future. For now I'm going to resolve this as there isn't active cleanup in progress.