Page MenuHomePhabricator

deployment-eventlogging03 out of disk space
Closed, ResolvedPublic3 Estimated Story Points

Description

Found while looking into T170521: deployment-logstash2 out of disk space

$ ssh deployment-eventlogging03.eqiad.wmflabs df -h
Filesystem                          Size  Used Avail Use% Mounted on
udev                                2.0G   12K  2.0G   1% /dev
tmpfs                               396M  424K  395M   1% /run
/dev/vda1                            18G   18G     0 100% /
none                                4.0K     0  4.0K   0% /sys/fs/cgroup
none                                5.0M     0  5.0M   0% /run/lock
none                                2.0G     0  2.0G   0% /run/shm
none                                100M     0  100M   0% /run/user
/dev/mapper/vd-second--local--disk   21G  1.4G   19G   7% /srv

Event Timeline

greg added a subscriber: greg.

eventlogging -> Analytics

Please help here as well. Thanks!

Nuria added a subscriber: Nuria.

I was dropping log database

Nuria moved this task from Next Up to In Progress on the Analytics-Kanban board.

I have dropped logs in var/log and also srv/eventlogging but still there is a lot of space consumed by db that i have not been able to drop

Thanks to help from some releng folks looks like this are getting better:
root@deployment-eventlogging03:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 2.0G 12K 2.0G 1% /dev
tmpfs 396M 404K 395M 1% /run
/dev/vda1 18G 14G 3.3G 81% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 0 2.0G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/mapper/vd-second--local--disk 21G 1.6G 19G 8% /srv
overflow 1.0M 4.0K 1020K 1% /tmp

There's around 700MB in /var/lib/mysql.5.5.46-0ubuntu0.14.04.2 that hasn't been touched since 2015

Dropped tables and logs and run apt-get clean and apt-get autoclean, we are now at 80% space. Restarted eventloging again.

Nuria set the point value for this task to 3.