Page MenuHomePhabricator

librenms.syslog table is 800GB
Closed, ResolvedPublic

Description

Similar to: T349362
librenms database lives in m1
The syslog table is more than 800GB:

-rw-rw---- 1 mysql mysql 864G Jan 22 14:56 syslog.ibd

Approximate number of rows:

cumin2024@db2160.codfw.wmnet[mysql]> select n_rows from innodb_table_stats where table_name='syslog';
+------------+
| n_rows     |
+------------+
| 2787667314 |
+------------+

Can we please truncate or delete things from it? We probably shouldn't be using a relational database for this sort of thing, but if we do, we should regularly clean up things? Maybe a cronjob or an alert for when it goes over...100GB?
At T349362#9267648 we discussed having just 15 days of retention, is that still the case? If it is, maybe we need to reduce it a bit further?

Details

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui added subscribers: cmooney, ayounsi.

At T349362#9267648 we discussed having just 15 days of retention, is that still the case? If it is, maybe we need to reduce it a bit further?

That would be an insane amount for 15 days. Tbh I'd have no issue with disabling syslogs to librenms I typically check kibana if I am looking for them, which seems a lot more suited to them.

@ayounsi is back tomorrow I will discuss with him and get back to you.

Probably the same root cause as T412143: ~5k/logs/sec from netdev.

We might need to temporarily ignore those log messages until JTAC provides us with a permanent fix.

Would it be possible to truncate it for now?

Would it be possible to truncate it for now?

Yep

Marostegui claimed this task.

Thanks

cumin2024@db1213.eqiad.wmnet[librenms]> truncate table syslog;
Query OK, 0 rows affected (0.012 sec)

cumin2024@db1213.eqiad.wmnet[librenms]>
root@db2160:/srv/sqldata.m1/librenms# ls -lh syslog.ibd
-rw-rw---- 1 mysql mysql 11M Jan 23 11:10 syslog.ibd
root@db2160:/srv/sqldata.m1/librenms#

Change #1237465 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Remove LibreNMS as syslog target

https://gerrit.wikimedia.org/r/1237465

Change #1237465 merged by jenkins-bot:

[operations/homer/public@master] Remove LibreNMS as syslog target

https://gerrit.wikimedia.org/r/1237465

I saw the above change being merged, can I go ahead and truncate this table again so it is left empty?

root@db1213:/srv/sqldata/librenms# ls -lh syslog.ibd
-rw-rw---- 1 mysql mysql 238G Feb  9 06:42 syslog.ibd
Marostegui moved this task from Done to In progress on the DBA board.