Page MenuHomePhabricator

Stop saving eventlogging data on eventlog1002
Closed, ResolvedPublic


Today, eventlog1002's /srv partition filled up. I don't think anything in particular has changed, we just reached a threshold. /srv is 870G on eventlog1002, and we are keeping the last 7 days of both 'all-events' (the eventlogging-valid-mixed topic) and client side raw events. This ends up being a lot!

Do we need to keep this anymore? We used to do this before we had Kafka so folks could check their data in files on disk. Now, we all this data in to Hadoop anyway, and it is in Kafka.

If not, let's just remove the eventlogging 'consumer's that are writing to log files on disk on eventlog1002, and remove the rsync cron on stat1007.

Event Timeline

It looks like the client-side consumer restarted from the beginning of the topic after my restart of eventlogging daemons yesterday:

So it has logged a ton of things, and now it seems to have caught up. I am +1 on removing this file-based consumer, it would be a nice clean up.

The only place where these files are used is on beta, to test event emission.

Change 616871 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] eventlogging Stop outputing to local files on eventlogging server

Change 616871 merged by Ottomata:
[operations/puppet@production] eventlogging - Stop outputing to local files on eventlogging server

Mentioned in SAL (#wikimedia-analytics) [2020-07-28T17:52:38Z] <ottomata> stopped riting eventlogging data log files on eventlog1002 and stopped syncing them to stat100[67] - T259030

fdans claimed this task.

Change 655791 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] eventlogging: Remove profile::eventlogging::analytics::files

Change 655791 merged by Ottomata:
[operations/puppet@production] eventlogging: Remove profile::eventlogging::analytics::files