Page MenuHomePhabricator
Create Task
Maniphest T259030

Stop saving eventlogging data on eventlog1002
Closed, ResolvedPublic
Actions

Description

Today, eventlog1002's /srv partition filled up. I don't think anything in particular has changed, we just reached a threshold. /srv is 870G on eventlog1002, and we are keeping the last 7 days of both 'all-events' (the eventlogging-valid-mixed topic) and client side raw events. This ends up being a lot!

Do we need to keep this anymore? We used to do this before we had Kafka so folks could check their data in files on disk. Now, we all this data in to Hadoop anyway, and it is in Kafka.

If not, let's just remove the eventlogging 'consumer's that are writing to log files on disk on eventlog1002, and remove the rsync cron on stat1007.

Details

SubjectRepoBranchLines +/-
eventlogging: Remove profile::eventlogging::analytics::filesoperations/puppetproduction+0 -72
eventlogging - Stop outputing to local files on eventlogging serveroperations/puppetproduction+0 -43
Customize query in gerrit

Event Timeline

Ottomata created this task.Jul 28 2020, 1:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 28 2020, 1:48 PM
elukey added a comment.Jul 28 2020, 2:47 PM

It looks like the client-side consumer restarted from the beginning of the topic after my restart of eventlogging daemons yesterday:

https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?orgId=1&from=now-24h&to=now&var-datasource=eqiad%20prometheus%2Fops&var-cluster=jumbo-eqiad&var-topic=All&var-consumer_group=eventlogging_consumer_client_side_events_log_00

So it has logged a ton of things, and now it seems to have caught up. I am +1 on removing this file-based consumer, it would be a nice clean up.

• Nuria added a comment.Jul 28 2020, 4:21 PM

The only place where these files are used is on beta, to test event emission.

gerritbot added a comment.Jul 28 2020, 5:00 PM

Change 616871 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] eventlogging Stop outputing to local files on eventlogging server

https://gerrit.wikimedia.org/r/616871

gerritbot added a project: Patch-For-Review.Jul 28 2020, 5:00 PM
gerritbot added a comment.Jul 28 2020, 5:46 PM

Change 616871 merged by Ottomata:
[operations/puppet@production] eventlogging - Stop outputing to local files on eventlogging server

https://gerrit.wikimedia.org/r/616871

Stashbot added a comment.Jul 28 2020, 5:52 PM

Mentioned in SAL (#wikimedia-analytics) [2020-07-28T17:52:38Z] <ottomata> stopped riting eventlogging data log files on eventlog1002 and stopped syncing them to stat100[67] - T259030

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2020, 6:10 PM
• Nuria added a comment.Jul 28 2020, 9:21 PM

Deleted big files from homedir

• fdans closed this task as Resolved.Aug 3 2020, 4:45 PM
• fdans claimed this task.
gerritbot added a comment.Jan 12 2021, 10:47 PM

Change 655791 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] eventlogging: Remove profile::eventlogging::analytics::files

https://gerrit.wikimedia.org/r/655791

gerritbot added a project: Patch-For-Review.Jan 12 2021, 10:47 PM
gerritbot added a comment.Jan 13 2021, 2:00 PM

Change 655791 merged by Ottomata:
[operations/puppet@production] eventlogging: Remove profile::eventlogging::analytics::files

https://gerrit.wikimedia.org/r/655791

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits