Page MenuHomePhabricator
Create Task
Maniphest T217350

Partition event-data daily instead of hourly (for sanitized data)
Closed, DuplicatePublic
Actions

Description

This would provide 2 benefits:

  • Smaller number of bigger files for events with low volumes (less pressure on HDFS)
  • Smaller number of partitions to maintain and work with (less pressure on metastore)

Event Timeline

JAllemandou created this task.Feb 28 2019, 5:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 28 2019, 5:20 PM
Milimetric renamed this task from Partition event-data daily instead of hourly to Partition event-data daily instead of hourly (for sanitized data).Feb 28 2019, 5:33 PM
Milimetric triaged this task as Low priority.
Milimetric moved this task from Incoming to Event Platform on the Analytics board.
Milimetric moved this task from Event Platform to Data Quality on the Analytics board.
• Tbayer subscribed.Mar 4 2019, 8:38 PM
JAllemandou closed this task as a duplicate of T236794: Find a strategy to mitigate small-files handling for long-term kept events.Oct 29 2019, 3:01 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL