Page MenuHomePhabricator
Create Task
Maniphest T377629

[Hive] Investigate Hadoop packaging, install, security monitoring.
Closed, ResolvedPublic
Actions

Description

Hadoop is a dependency for Hive, which is a dependency for Trino.

We need to investigate how we want to fetch, install and configure the service, and develop a process to monitor for vulnerabilities and handle software updates.

See https://kontext.tech/article/448/install-hadoop-330-on-linux for example.

Related Objects
Search...

Event Timeline

Jgreen created this task.Oct 18 2024, 8:47 PM
Irisbdc moved this task from Backlog to In Progress on the Fundraising analytics stack board.Oct 23 2024, 5:04 PM
Irisbdc removed a project: Epic.
Jgreen added a comment.EditedOct 25 2024, 9:20 PM

First pass of getting the prototype config into puppet is done:

  • /etc/default/hadoop - has environment variables from the prototype .bashrc
  • /var/lib/hadoop/etc-test - has config files from /var/lib/hadoop/etc/hadoop, templated for several site variables
  • /var/log/hadoop - created and writable by user hadoop
  • hadoop user has ssh keys configured and host keys are cached

Still to do:

  • go through the configs and make sure everything is according to plan
  • systemd units for each of the hadoop-related processes
  • security tracking and maintenance process
Jgreen closed subtask T378087: [mariadb-connector-j] Investigate packaging, install, security monitoring. as Resolved.Oct 28 2024, 12:14 PM
Jgreen reopened subtask T378087: [mariadb-connector-j] Investigate packaging, install, security monitoring. as Open.
Jgreen triaged this task as Medium priority.Oct 28 2024, 5:25 PM
Jgreen moved this task from In Progress to Backlog on the Fundraising analytics stack board.Oct 29 2024, 8:30 PM

Puppet config is done, it does the install and configuration to the point where hadoop is up and running. The following are known and deferred for the moment:

  • firejail config needed, it is not running in firejail yet
  • firewall policy
  • monitoring
  • security tracking and maintenance process

Moving the task to Backlog for now.

Jgreen moved this task from Backlog to Paused on the Fundraising analytics stack board.Nov 5 2024, 9:34 PM
Jgreen closed subtask T378087: [mariadb-connector-j] Investigate packaging, install, security monitoring. as Resolved.Nov 8 2024, 2:36 PM
Irisbdc moved this task from Paused to Completed on the Fundraising analytics stack board.Dec 10 2024, 3:12 PM
Irisbdc closed this task as Resolved.Dec 10 2024, 7:16 PM
Jgreen edited parent tasks, added: T377635: [Hive] Investigate packaging, install, security monitoring.; removed: T377362: EPIC: Trino/MinIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation.Apr 16 2025, 2:06 PM
Jgreen edited parent tasks, added: T377368: EPIC: Hive (hive-standalone-metastore) Implementation; removed: T377635: [Hive] Investigate packaging, install, security monitoring..
Jgreen removed a subtask: T378087: [mariadb-connector-j] Investigate packaging, install, security monitoring..Apr 16 2025, 2:09 PM
Jgreen renamed this task from [Hadoop] Investigate packaging, install, security monitoring. to [Hive] Investigate Hadoop packaging, install, security monitoring..Apr 16 2025, 2:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits