Page MenuHomePhabricator
Create Task
Maniphest T298710

Connect Atlas to a Data Source
Closed, DeclinedPublic
Actions

Description

This task is to connect the cloud test Atlas instance to a test data source - either Hive and/or a MariaDB instance. This would involve installing a test data source and configuring Altas to read the metadata. Ideally having a copy of the mediawikidatabase would give us ability to view pertinent data information. It might be possible to connect to an actual production data source in which case that might be an easier option for loading test metadata.

This would allow us to evaluate Atlas more comprehensively both from a UX perspective and technology integrations.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedMilimetricT293643 Data Catalog Technical Evaluation
 DuplicateMilimetricT299165 Evaluate Atlas
 Resolved• razziT296670 Run Atlas on test cluster
 DeclinedNoneT298710 Connect Atlas to a Data Source

Event Timeline

odimitrijevic created this task.Jan 6 2022, 5:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 6 2022, 5:00 PM
odimitrijevic triaged this task as High priority.Jan 6 2022, 5:00 PM
odimitrijevic added a parent task: T296670: Run Atlas on test cluster.
odimitrijevic moved this task from Incoming (new tickets) to Security & Governance on the Data-Engineering board.
odimitrijevic added a project: Data-Catalog.Jan 13 2022, 7:51 PM
Milimetric subscribed.Jan 13 2022, 7:56 PM

Atlas doesn't seem to have a first-class connector for MySQL / MariaDB, people have created scripts that use the REST API to manage this kind of import. The conclusion here is that Atlas mostly cares about the Hive metastore.

Other metadata we're interested in, like AirFlow lineage, is pushed to Atlas via connectors on AirFlow's side.

BTullis subscribed.Jan 19 2022, 6:20 PM

As discussed in T296670#7633000 it looks like we have a serious blocker in connecting our test instance of Atlas to an existing Hive metastore.
We would either need to upgrade Hive to 3.1+ (which necessitates an upgrade to Hadoop 3+) or we would have to deploy an older version (1.2.0) of Atlas.

• EChetty moved this task from Backlog to Next Up on the Data-Catalog board.Jan 24 2022, 12:40 PM
• EChetty moved this task from Next Up to Done on the Data-Catalog board.
• EChetty moved this task from Done to Blocked on the Data-Catalog board.
BTullis closed this task as Declined.Feb 8 2022, 11:13 AM
BTullis moved this task from Next Up to Done on the Data-Engineering-Kanban board.

We can't easily do this with our evaluation setup, due to Hive incompatibility.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL