Page MenuHomePhabricator

Connect Kafka to the MVP [Mile Stone 5]
Closed, ResolvedPublic

Description

Goal: Connect Kafka to the MVP and test the functionality

Success Criteria:

  • Kafka connected to MVP
  • Have search and filtering working as expected
  • Have the connection process documented.

Event Timeline

Milimetric renamed this task from Connect remaining Data Sources to the MVP [Mile Stone 5] to Connect Metadata Sources to the MVP [Mile Stone 5].Feb 14 2022, 5:01 PM
Milimetric updated the task description. (Show Details)
Milimetric moved this task from Backlog to Next Up on the Data-Catalog board.
Milimetric renamed this task from Connect Metadata Sources to the MVP [Mile Stone 5] to Connect Kafka to the MVP [Mile Stone 5].Feb 14 2022, 5:03 PM
Milimetric updated the task description. (Show Details)

I have ingested kafka-jumbo with the following recipe.

source:
  type: "kafka"
  config:
    connection:
      bootstrap: "kafka-jumbo1001.eqiad.wmnet:9092"
      schema_registry_url: http://karapace1001.eqiad.wmnet:8081

sink:
  type: "datahub-rest"
  config:
    server: 'https://datahub-gms.discovery.wmnet:30443'

I invoked it like this:

datahub ingest -c kakfa-jumbo.yml`

At the moment there is no schema information present, nor anything like retention time.

There are 807 datasources from kafka-jumbo.

BTullis moved this task from In Progress to Done on the Data-Catalog board.

There is an issue when attempting to browse the Kafka datasets, resulting in an error 500 being sent to the client.

I have created this ticket to track the fix: T308736: Resolve 500 errors when browsing Kafka datasets

It is likely that upstream will address it in a future release.

EChetty raised the priority of this task from Medium to High.May 19 2022, 9:54 AM
EChetty moved this task from Done to Blocked on the Data-Catalog board.