Goal: Connect Kafka to the MVP and test the functionality
Success Criteria:
- Kafka connected to MVP
- Have search and filtering working as expected
- Have the connection process documented.
Goal: Connect Kafka to the MVP and test the functionality
Success Criteria:
I have ingested kafka-jumbo with the following recipe.
source: type: "kafka" config: connection: bootstrap: "kafka-jumbo1001.eqiad.wmnet:9092" schema_registry_url: http://karapace1001.eqiad.wmnet:8081 sink: type: "datahub-rest" config: server: 'https://datahub-gms.discovery.wmnet:30443'
I invoked it like this:
datahub ingest -c kakfa-jumbo.yml`
At the moment there is no schema information present, nor anything like retention time.
There are 807 datasources from kafka-jumbo.
There is an issue when attempting to browse the Kafka datasets, resulting in an error 500 being sent to the client.
I have created this ticket to track the fix: T308736: Resolve 500 errors when browsing Kafka datasets
It is likely that upstream will address it in a future release.