Summary

Our highest-volume data arrives as streams in Kafka. That includes webrequest, the record of requests to our sites, and the event platform streams. Apache Gobblin copies those streams into the Data Lake in batches.

Gobblin needs replacing for two reasons. The project is no longer actively developed. And it writes to HDFS, which we are moving away from in T435461: [Epic] Migrate the Data Lake from Hadoop to Kubernetes and Ceph/S3.

This task is an evaluation of possible alternatives. It compares the tools that could do this job against criteria we agree first, and ends in a recommendation. Making the change is the parent task, T435489: Data Lake ingestion after Hadoop: decide the future of Sqoop and Gobblin.

This might be the more straightforward of the two ingestion decisions. We run Gobblin much as it comes, with no custom tooling wrapped around it, so there is less of our own logic to carry across.
It therefore need not wait for the database import evaluation in T437079: [Tech evaluation] How we import the MediaWiki databases into the S3 data lake, although one tool might serve both jobs.
If that looks likely, then the two evaluations should compare notes.

Candidates

• Flink's dynamic Iceberg sink, the preference stated in review. It is part of Apache Iceberg itself, it writes to many tables from one job, and it documents which schema changes it can absorb and which it cannot. We already run Flink on dse-k8s, with the operator and a stream application, so much of the operational groundwork exists. It might also take over the Refine Spark jobs, removing a step rather than replacing one.
• Apache SeaTunnel, release 2.3.13, an Apache top-level project, reading Kafka and writing Iceberg. A job is a configuration file rather than a program, and it can run on Flink or on Spark. Its particular appeal here is that it might also do the database import job, leaving us one tool to operate rather than two.
• Kafka Connect with the Iceberg sink, which comes with a licensing question and some history. We tried this once: T223628: Replace Camus with Kafka Connect for event data imports proposed replacing Camus with Kafka Connect and was dropped because Confluent had moved the connector we needed to a licence that is not free software. T238400: Evaluate possible replacements for Camus: Gobblin, Marmaray, Kafka Connect HDFS, etc., T185233: Modern Event Platform and T214430: Event Platform: Stream Connectors record the same obstacle, and it is much of why we run Gobblin today. What has changed is that the parts we would need now are free software: Kafka Connect ships as part of Apache Kafka, and the Iceberg sink is a component of Apache Iceberg, with multi-table routing of its own. So if we wish to re-evaluate this tool, we should confirm the relevant licences. It would be new for us to operate, as we currently run no Kafka Connect.
• Spark, reading Kafka, either as a streaming job or as a scheduled batch read. This keeps the work on the engine we already use for batch processing, at the cost of doing streaming work on an engine designed for batches.

What we need to find out

Can it understand our event schemas? This is the question most likely to decide the outcome, so test it early with a real stream. Our streams carry events described by JSONSchema, and those schemas live in our own repositories (primary, secondary). Tools of this kind generally expect Avro-encoded events and a Confluent schema registry, which we do not run. A candidate that cannot follow our arrangement will need code of our own, which removes much of the point of adopting it.

Does it keep up with webrequest? This is by far our largest stream. Establish the real peak rate and test against it, rather than relying on general claims about scale.

Streaming or batch? As noted in review, a streaming engine might best be used in streaming mode, and if we would rather the work ran in batches then a batch engine is the better choice. Judge each candidate in the mode we would actually run it.

What does it guarantee when things go wrong? Ask what happens when a job restarts or a task fails: whether records can be lost or duplicated, how late-arriving data is treated, and how the files it writes are partitioned and later compacted.

Could it replace the Refine jobs too? Today Gobblin writes raw data and Spark Refine jobs turn it into tables. A tool that writes usable Iceberg tables directly would remove that second step. Worth assessing, while recognising that it widens the change.

Does it write Iceberg tables, and suit our catalog? The catalog is being chosen in T435604: [Tech evaluation] Table catalog options for the S3 data lake, so check each candidate against the likely outcomes rather than assuming one.

Can it get storage credentials the way the rest of the platform will? The new storage issues short-lived keys in exchange for a workload's identity token. A tool that can only use a stored access key is not disqualified, but it needs recording as an exception.

Is every part of it free software? Check this component by component rather than project by project. That is precisely how the earlier Kafka Connect attempt came unstuck.

Technical notes

• Judge connectors, not projects. The parts that matter here are the Kafka reader and the Iceberg writer.
• We might like to prototype the shortlist in codfw against the Ceph cluster, so the S3 write path and the credentials are exercised for real.
• We already run the Flink operator and a stream application on dse-k8s, so a Flink-based answer is the least new to us, while SeaTunnel or Kafka Connect would each add a system to operate.
• Review the requirements discusssed in the DPE deep dive, "What do we need from a Gobblin replacement?", linked from T435489: Data Lake ingestion after Hadoop: decide the future of Sqoop and Gobblin.

Acceptance criteria

• Each candidate is assessed against the questions above, with the results in a comparison document linked from this task.
• Event schema handling is tested with a real stream and the result recorded.
• A recommendation is agreed with Data Engineering, together with the follow-on tasks it implies.