Okay, done a buncha learning and experiementing. Mostly I found it really difficult at first to understand the relationship between Kafka and Eventing, but I feel like I have a good idea now. I'll post my raw notes here, and then do some summarizing in the next comment.

• https://knative.tips/ is way easier to understand than knative.dev docs.

• Knative keeps track of deployment 'revisions', and because it autoscales, the default is just to always use the latest revision for handling traffic. Traffic to knative will automatically spin up pods with the latest deployment revision to serve the request. However, old deployment revisions still exist and are usable in knative, and even can have traffic configured to route to them. This allows for very easy canary deployments and rollbacks.

Eventing uses CloudEvents.

https://knative.tips/eventing/basics/cloud-events/
Is this interoperable with our Event Platform Schemas? If not, should we migrate?
Hm, I think it can be interoperable:

https://knative.tips/eventing/events/reading-external-events/

This would convert our events into CloudEvents as they inter the Eventing cluster(?). I believe if we use Kafka as the Eventing Broker, this would effectively wrap the events in the external topic in a CloudEvent and produce to an Eventing managed topic.
However, if the original events in our topics were CloudEvents, I think they could just be event Sources in the Broker already, and downstream Triggers,Sinks, etc. can use them as is.

Q: Channels vs Brokers?

Ah: Broker combines channel, reply, and filter functionality into a single resource
https://knative.tips/eventing/basics/broker/

Eventing Brokers using Kafka Brokers:

It looks like the Eventing Kafka Broker
https://knative.dev/docs/eventing/broker/kafka-broker/
Is more of an abstraction on top of kafka producers and consumers. By setting up an Eventing Kafka Broker, we just configure the Eventing resources (Sources, Sinks, Services, Triggers, etc.) to use the Kafka Broker as the backend. This means that Eventing producers and consumers don't actually know anything about Kafka.

Q: How does this work with existent topics work?

Okay, all existent Kafka topics are defined using Sources, KafkaSources.
https://knative.dev/docs/eventing/sources/kafka-source/

To be used in Knative Eventing, they have to be explicitly configured as Sources.

A Knative Broker can be a Sink. Knative KafkaBroker is just a broker but backed by a SINGLE KAFKA TOPIC. This means that in order to subscribe to Kafka topics (via Triggers, Sinks, etc) to events using a Knative KafkaBroker, all messages from any Kafka topics KafkaSources that target the KafkaBroker as a sink will go into ONE KAFKA TOPIC. Once there, All kinds of fancy Eventing stuff can be used, sort of as an 'event mesh'. Makes for really nice composable and simple HTTP only Services that don't need to know about how events are delivered.

However, this seems really annoying and difficult to scale. All messages from all existent topics would go into one Eventing KafkaBroker topic (in my tests this was named 'knative-broker-knative-eventing-default').

Q: How are topics created when used as an event sink?

They are specified. The default is to produce the 'structured' CloudEvent, with the original kafka messages in the 'data' field. E.g. I produced {"a": "1651248318"} to topic otto1 with a KafkaSource(topic:otto1) -> Eventing KafkaBroker, and got from a KafkaSink:

{
  "data": {
    "a": "1651248318"
  },
  "time": "2022-04-29T16:05:18.439Z",
  "subject": "partition:1#6",
  "type": "dev.knative.kafka.event",
  "source": "/apis/v1/namespaces/knative-eventing/kafkasources/kafka-source#otto1",
  "id": "partition:1/offset:6",
  "specversion": "1.0"
}

However, if I set contentMode: binary on the KafkaSink spec,the data value is produced as is, without any CE envelope.
https://knative.tips/eventing/events/cloudevents/

Q: Understanding how kafka fits in with Sources, Sinks, Triggers, Services, and Sequences.

I want to accomplish the following:

• consume Kafka topic A into service, transform, produce back to a Kafka topic B

Several ways to do this I think.

First without a Broker

Without a Broker, other Kafka Eventing Services/Flows (Sequences, Parallel) etc. have to explicitly define all Sources and Sinks they use.

Resources:

• KafkaSource topic A.
• KafkaSink topic B.
• Service for transformer code with HTTP POST endpoint that accepts CloudEvent and returns CloudEvent in HTTP response.
• Sequence with a reply to the KafkaSink: https://knative.dev/docs/eventing/flows/sequence/sequence-reply-to-event-display/ that wraps the Service and the KafkaSink together

KafkaSource topic A -> Sequence(steps[Service transformer], reply: KafkaSink topic B)

With a Broker

With a Broker, instead, Triggers are used to filter all events by type, and then respond to those and produce new events with different types back to the Broker. Those output events can then be used by other Eventing Services/resources, or in this example to a KafkaSink topic, to allow them to be used by things outside of Knative Eventing.

https://knative.dev/docs/eventing/flows/sequence/sequence-with-broker-trigger/
https://knative.tips/eventing/delivery-methods/broker-trigger-delivery/

• KafkaBroker broker (single interally managed topic for Eventing)
• KafkaSource topic A Input
• KafkaSink topic B Output
• Service for transformer code with HTTP POST endpoint that accepts CloudEvent and returns CloudEvent in HTTP response.
• Trigger to get events (filtered by input type) from Broker and reply to Broker (Triggers to a Service that replies with CloudEvent send event back to Broker).
• Trigger to get events (filtered by output type) from Broker to the KafkaSink

1. INPUT: KafkaSource topic A Input -> KafkaBroker type topic:A
2. PROCESS: Trigger (KafkaBroker (type topic:A) -> ServiceTransformer, reply: kafkaBroker (type topic:B) -> KafkaBroker internal topic.
3. OUTPUT: Trigger(KafkaBroker (type topic:B) -> KafkaSink topic B Output

Q: Event filtering: does this only work on CloudEvents fields?

I think so. But also:
https://knative.dev/development/eventing/experimental-features/new-trigger-filters/
https://knative.dev/development/eventing/experimental-features/new-trigger-filters/#cesql

So, filtering is only possible on CloudEvents fields, but you can make custom 'extension' CE/header fields and filter on those if needed. This is kind of like using structured Kafka message headers for filtering. The data value is totally opaque to Eventing, as far as I can tell, so you can only leverage data fields in custom Services with code. Can't rely on them for Eventing routing (unless you hoist them into a CloudEvents header).

Event registry?

https://knative.tips/eventing/events/event-registry/
https://knative.dev/docs/eventing/event-registry/

This makes it so you can more easily registry and discover CloudEvent types. Supposedly, any Source that uses a Broker as a sink automatically registers its eventtypes, but I did not experience this. I probably was just doing something wrong.

Q: Eventing and CEP? Stateful?

• CEP: no way.
• Stateful: Maybe, if your app deals with it, but it would have to be custom, so, no.

My opinion is that Knative Eventing looks really nice if you want to build a very self contained event driven application composed of many microservices. I wouldn't use it for 'stream processing' of any kind of high volume or complexity.

Using it to build a replacement of change-prop job queue might be quite nice. There are a lot of really nice scaling and observability features (e.g. event tracing).

I would not use Eventing to build generic streaming connector support. I would not use it for a Stream Data Platform (or a 'Shared Data Platform :p )'

Eventing looks nice for a very self contained application, but not for platform.

You know, Knative Eventing almost reminds me of ZeroMQ, except with k8s CRDs and HTTP for transport and the ability to use Kafka as a persistent message queue.