Skip to content

Kafka

Kafka is a widely used event streaming platform. We recommend using Kafka if you have a lot of events and want to horizontally scale your Sensors. If you are looking to get started quickly with Argo Events we recommend using Jetstream instead.

When using a Kafka EventBus you must already have a Kafka cluster set up and topics created (unless you have auto create enabled, see topics below).

Example

kind: EventBus
metadata:
  name: default
spec:
  kafka:
    url: kafka:9092   # must be managed independently
    topic: "example"  # optional

See here for the full specification.

Properties

url

Comma seperated list of kafka broker urls, the kafka broker must be managed independently of Argo Events.

topic

The topic name, defaults to {namespace-name}-{eventbus-name}. Two additional topics per Sensor are also required, see see topics below for more information.

version

Kafka version, we recommend not manually setting this field in most circumstances. Defaults to the oldest supported stable version.

tls

Enables TLS on the kafka connection. 

tls:
  caCertSecret:
    name: my-secret
    key: ca-cert-key
  clientCertSecret:
    name: my-secret
    key: client-cert-key
  clientKeySecret:
    name: my-secret
    key: client-key-key

sasl

Enables SASL authentication on the kafka connection. 

sasl:
  mechanism: PLAIN
  passwordSecret:
    key: password
    name: my-user
  userSecret:
    key: user
    name: my-user

consumerGroup.groupName

Consumer group name, defaults to {namespace-name}-{sensor-name}.

consumerGroup.rebalanceStrategy

The kafka rebalance strategy, can be one of: sticky, roundrobin, range. Defaults to range.

consumerGroup.startOldest

When starting up a new group do we want to start from the oldest event (true) or the newest event (false). Defaults to false

Security

You can enable TLS or SASL authentication, see above for configuration details. You must enable these features in your Kafka Cluster and make the certifactes/credentials available in a Kubernetes secret.

Topics

The Kafka EventBus requires one event topic and two additional topics (trigger and action) per Sensor. These topics will not be created automatically unless the Kafka auto.create.topics.enable cluster configuration is set to true, otherwise it is your responsibility to create these topics. If a topic does not exist and cannot be automatically created, the EventSource and/or Sensor will exit with an error.

If you want to take advantage of the horizontal scaling enabled by the Kafka EventBus be sure to create topics with more than one partition.

By default the topics are named as follows.

topic name
event {namespace}-{eventbus-name}
trigger {namespace}-{eventbus-name}-{sensor-name}-trigger
action {namespace}-{eventbus-name}-{sensor-name}-action

If a topic name is specified in the EventBus specification, then the topics are named as follows.

topic name
event {spec.kafka.topic}
trigger {spec.kafka.topic}-{sensor-name}-trigger
action {spec.kafka.topic}-{sensor-name}-action

Horizontal Scaling and Leader Election

Sensors that use a Kafka EventBus can scale horizontally. Specifiying replicas greater than one will result in all Sensor pods actively processing events. However, an EventSource that uses a Kafka EventBus cannot necessarily be horizontally scaled in an active-active manner, see EventSource HA for more details. In an active-passive scenario a Kubernetes leader election is used.

Back to top