tinsaetadesse/data-bus

By tinsaetadesse

•Updated 5 days ago

A lightweight, secure, cloud-native Apache Kafka container image for production deployments.

Image

Message queues

1.9K

Overview Tags

tinsaetadesse/data-bus repository overview

⁠Data Bus Container Image

⁠Introduction

Data Bus is a lightweight, production-ready Apache Kafka container image optimized for secure, cloud-native deployments. The image runs in KRaft mode by default, eliminating the need for ZooKeeper, and supports both AMD64 and ARM64 architectures. It provides a secure listener model, mutual TLS (mTLS) authentication, and configurable runtime properties.

⁠Features

The Data Bus image comes with the below features that make it standout.

KRaft-based Apache Kafka (ZooKeeper-free)
Multi-architecture images (AMD64 and ARM64)
Secure SSL/TLS client communication
Mutual TLS (mTLS) authentication
Configurable runtime using environment variables
Production-ready container layout
Compatible with Docker and Podman

⁠Security Model

The Data Bus image adopts a layered security model that separates internal cluster communication from external client access while supporting encrypted client connectivity through SSL/TLS and mutual TLS authentication.

⁠SSL/TLS Encryption

All producer, consumer, and administrative client traffic can be encrypted using SSL/TLS.

⁠Mutual TLS (mTLS)

The image supports client certificate authentication. This allows only trusted clients possessing valid certificates to connect to the cluster.

ssl.client.auth=required

⁠Listener Architecture

The image separates internal cluster communication from external client communication.

Listener	Purpose	Protocol
BROKER	Broker-to-broker communication	PLAINTEXT
CONTROLLER	KRaft controller quorum	PLAINTEXT
CLIENT	Producers, Consumers, AdminClient	SSL

Internal broker replication and KRaft controller traffic remain isolated from client traffic. Only the CLIENT listener is intended for producer, consumer and administrative connections.

⁠Certificates & Secrets

Data Bus brokers require certificates and credential files to establish encrypted connections and authenticate clients. For this reason, the directory containing these files should be mounted to each broker's volume.

volumes:
      - ./secrets:/etc/t-soft/data-bus/secrets:ro,z

⁠Supported Architectures

Architecture	Platform
x86_64	linux/amd64
ARM64	linux/arm64/v8

⁠Container Layout

The image exposes a stable internal filesystem contract.

Path	Description
`/usr/local/t-soft/data-bus`	Application home
`/etc/t-soft/data-bus`	Configuration files
`/data/t-soft/data-bus`	Persistent data storage
`/var/log/t-soft/data-bus`	Application logs

⁠Persistent Volumes

The image defines the following persistent storage volumes:

Volume	Purpose
`/data/t-soft/data-bus`	Data-bus data
`/var/log/t-soft/data-bus`	Data-bus logs
`/etc/t-soft/data-bus/secrets`	Files used for SSL/mTLS

Example:

volumes:
    - dbus-data:/data/t-soft/data-bus
    - dbus-log:/var/log/t-soft/data-bus
    - ./secrets:/etc/t-soft/data-bus/secrets:ro,z

⁠Runtime Configuration

All Kafka broker properties used in the Data Bus image can be configured using environment variables prefixed with KAFKA_. During container startup, these variables are translated into the corresponding Kafka configuration properties.

Example:

environment:
    ...
    KAFKA_LOG_DIRS: /custom/path/kraft-logs

Security-related environment variables include:

KAFKA_SSL_KEYSTORE_LOCATION
KAFKA_SSL_KEYSTORE_CREDENTIALS
KAFKA_SSL_KEY_CREDENTIALS
KAFKA_SSL_TRUSTSTORE_LOCATION
KAFKA_SSL_TRUSTSTORE_CREDENTIALS
KAFKA_SSL_CLIENT_AUTH

⁠Deployment Example

The compose script below is an example template you can use to quickly spin-up controller and broker containers.

services:
  controller-1:
    image: docker.io/tinsaetadesse/data-bus:latest
    container_name: controller-1
    hostname: controller-1.tsoft.internal
    environment:
      KAFKA_NODE_ID: 1
      # This instance acts as a controller
      KAFKA_PROCESS_ROLES: 'controller'
      # This broker uses BROKER, and CONTROLLER as listener names 
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'BROKER'
      # This instance listens through one (1) port for connecting with brokers
      KAFKA_LISTENERS: 'CONTROLLER://0.0.0.0:9093'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '[email protected]:9093,[email protected]:9093,[email protected]:9093'
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    networks:
      my-network:
        aliases:
          - controller-1.tsoft.internal

  controller-2:
    image: docker.io/tinsaetadesse/data-bus:latest
    container_name: controller-2
    hostname: controller-2.tsoft.internal
    environment:
      KAFKA_NODE_ID: 2
      # This instance acts as a controller
      KAFKA_PROCESS_ROLES: 'controller'
      # This broker uses BROKER, and CONTROLLER as listener names 
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'BROKER'
      # This instance listens through one (1) port for connecting with brokers
      KAFKA_LISTENERS: 'CONTROLLER://0.0.0.0:9093'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '[email protected]:9093,[email protected]:9093,[email protected]:9093'
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    networks:
      my-network:
        aliases:
          - controller-2.tsoft.internal

  controller-3:
    image: docker.io/tinsaetadesse/data-bus:latest
    container_name: controller-3
    hostname: controller-3.tsoft.internal
    environment:
      KAFKA_NODE_ID: 3
      # This instance acts as a controller
      KAFKA_PROCESS_ROLES: 'controller'
      # This broker uses BROKER, and CONTROLLER as listener names 
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'BROKER'
      # This instance listens through one (1) port for connecting with brokers
      KAFKA_LISTENERS: 'CONTROLLER://0.0.0.0:9093'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '[email protected]:9093,[email protected]:9093,[email protected]:9093'
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    networks:
      my-network:
        aliases:
          - controller-3.tsoft.internal

  broker-1:
    image: docker.io/tinsaetadesse/data-bus:latest
    container_name: broker-1
    hostname: broker-1.tsoft.internal
    ports:
      # Listner ports (port 19092 on the host is mapped to 9192 of the container port)
      - '19092:9192'
    environment:
      KAFKA_NODE_ID: 4
      # This instance acts as a broker
      KAFKA_PROCESS_ROLES: 'broker'
      # This broker uses BROKER, CONTROLLER, and CLIENT as listener names 
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT,CLIENT:SSL'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'BROKER'
      # This instance listens through two (2) ports for connecting with brokers, and external clients(producer/consumer)
      KAFKA_LISTENERS: 'BROKER://0.0.0.0:9092,CLIENT://0.0.0.0:9192'
      # This broker exposes it's container name for internal communications, and 
      # 'localhost' for connections from clients(producer/consumer) running on the host
      KAFKA_ADVERTISED_LISTENERS: 'BROKER://broker-1.tsoft.internal:9092,CLIENT://broker-1.tsoft.internal:9192'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '[email protected]:9093,[email protected]:9093,[email protected]:9093'
      KAFKA_SSL_KEYSTORE_LOCATION: '/etc/t-soft/data-bus/secrets/star.keystore.jks'
      KAFKA_SSL_KEYSTORE_CREDENTIALS: 'ssl.keystore.credential'
      KAFKA_SSL_KEY_CREDENTIALS: 'ssl.key.credential'
      KAFKA_SSL_TRUSTSTORE_LOCATION: '/etc/t-soft/data-bus/secrets/star.truststore.jks'
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: 'ssl.truststore.credential'
      KAFKA_SSL_CLIENT_AUTH: 'required'
      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: 'HTTPS'
      KAFKA_CLIENT_BOOTSTRAP_SERVERS: 'broker-1.tsoft.internal:9192,broker-2.tsoft.internal:9192,broker-3.tsoft.internal:9192'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
    volumes:
      - ./include/secrets:/etc/t-soft/data-bus/secrets:ro,z
    depends_on:
      - controller-1
      - controller-2
      - controller-3
    networks:
      my-network:
        aliases:
          - broker-1.tsoft.internal

  broker-2:
    image: docker.io/tinsaetadesse/data-bus:latest
    container_name: broker-2
    hostname: broker-2.tsoft.internal
    ports:
      # Listner ports (port 29092 on the host is mapped to 9192 of the container port)
      - '29092:9192'
    environment:
      KAFKA_NODE_ID: 5
      # This instance acts as a broker
      KAFKA_PROCESS_ROLES: 'broker'
      # This broker uses BROKER, CONTROLLER, and CLIENT as listener names 
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT,CLIENT:SSL'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'BROKER'
      # This instance listens through two (2) ports for connecting with brokers, and external clients(producer/consumer)
      KAFKA_LISTENERS: 'BROKER://0.0.0.0:9092,CLIENT://0.0.0.0:9192'
      # This broker exposes it's container name for internal communications, and 
      # 'localhost' for connections from clients(producer/consumer) running on the host
      KAFKA_ADVERTISED_LISTENERS: 'BROKER://broker-2.tsoft.internal:9092,CLIENT://broker-2.tsoft.internal:9192'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '[email protected]:9093,[email protected]:9093,[email protected]:9093'
      KAFKA_SSL_KEYSTORE_LOCATION: '/etc/t-soft/data-bus/secrets/star.keystore.jks'
      KAFKA_SSL_KEYSTORE_CREDENTIALS: 'ssl.keystore.credential'
      KAFKA_SSL_KEY_CREDENTIALS: 'ssl.key.credential'
      KAFKA_SSL_TRUSTSTORE_LOCATION: '/etc/t-soft/data-bus/secrets/star.truststore.jks'
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: 'ssl.truststore.credential'
      KAFKA_SSL_CLIENT_AUTH: 'required'
      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: 'HTTPS'
      KAFKA_CLIENT_BOOTSTRAP_SERVERS: 'broker-1.tsoft.internal:9192,broker-2.tsoft.internal:9192,broker-3.tsoft.internal:9192'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
    volumes:
      - ./include/secrets:/etc/t-soft/data-bus/secrets:ro,z
    depends_on:
      - controller-1
      - controller-2
      - controller-3
    networks:
      my-network:
        aliases:
          - broker-2.tsoft.internal

  broker-3:
    image: docker.io/tinsaetadesse/data-bus:latest
    container_name: broker-3
    hostname: broker-3.tsoft.internal
    ports:
      # Listner ports (port 39092 on the host is mapped to 9192 of the container port)
      - '39092:9192'
    environment:
      KAFKA_NODE_ID: 6
      # This instance acts as a broker
      KAFKA_PROCESS_ROLES: 'broker'
      # This broker uses BROKER, CONTROLLER, and CLIENT as listener names 
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT,CLIENT:SSL'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'BROKER'
      # This instance listens through two (2) ports for connecting with brokers, and external clients(producer/consumer)
      KAFKA_LISTENERS: 'BROKER://0.0.0.0:9092,CLIENT://0.0.0.0:9192'
      # This broker exposes it's container name for internal communications, and 
      # 'localhost' for connections from clients(producer/consumer) running on the host
      KAFKA_ADVERTISED_LISTENERS: 'BROKER://broker-3.tsoft.internal:9092,CLIENT://broker-3.tsoft.internal:9192'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '[email protected]:9093,[email protected]:9093,[email protected]:9093'
      KAFKA_SSL_KEYSTORE_LOCATION: '/etc/t-soft/data-bus/secrets/star.keystore.jks'
      KAFKA_SSL_KEYSTORE_CREDENTIALS: 'ssl.keystore.credential'
      KAFKA_SSL_KEY_CREDENTIALS: 'ssl.key.credential'
      KAFKA_SSL_TRUSTSTORE_LOCATION: '/etc/t-soft/data-bus/secrets/star.truststore.jks'
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: 'ssl.truststore.credential'
      KAFKA_SSL_CLIENT_AUTH: 'required'
      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: 'HTTPS'
      KAFKA_CLIENT_BOOTSTRAP_SERVERS: 'broker-1.tsoft.internal:9192,broker-2.tsoft.internal:9192,broker-3.tsoft.internal:9192'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
    volumes:
      - ./include/secrets:/etc/t-soft/data-bus/secrets:ro,z
    depends_on:
      - controller-1
      - controller-2
      - controller-3
    networks:
      my-network:
        aliases:
          - broker-3.tsoft.internal

networks:
  my-network:
    external: true

⁠Example SSL Client Configuration

The properties file below is an example template you can use to quickly configure Producers, Consumers, and AdminClients.

security.protocol=SSL

ssl.truststore.location=/path/client.truststore.jks
ssl.truststore.password=<<your-password>>

ssl.keystore.location=/path/client.keystore.jks
ssl.keystore.password=<<your-password>>

ssl.key.password=<<your-password>>

⁠Production Recommendations

For production deployments:

Enable SSL for all client connections.
Require mutual TLS authentication.
Store certificates and passwords as container secrets.
Rotate certificates regularly.
Monitor certificate expiration.

⁠Troubleshooting

⁠Listener configuration

If clients can reach the bootstrap server but fail to obtain metadata, verify that advertised.listeners contains addresses reachable from the client environment.

⁠SSL handshake failures

Verify:

Truststore location
Keystore location
Password credential files
Client certificates
Listener configuration

Tag summary

Recent tags

Content type

Image

Digest

sha256:8e2c44071…

Size

320.7 MB

Last updated

5 days ago

docker pull tinsaetadesse/data-bus

Help

System theme

Docker Suite