【Kafka】Docker compose で Kafka構築時に Topic を作るには

■ はじめに

https://dk521123.hatenablog.com/entry/2023/04/24/153846

において、Docker compose で Kafka の開発環境構築を行ったが
どうせなら、Topic 作成とか環境の初期処理を行えるようにしたいので
調べてみた。

目次

【１】Docker compose で Kafka構築時に Topic を作るには
　１）方法概要
【２】サンプル
　１）コマンド例

【１】Docker compose で Kafka構築時に Topic を作るには

https://stackoverflow.com/questions/64865361/docker-compose-create-kafka-topics

を参考にした。

１）方法概要

やり方としては、初期化用の container を用意して、
その entrypoint/command から、kafka-topics --create を実行する

The simplest way is to start a separate container
 inside the docker-compose file (called init-kafka in the example below)
to launch the various kafka-topics --create ... commands,
while first making it wait for Kafka to be reachble by simply running kafka-topics --list ...

一部抜粋

  init-kafka:
    image: confluentinc/cp-kafka:6.1.1
    depends_on:
      - kafka
    entrypoint: [ '/bin/sh', '-c' ]
    command: |
      "
      # blocks until kafka is reachable
      kafka-topics --bootstrap-server kafka:29092 --list

      echo -e 'Creating kafka topics'
      kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic my-topic-1 --replication-factor 1 --partitions 1
      kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic my-topic-2 --replication-factor 1 --partitions 1

      echo -e 'Successfully created the following topics:'
      kafka-topics --bootstrap-server kafka:29092 --list
      "

【２】サンプル

version: '3'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.2
    container_name: zookeeper
    hostname: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  broker:
    image: confluentinc/cp-kafka:7.3.2
    container_name: broker
    hostname: broker
    ports:
    # To learn about configuring Kafka for access across networks see
    # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
      - "9092:9092"
    expose:
      - '29092'
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_MIN_INSYNC_REPLICAS: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
  init-kafka:
    image: confluentinc/cp-kafka:7.3.2
    container_name: init-kafka
    depends_on:
      - broker
    entrypoint: [ '/bin/sh', '-c' ]
    command: |
      "
      # blocks until kafka is reachable
      kafka-topics --bootstrap-server broker:29092 --list

      echo -e 'Creating kafka topics'
      kafka-topics --bootstrap-server broker:29092 --create --if-not-exists --topic demo-topic-1 --replication-factor 1 --partitions 1
      kafka-topics --bootstrap-server broker:29092 --create --if-not-exists --topic demo-topic-2 --replication-factor 1 --partitions 1

      echo -e 'Successfully created the following topics:'
      kafka-topics --bootstrap-server broker:29092 --list
      "

１）コマンド例

# Step1: 立ち上げる
$ sudo docker compose up -d

# Step2: 確認

# 2-1) ログの確認
$ docker logs init-kafka > log.log
$ more log.log
# どうやらうまくいってるっぽい
~~~

Creating kafka topics
Created topic demo-topic-1.
Created topic demo-topic-2.
Successfully created the following topics:
~~~

# 2-2) 中身を直接確認
$ sudo docker exec -it broker bash
# kafka-topics --list で確認
kafka-topics --bootstrap-server broker:29092 --list
demo-topic-1
demo-topic-2
# できてた！

$ exit
exit

関連記事

Apache Kafka ～ 基礎知識編 ～
https://dk521123.hatenablog.com/entry/2023/04/26/103421
Apache Kafka ～ 入門編 ～
https://dk521123.hatenablog.com/entry/2023/05/01/000000
Apache Kafka ～ 環境構築 / Docker compose編 ～
https://dk521123.hatenablog.com/entry/2023/04/24/153846
Dockerfile / Docker Compose でシェルを実行する
https://dk521123.hatenablog.com/entry/2023/07/21/234509