Apache Kafka Cluster Setup for Web Applications

Setting up an Apache Kafka Cluster for a Web Application

Imagine your web application processes thousands of orders per minute. Every event — sending an email, updating a search index, syncing data — happens synchronously. Servers choke, latency grows, and failures cause data loss. Apache Kafka solves these problems, but incorrect cluster configuration (e.g., single broker with replication factor = 1) leads to disaster. One of our clients — an e-commerce platform handling 500,000 events per second — migrated to Kafka after RabbitMQ failed under peak load.

Kafka is a distributed log with ordering and replication guarantees. It decouples services, provides an audit log, and enables real-time analytics. A production cluster requires careful setup: mode selection, broker configuration, security, and monitoring. Our team has deployed over 30 clusters for highload projects — from fintech to e-commerce. Proper configuration can reduce infrastructure costs by up to 40% compared to alternatives, with payback within six months.

How to Set Up an Apache Kafka Cluster: KRaft or ZooKeeper?

For new projects, use only KRaft (no ZooKeeper). Since KRaft became production-ready in version 3.3, it simplifies architecture, reduces failure points, and speeds up recovery. ZooKeeper remains for legacy setups, but we do not recommend starting with it.

Source: Kafka official documentation

Installation and Configuration on Ubuntu

Install the latest stable version, set up the systemd service, and initialize storage:

apt install -y openjdk-21-jdk-headless KAFKA_VERSION=3.7.0 SCALA_VERSION=2.13 wget https://downloads.apache.org/kafka/${KAFKA_VERSION}/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz tar -xzf kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt/ ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} /opt/kafka useradd -r -s /bin/false kafka chown -R kafka:kafka /opt/kafka mkdir -p /var/log/kafka /data/kafka chown kafka:kafka /var/log/kafka /data/kafka cat > /etc/systemd/system/kafka.service << 'EOF' [Unit] Description=Apache Kafka After=network.target [Service] Type=simple User=kafka Environment="KAFKA_HEAP_OPTS=-Xmx4g -Xms4g" Environment="KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true" ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties ExecStop=/opt/kafka/bin/kafka-server-stop.sh Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF CLUSTER_UUID=$(kafka-storage.sh random-uuid) kafka-storage.sh format -t $CLUSTER_UUID -c /opt/kafka/config/kraft/server.properties 

To tailor the configuration to your project, contact us.

KRaft configuration on each node (example for node 1; for nodes 2 and 3 change node.id and advertised.listeners):

node.id=1 process.roles=broker,controller controller.quorum.voters=1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093 listeners=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 advertised.listeners=PLAINTEXT://kafka-1.internal:9092 inter.broker.listener.name=PLAINTEXT controller.listener.names=CONTROLLER listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT log.dirs=/data/kafka num.recovery.threads.per.data.dir=4 num.io.threads=16 num.network.threads=8 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 socket.request.max.bytes=104857600 default.replication.factor=3 min.insync.replicas=2 num.partitions=6 offsets.topic.replication.factor=3 transaction.state.log.replication.factor=3 transaction.state.log.min.isr=2 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 compression.type=lz4 

Configuring TLS Between Brokers

Without TLS, traffic is transmitted in plain text. For protection, use your own CA and certificates for each broker. Detailed instructions for generating certificates will be provided as part of the project.

Add to server.properties on each broker:

listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9094,CONTROLLER://0.0.0.0:9093 ssl.keystore.location=/etc/kafka/ssl/kafka-1.keystore.jks ssl.keystore.password=changeit ssl.key.password=changeit ssl.truststore.location=/etc/kafka/ssl/kafka.truststore.jks ssl.truststore.password=changeit ssl.client.auth=required ssl.enabled.protocols=TLSv1.3,TLSv1.2 

Monitoring: Key Metrics and Alerts

Track under-replicated partitions, controller activity, p99 producer/consumer latency, consumer lag. Use JMX Exporter + Prometheus for collection, Grafana for visualization. Example JMX Exporter configuration:

startDelaySeconds: 0 hostPort: 127.0.0.1:9999 lowercaseOutputName: true rules: - pattern: kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec><>OneMinuteRate name: kafka_server_broker_topic_messages_in_per_sec - pattern: kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value name: kafka_server_under_replicated_partitions - pattern: kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value name: kafka_controller_active_count - pattern: kafka.network<type=RequestMetrics, name=TotalTimeMs, request=Produce><>99thPercentile name: kafka_network_produce_total_time_ms_p99 

Key alerts: kafka_server_under_replicated_partitions > 0, kafka_controller_active_count != 1, consumer lag above threshold. Contact us to implement monitoring and get ready-made dashboards for your load.

How to Ensure Fault Tolerance?

1. Create a test topic with replication factor 3.
2. Shut down one broker and verify that producers do not lose data (with acks=all and min.insync.replicas=2).
3. Bring the broker back and ensure replicas synchronize.
4. Check consumer lag — it should return to zero.
5. Repeat for each broker.

kafka-topics.sh --bootstrap-server kafka-1:9092 --create --topic test-topic --partitions 6 --replication-factor 3 kafka-producer-perf-test.sh --topic test-topic --num-records 1000000 --record-size 1024 --throughput -1 --producer-props bootstrap.servers=kafka-1:9092,kafka-2:9092,kafka-3:9092 acks=all compression.type=lz4 kafka-consumer-perf-test.sh --bootstrap-server kafka-1:9092 --topic test-topic --messages 1000000 --group perf-test-group 

We guarantee SLA 99.99% for your cluster. Order deployment — contact us for a consultation. Get a preliminary cost and timeline estimate.

Process and Timelines

Total: 5-7 working days for a basic cluster. Cost is calculated individually.

What's Included

• Cluster architecture design (roles, partitions, replication)
• Installation and configuration in KRaft mode
• TLS configuration between brokers and clients
• Monitoring implementation (JMX Exporter + Prometheus + Grafana)
• Creation of production topics with optimal parameters
• Load testing and fault tolerance verification
• Documentation (cluster diagram, instructions, runbook)
• Team training (basic administration, CLI usage)
• Technical support for 2 weeks after launch

Results and Guarantees

Over 30 deployed clusters with loads up to 1 million messages per second. Average infrastructure cost reduction — 35%. Payback time — up to 6 months. Contact our engineers — we will help configure a cluster for your load.