Message Queue Setup: Apache Kafka for Microservices

Our Apache Kafka setup for microservices integrates the Kafka message queue into your architecture, handling over 100,000 messages per second with zero data loss. In one project, a client lost orders due to setting acks=0; after migrating to acks=all with enable.idempotence=true, delivery became guaranteed and event order preserved. Each Kafka consumer group processes a subset of partitions to ensure load distribution. We introduce Schema Registry for message version control, allowing different services to evolve safely. Our setup, tailored for your project, is delivered in 3–5 days with full documentation.

Why Kafka Over RabbitMQ for Stream Processing

Kafka stores messages by retention policy (e.g., 7 days or 100 GB), enabling multiple consumers to replay from different offsets. RabbitMQ deletes messages after acknowledgment — good for task queues, not for auditing or replay. Kafka is 10 times better than RabbitMQ for throughput, and 3 times cheaper for large data volumes.

According to Apache Kafka documentation, Kafka's throughput reaches 2–3 million messages per second on a 3-node cluster; RabbitMQ tops at 300–500 thousand. Default retention is 7 days, but we adapt to business logic: e.g., 30 days for audits, 100 GB for logs.

How We Configure Kafka: Stack, Configs, Process

We use Confluent Kafka 7.6+ with mandatory Schema Registry. For PHP we use the PHP rdkafka producer library (librdkafka); for Node.js, the Node.js Kafka client (kafkajs). Process:

1. Load analysis and Kafka topics and partitions design (partitions, replication factor).
2. Deployment via Kafka Docker compose configuration (Docker Compose) or Kubernetes (Strimzi).
3. Producer configuration with idempotent and acks=all, plus max.in.flight.requests.per.connection=1 to preserve message order.
4. Implementation of consumers with manual commit after processing.
5. Monitoring via JMX + Grafana with alerts at lag > 10,000.

Docker Setup

# docker-compose.yml services: zookeeper: image: confluentinc/cp-zookeeper:7.6.0 environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 volumes: - zookeeper_data:/var/lib/zookeeper/data - zookeeper_log:/var/lib/zookeeper/log kafka: image: confluentinc/cp-kafka:7.6.0 depends_on: [zookeeper] environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false" KAFKA_LOG_RETENTION_HOURS: 168 # 7 days KAFKA_LOG_RETENTION_BYTES: 107374182400 # 100 GB KAFKA_NUM_PARTITIONS: 6 KAFKA_DEFAULT_REPLICATION_FACTOR: 1 volumes: - kafka_data:/var/lib/kafka/data ports: - "9092:9092" kafka-ui: image: provectuslabs/kafka-ui:latest depends_on: [kafka] environment: KAFKA_CLUSTERS_0_NAME: local KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092 ports: - "8080:8080" volumes: zookeeper_data: zookeeper_log: kafka_data: 

Topic Creation

# Create a topic with 6 partitions and replication factor 1 (for single broker) kafka-topics.sh --bootstrap-server kafka:9092 \ --create \ --topic user-events \ --partitions 6 \ --replication-factor 1 \ --config retention.ms=604800000 \ --config cleanup.policy=delete # View describe --topic user-events 

PHP: Producer (librdkafka)

use RdKafka\Producer; use RdKafka\Conf; class KafkaProducer { private Producer $producer; public function __construct() { $conf = new Conf(); $conf->set('bootstrap.servers', config('kafka.brokers')); $conf->set('security.protocol', 'PLAINTEXT'); $conf->set('acks', 'all'); $conf->set('retries', '3'); $conf->set('enable.idempotence', 'true'); $conf->set('compression.type', 'snappy'); $conf->setDrMsgCb(function ($kafka, $message) { if ($message->err !== RD_KAFKA_RESP_ERR_NO_ERROR) { Log::error('Kafka delivery failed', [ 'error' => $message->errstr(), 'topic' => $message->topic_name, ]); } }); $this->producer = new Producer($conf); } public function publish(string $topic, string $key, array $payload): void { $rdTopic = $this->producer->newTopic($topic); $rdTopic->produce( partition: RD_KAFKA_PARTITION_UA, msgflags: 0, payload: json_encode($payload), key: $key, ); $this->producer->poll(0); } public function flush(): void { $result = $this->producer->flush(10000); if (RD_KAFKA_RESP_ERR_NO_ERROR !== $result) { throw new \RuntimeException('Kafka flush failed: ' . rd_kafka_err2str($result)); } } } // Usage $producer->publish('user-events', (string) $user->id, [ 'event' => 'user.registered', 'user_id' => $user->id, 'email' => $user->email, 'timestamp' => now()->toIso8601String(), ]); $producer->flush(); 

Node.js: Producer and Consumer on kafkajs

import { Kafka, CompressionTypes } from 'kafkajs'; const kafka = new Kafka({ clientId: 'myapp-api', brokers: [process.env.KAFKA_BROKERS!], retry: { retries: 5, initialRetryTime: 300, factor: 0.2, }, }); // Producer const producer = kafka.producer({ allowAutoTopicCreation: false, idempotent: true, maxInFlightRequests: 5, }); await producer.connect(); await producer.send({ topic: 'user-events', compression: CompressionTypes.Snappy, messages: [{ key: String(userId), value: JSON.stringify({ event: 'user.login', userId, ip, timestamp: Date.now() }), headers: { 'content-type': 'application/json' }, }], }); // Consumer const consumer = kafka.consumer({ groupId: 'audit-service' }); await consumer.connect(); await consumer.subscribe({ topic: 'user-events', fromBeginning: false }); await consumer.run({ eachMessage: async ({ topic, partition, message }) => { const payload = JSON.parse(message.value!.toString()); await AuditLog.create({ event: payload.event, userId: payload.userId, metadata: payload, }); }, }); 

Achieving Idempotent Producer

Enable enable.idempotence=true and acks=all. We also configure max.in.flight.requests.per.connection=1 to ensure message ordering. The producer gets a unique ID, and the broker discards duplicates. This is mandatory for financial transactions and audits.

Monitoring Consumer Group Lag

Monitor consumer group lag using kafka-consumer-groups.sh --describe. For automation, deploy JMX Exporter feeding Prometheus, then build Grafana dashboards with alert threshold at lag > 10,000.

What’s Included in Turnkey Setup (Commercial Deliverables)

• Current architecture audit – load analysis, topic and partition design.
• Cluster deployment – Docker Compose or Kubernetes (Strimzi) with fault tolerance.
• Application integration – producers/consumers in PHP or Node.js with error handling and retry logic.
• Schema Registry – Avro schema implementation for version control.
• Monitoring – Grafana dashboard with alerts on lag and broker load.
• Team training – documentation, runbooks, and disaster recovery procedures.
• Post-launch support – 14 days of incident response and fine-tuning.

Our Metrics & Pricing

Trusted by 50+ satisfied clients with 5+ years of experience and 20+ successful queue projects. This ensures high reliability and cost efficiency. For instance, one client saves $15,000 annually on infrastructure costs after migrating from a managed service. Typical annual savings: $5,000–$20,000 compared to managed cloud services over 12 months. For a typical project, total cost ranges from $2,500 to $4,000 with average annual savings of $10,000.

Cost is calculated individually; a typical case saves clients 30–50% compared to managed services over 12 months. Get a free one-day consultation — we’ll assess your task and propose a solution.