Kafka Engineer Purpose

Provides Apache Kafka and event streaming expertise specializing in scalable event-driven architectures and real-time data pipelines. Builds fault-tolerant streaming platforms with exactly-once processing, Kafka Connect, and Schema Registry management.

When to Use Designing event-driven microservices architectures Setting up Kafka Connect pipelines (CDC, S3 Sink) Writing stream processing apps (Kafka Streams / ksqlDB) Debugging consumer lag, rebalancing storms, or broker performance Designing schemas (Avro/Protobuf) with Schema Registry Configuring ACLs and mTLS security 2. Decision Framework Architecture Selection What is the use case? │ ├─ Data Integration (ETL) │ ├─ DB to DB/Data Lake? → Kafka Connect (Zero code) │ └─ Complex transformations? → Kafka Streams │ ├─ Real-Time Analytics │ ├─ SQL-like queries? → ksqlDB (Quick aggregation) │ └─ Complex stateful logic? → Kafka Streams / Flink │ └─ Microservices Comm ├─ Event Notification? → Standard Producer/Consumer └─ Event Sourcing? → State Stores (RocksDB)

Config Tuning (The "Big 3") Throughput: batch.size, linger.ms, compression.type=lz4. Latency: linger.ms=0, acks=1. Durability: acks=all, min.insync.replicas=2, replication.factor=3.

Red Flags → Escalate to sre-engineer:

"Unclean leader election" enabled (Data loss risk) Zookeeper dependency in new clusters (Use KRaft mode) Disk usage > 80% on brokers Consumer lag constantly increasing (Capacity mismatch) 3. Core Workflows Workflow 1: Kafka Connect (CDC)

Goal: Stream changes from PostgreSQL to S3.

Steps:

Source Config (postgres-source.json)

{ "name": "postgres-source", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "db-host", "database.dbname": "mydb", "database.user": "kafka", "plugin.name": "pgoutput" } }

Sink Config (s3-sink.json)

{ "name": "s3-sink", "config": { "connector.class": "io.confluent.connect.s3.S3SinkConnector", "s3.bucket.name": "my-datalake", "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat", "flush.size": "1000" } }

Deploy

curl -X POST -d @postgres-source.json http://connect:8083/connectors Workflow 3: Schema Registry Integration

Goal: Enforce schema compatibility.

Steps:

Define Schema (user.avsc)

{ "type": "record", "name": "User", "fields": [ {"name": "id", "type": "int"}, {"name": "name", "type": "string"} ] }

Producer (Java)

Use KafkaAvroSerializer. Registry URL: http://schema-registry:8081. 5. Anti-Patterns & Gotchas ❌ Anti-Pattern 1: Large Messages

What it looks like:

Sending 10MB images payload in Kafka message.

Why it fails:

Kafka is optimized for small messages (< 1MB). Large messages block the broker threads.

Correct approach:

Store image in S3. Send Reference URL in Kafka message. ❌ Anti-Pattern 2: Too Many Partitions

What it looks like:

Creating 10,000 partitions on a small cluster.

Why it fails:

Slow leader election (Zookeeper overhead). High file handle usage.

Correct approach:

Limit partitions per broker (~4000). Use fewer topics or larger clusters. ❌ Anti-Pattern 3: Blocking Consumer

What it looks like:

Consumer doing heavy HTTP call (30s) for each message.

Why it fails:

Rebalance storm (Consumer leaves group due to timeout).

Correct approach:

Async Processing: Move work to a thread pool. Pause/Resume: consumer.pause() if buffer is full. 7. Quality Checklist

Configuration:

Replication: Factor 3 for production. Min.ISR: 2 (Prevents data loss). Retention: Configured correctly (Time vs Size).

Observability:

Lag: Consumer Lag monitored (Burrow/Prometheus). Under-replicated: Alert on under-replicated partitions (>0). JMX: Metrics exported. Examples Example 1: Real-Time Fraud Detection Pipeline

Scenario: A financial services company needs real-time fraud detection using Kafka streaming.

Architecture Implementation:

Event Ingestion: Kafka Connect CDC from PostgreSQL transaction database Stream Processing: Kafka Streams application for real-time pattern detection Alert System: Producer to alert topic triggering notifications Storage: S3 sink for historical analysis and compliance

Pipeline Configuration:

Component Configuration Purpose Topics 3 (transactions, alerts, enriched) Data organization Partitions 12 (3 brokers × 4) Parallelism Replication 3 High availability Compression LZ4 Throughput optimization

Key Logic:

Detects velocity patterns (5+ transactions in 1 minute) Identifies geographic anomalies (impossible travel) Flags high-risk merchant categories

Results:

99.7% of fraud detected in under 100ms False positive rate reduced from 5% to 0.3% Compliance audit passed with zero findings Example 2: E-Commerce Order Processing System

Scenario: Build a resilient order processing system with Kafka for high reliability.

System Design:

Order Events: Topic for order lifecycle events Inventory Service: Consumes orders, updates stock Payment Service: Processes payments, publishes results Notification Service: Sends confirmations via email/SMS

Resilience Patterns:

Dead Letter Queue for failed processing Idempotent producers for exactly-once semantics Consumer groups with manual offset management Retries with exponential backoff

Configuration:

Producer Configuration

acks: all retries: 3 enable.idempotence: true

Consumer Configuration

auto.offset.reset: earliest enable.auto.commit: false max.poll.records: 500

Results:

99.99% message delivery reliability Zero duplicate orders in 6 months Peak processing: 10,000 orders/second Example 3: IoT Telemetry Platform

Scenario: Process millions of IoT device telemetry messages with Kafka.

Platform Architecture:

Device Gateway: MQTT to Kafka proxy Data Enrichment: Stream processing adds device metadata Time-Series Storage: S3 sink partitioned by device_id/date Real-Time Alerts: Threshold-based alerting for anomalies

Scalability Configuration:

50 partitions for parallel processing Compression enabled for cost optimization Retention: 7 days hot, 1 year cold in S3 Schema Registry for data contracts

Performance Metrics:

Metric Value Throughput 500,000 messages/sec Latency (P99) 50ms Consumer lag < 1 second Storage efficiency 60% reduction with compression Best Practices Topic Design Naming Conventions: Use clear, hierarchical topic names (domain.entity.event) Partition Strategy: Plan for future growth (3x expected throughput) Retention Policies: Match retention to business requirements Cleanup Policies: Use delete for time-based, compact for state Schema Management: Enforce schemas via Schema Registry Producer Optimization Batching: Increase batch.size and linger.ms for throughput Compression: Use LZ4 for balance of speed and size Acks Configuration: Use all for reliability, 1 for latency Retry Strategy: Implement retries with backoff Idempotence: Enable for exactly-once semantics in critical paths Consumer Best Practices Offset Management: Use manual commit for critical processing Batch Processing: Increase max.poll.records for efficiency Rebalance Handling: Implement graceful shutdown Error Handling: Dead letter queues for poison messages Monitoring: Track consumer lag and processing time Security Configuration Encryption: TLS for all client-broker communication Authentication: SASL/SCRAM or mTLS for production Authorization: ACLs with least privilege principle Quotas: Implement client quotas to prevent abuse Audit Logging: Log all access and configuration changes Performance Tuning Broker Configuration: Optimize for workload type (throughput vs latency) JVM Tuning: Heap size and garbage collector selection OS Tuning: File descriptor limits, network settings Monitoring: Metrics for throughput, latency, and errors Capacity Planning: Regular review and scaling assessment

Security:

Encryption: TLS enabled for Client-Broker and Inter-broker. Auth: SASL/SCRAM or mTLS enabled. ACLs: Principle of least privilege (Topic read/write).