Apache Kafka Expert Expert guidance for Apache Kafka, event streaming, Kafka Streams, and building event-driven architectures. Core Concepts Topics, partitions, and offsets Producers and consumers Consumer groups Kafka Streams Kafka Connect Exactly-once semantics Producer from kafka import KafkaProducer import json producer = KafkaProducer ( bootstrap_servers = [ 'localhost:9092' ] , value_serializer = lambda v : json . dumps ( v ) . encode ( 'utf-8' ) , acks = 'all' ,

Wait for all replicas

retries

3 )

Send message

future

producer . send ( 'user-events' , { 'user_id' : '123' , 'event' : 'login' , 'timestamp' : '2024-01-01T00:00:00Z' } )

Wait for acknowledgment

record_metadata

future . get ( timeout = 10 ) print ( f"Topic: { record_metadata . topic } , Partition: { record_metadata . partition } " ) producer . flush ( ) producer . close ( ) Consumer from kafka import KafkaConsumer consumer = KafkaConsumer ( 'user-events' , bootstrap_servers = [ 'localhost:9092' ] , group_id = 'my-group' , auto_offset_reset = 'earliest' , enable_auto_commit = False , value_deserializer = lambda m : json . loads ( m . decode ( 'utf-8' ) ) ) for message in consumer : print ( f"Received: { message . value } " )

Process message

process_event ( message . value )

Manual commit

consumer . commit ( ) Kafka Streams Properties props = new Properties ( ) ; props . put ( StreamsConfig . APPLICATION_ID_CONFIG , "streams-app" ) ; props . put ( StreamsConfig . BOOTSTRAP_SERVERS_CONFIG , "localhost:9092" ) ; StreamsBuilder builder = new StreamsBuilder ( ) ; KStream < String , String

source

builder . stream ( "input-topic" ) ; // Transform and filter KStream < String , String

transformed

source . filter ( ( key , value ) -> value . length ( )

10 ) . mapValues ( value -> value . toUpperCase ( ) ) ; transformed . to ( "output-topic" ) ; KafkaStreams streams = new KafkaStreams ( builder . build ( ) , props ) ; streams . start ( ) ; Best Practices Use appropriate partition keys Monitor consumer lag Implement idempotent producers Use consumer groups for scaling Set proper retention policies Handle rebalancing gracefully Monitor cluster metrics Anti-Patterns ❌ Single partition topics ❌ No error handling ❌ Ignoring consumer lag ❌ Producing to wrong partitions ❌ Not using consumer groups ❌ Synchronous processing ❌ No monitoring Resources Apache Kafka: https://kafka.apache.org/ Confluent Platform: https://www.confluent.io/