Skip to main content

Learning path

Apache Kafka learning path: everything to study, topic by topic

Master distributed event streaming with Apache Kafka. Learn core architecture, producer and consumer APIs, stream processing with Kafka Streams and KSQL, cluster operations, security, and performance optimization for building scalable real-time data pipelines.

Short answer

To learn Apache Kafka, work through 6 modules and 26 topics in the order they build on each other. It starts with Core Concepts & Architecture, then moves on to Producers & Client Config and Consumers & Consumer Groups. Use the map as a checklist, or measure your level and study only what is missing.

6
modules
26
topics
60
min to assess

What you will study

Topics are grouped by module. Open one to see its objectives and key concepts.

1 · Core Concepts & Architecture4 topics
  • Kafka Fundamentals

    Understand Kafka's core architecture: publish-subscribe model, distributed commit log, and role in event streaming ecosystems.

    Objectives

    • Explain Kafka's role in event streaming
    • Understand the publish-subscribe model
    • Identify Kafka use cases

    Key concepts

    • Event streaming
    • Publish-subscribe
    • Distributed log
    • Immutable records
  • Topics & Partitions

    Master topics, partitions, and how Kafka distributes data across partitions for parallelism.

    Objectives

    • Design topic structures
    • Understand partition distribution
    • Apply partition keys effectively

    Key concepts

    • Topics
    • Partitions
    • Partition keys
    • Ordering guarantees
  • Brokers & Cluster Architecture

    Learn how brokers form a cluster, leader election, and cluster coordination.

    Objectives

    • Understand broker roles
    • Explain cluster coordination
    • Identify single points of failure

    Key concepts

    • Brokers
    • Cluster
    • Leader election
    • Metadata
  • Replication & In-Sync Replicas

    Deep dive into replication strategy, in-sync replicas (ISR), and durability guarantees.

    Objectives

    • Configure replication
    • Monitor ISR health
    • Understand durability trade-offs

    Key concepts

    • Replication factor
    • ISR
    • Durability
    • Replica lag
2 · Producers & Client Config5 topics
  • Producer API & Usage

    Use KafkaProducer API to send records: synchronous and asynchronous patterns.

    Objectives

    • Send records with KafkaProducer
    • Handle send callbacks
    • Implement error handling

    Key concepts

    • KafkaProducer
    • send()
    • Callbacks
    • Async vs sync
  • Producer Configuration

    Master producer settings: acks, compression, buffer management, and delivery semantics.

    Objectives

    • Configure producer settings
    • Tune for throughput vs latency
    • Understand delivery semantics

    Key concepts

    • acks
    • compression
    • buffer.memory
    • delivery.timeout.ms
  • Serialization & Data Formats

    Choose and implement serializers: JSON, Avro, Protobuf. Schema management.

    Objectives

    • Implement custom serializers
    • Use schema registries
    • Compare data formats

    Key concepts

    • Serialization
    • Avro
    • Protobuf
    • Schema registry
  • Producer Reliability & Retries

    Ensure reliable delivery: retries, idempotence, transactional writes.

    Objectives

    • Configure idempotent producers
    • Implement transactional writes
    • Handle failures gracefully

    Key concepts

    • Retries
    • Idempotent producer
    • Transactions
    • Exactly-once semantics
  • Batching & Partitioning Strategy

    Optimize throughput via batching, partitioner selection, and partition key design.

    Objectives

    • Implement custom partitioners
    • Design partition keys
    • Avoid hot partition issues

    Key concepts

    • Batching
    • Partitioner
    • Partition selection
    • Hot partitions
3 · Consumers & Consumer Groups5 topics
  • Consumer API & Usage

    Implement KafkaConsumer: subscribe, poll, and message processing patterns.

    Objectives

    • Use KafkaConsumer API
    • Process records from topics
    • Handle consumer lifecycle

    Key concepts

    • KafkaConsumer
    • subscribe()
    • poll()
    • Event loop
  • Consumer Groups & Rebalancing

    Manage consumer groups, rebalancing mechanisms, and group coordination.

    Objectives

    • Design consumer groups
    • Understand rebalancing triggers
    • Minimize rebalancing impact

    Key concepts

    • Consumer group
    • Rebalancing
    • Group coordinator
    • Static membership
  • Offset Management & Commits

    Manage offsets: auto-commit vs manual, commit strategies, and consistency.

    Objectives

    • Implement offset commits
    • Choose commit strategies
    • Handle offset reset scenarios

    Key concepts

    • Offsets
    • Auto-commit
    • Manual commit
    • Offset reset
  • Consumer Lag & Monitoring

    Monitor consumer lag, detect bottlenecks, and optimize consumer throughput.

    Objectives

    • Calculate consumer lag
    • Monitor lag in production
    • Identify slow consumers

    Key concepts

    • Consumer lag
    • Lag metrics
    • Monitoring
    • Alerting
  • Consumer Configuration

    Tune consumer settings for throughput, latency, and reliability.

    Objectives

    • Configure consumer for use case
    • Tune for performance
    • Balance reliability and latency

    Key concepts

    • fetch.min.bytes
    • fetch.max.wait.ms
    • max.poll.records
    • session.timeout.ms
4 · Streams & KSQL5 topics
  • Kafka Streams Fundamentals

    Introduction to Kafka Streams: topology, DSL, and stream processing patterns.

    Objectives

    • Build basic topologies
    • Understand KStream vs KTable
    • Apply stream processing patterns

    Key concepts

    • Topology
    • DSL
    • KStream
    • KTable
    • GlobalKTable
  • Stream Topology & Operators

    Design complex topologies: filter, map, flatMap, branch, and joins.

    Objectives

    • Design complex topologies
    • Implement joins efficiently
    • Optimize processor chains

    Key concepts

    • Operators
    • KStream joins
    • Interactive queries
    • Branching
  • State Management & Stores

    Manage state: state stores, changelog topics, interactive queries, and state restoration.

    Objectives

    • Implement stateful operations
    • Query state stores
    • Handle state restoration

    Key concepts

    • StateStore
    • Changelog topic
    • Interactive queries
    • State restoration
  • KSQL & SQL on Streams

    Write streaming SQL queries with KSQL: SELECT, JOIN, aggregate.

    Objectives

    • Write KSQL queries
    • Aggregate streams
    • Join streams and tables

    Key concepts

    • KSQL
    • Stream queries
    • Aggregations
    • KSQL joins
  • Testing Streams Applications

    Unit test topology logic, integration test with embedded Kafka, and performance testing.

    Objectives

    • Write topology unit tests
    • Integration test streams apps
    • Validate correctness

    Key concepts

    • TopologyTestDriver
    • Embedded Kafka
    • Test fixtures
    • Performance testing
5 · Cluster Operations4 topics
  • Cluster Setup & Installation

    Deploy and configure Kafka: broker properties, JVM tuning, and infrastructure.

    Objectives

    • Deploy Kafka cluster
    • Configure brokers
    • Optimize JVM settings

    Key concepts

    • broker.id
    • log.dirs
    • JVM heap
    • ZooKeeper coordination
  • Broker Management & Scaling

    Add/remove brokers, rebalancing, leadership, and controlled shutdown.

    Objectives

    • Scale clusters horizontally
    • Manage broker failures
    • Optimize leadership distribution

    Key concepts

    • Broker scaling
    • Partition reassignment
    • Leader election
    • Preferred leaders
  • Monitoring & Metrics

    Monitor cluster health: JMX metrics, broker metrics, consumer group metrics.

    Objectives

    • Collect Kafka metrics
    • Set up monitoring dashboards
    • Define alerting rules

    Key concepts

    • JMX
    • Prometheus
    • Broker metrics
    • Alert thresholds
  • Troubleshooting & Debugging

    Debug common issues: broker failures, rebalancing storms, message loss, partition leadership.

    Objectives

    • Diagnose broker problems
    • Resolve rebalancing issues
    • Investigate message loss

    Key concepts

    • Log analysis
    • Broker diagnostics
    • Network issues
    • Debugging tools
6 · Security & Performance3 topics
  • Security & Authentication

    Secure Kafka: SASL/SSL, ACLs, and authorization.

    Objectives

    • Configure SASL authentication
    • Set up ACLs
    • Enable encryption and authorization

    Key concepts

    • SASL
    • SSL/TLS
    • ACLs
    • Encryption in transit
  • Performance Tuning

    Optimize Kafka: network tuning, disk I/O, CPU usage, and memory management.

    Objectives

    • Profile and optimize performance
    • Reduce latency
    • Maximize throughput

    Key concepts

    • num.network.threads
    • num.io.threads
    • Compression
    • CPU affinity
  • Scaling & Multi-Cluster Design

    Scale Kafka for production: multi-cluster setups, mirror-maker, cross-DC replication.

    Objectives

    • Design scalable architectures
    • Implement multi-cluster setups
    • Handle geographic distribution

    Key concepts

    • Multi-cluster
    • MirrorMaker
    • Cross-DC replication
    • Federation

Levels: Junior

Explain it in your own words

Graded by AI · no account · 1 question

No options to guess from: you write, the AI grades against the topic's rubric and tells you what level your answer reads at.

0 / 600

Already studying Apache Kafka?

The assessment measures topic by topic, and your plan skips what you already know.

Measure my level

Open to read. An account is only needed to be measured.

Frequently asked questions

Where should I start studying Apache Kafka?

Start with Core Concepts & Architecture: Kafka Fundamentals, Topics & Partitions, and Brokers & Cluster Architecture. The map orders topics by what they depend on, so each one prepares the next.

How long does a full Apache Kafka assessment take?

A full assessment of this path takes about 60 minutes. Study time depends on where you start; a diagnostic shows which topics you can skip.

Do I need an account to use this page?

No. The map, objectives and key concepts are open. You only need an account to be measured, get your gap by topic and receive a plan.