Skip to main content

Learning path

Apache Spark learning path: everything to study, topic by topic

Comprehensive Apache Spark assessment covering distributed computing fundamentals, RDDs and DataFrames, Spark SQL, structured streaming, MLlib, performance tuning, and production deployment patterns.

Short answer

To learn Apache Spark, work through 6 modules and 22 topics in the order they build on each other. It starts with Spark Fundamentals, then moves on to RDDs & DataFrames and Spark SQL. Use the map as a checklist, or measure your level and study only what is missing.

6
modules
22
topics
45
min to assess

What you will study

Topics are grouped by module. Open one to see its objectives and key concepts.

1 · Spark Fundamentals3 topics
  • Spark Architecture

    Driver and executor model, SparkContext, SparkSession, cluster managers (YARN, Mesos, Kubernetes, Standalone), job-stage-task hierarchy, DAG scheduler, and lazy evaluation

    Objectives

    • Explain the Spark driver-executor architecture
    • Understand how Spark breaks jobs into stages and tasks

    Key concepts

    • driver program
    • executor processes
    • SparkContext
    • SparkSession
    • cluster manager
    • DAG scheduler
    • lazy evaluation
    • job-stage-task hierarchy
  • Spark Ecosystem

    Spark Core, Spark SQL, Spark Streaming, MLlib, GraphX, PySpark, Scala API, SparkR, and integration with Hadoop ecosystem

    Objectives

    • Identify the components of the Spark ecosystem
    • Choose the right Spark module for a given use case

    Key concepts

    • Spark Core
    • Spark SQL
    • Spark Streaming
    • MLlib
    • GraphX
    • PySpark
    • Hadoop integration
    • unified analytics engine
  • Spark Setup & Configuration

    SparkSession builder, spark-submit, configuration properties, local vs cluster mode, application packaging, and dependency management

    Objectives

    • Configure and launch Spark applications
    • Understand spark-submit options and deployment modes

    Key concepts

    • SparkSession.builder
    • spark-submit
    • spark.conf properties
    • local mode
    • cluster mode
    • application JAR
    • dependency management
    • spark-defaults.conf
2 · RDDs & DataFrames5 topics
  • RDD Basics

    Creating RDDs from collections and files, partitioning, immutability, fault tolerance through lineage, narrow vs wide dependencies

    Objectives

    • Create RDDs from various data sources
    • Explain how Spark achieves fault tolerance through lineage

    Key concepts

    • RDD creation (parallelize, textFile)
    • partitions
    • immutability
    • lineage graph
    • fault tolerance
    • narrow dependencies
    • wide dependencies
    • RDD persistence
  • RDD Transformations & Actions

    Transformations (map, filter, flatMap, reduceByKey, groupByKey, join), actions (collect, count, reduce, saveAsTextFile), lazy evaluation chain

    Objectives

    • Distinguish between transformations and actions
    • Apply common RDD transformations correctly

    Key concepts

    • map transformation
    • filter transformation
    • flatMap transformation
    • reduceByKey
    • groupByKey
    • actions vs transformations
    • collect action
    • lazy evaluation chain
  • DataFrame API

    Creating DataFrames, column operations (select, filter, withColumn, groupBy, agg), joins, UDFs, and the Catalyst optimizer integration

    Objectives

    • Perform data manipulation using the DataFrame API
    • Understand when to use DataFrames over RDDs

    Key concepts

    • DataFrame creation
    • select and filter
    • withColumn
    • groupBy and agg
    • join types
    • UDFs (User Defined Functions)
    • Column expressions
    • DataFrame vs RDD
  • Dataset API

    Typed Dataset operations, Encoders, case classes, Dataset vs DataFrame vs RDD trade-offs, type safety at compile time

    Objectives

    • Use the typed Dataset API for compile-time safety
    • Evaluate trade-offs between Dataset, DataFrame, and RDD

    Key concepts

    • Dataset[T]
    • Encoders
    • case classes
    • type safety
    • map on Dataset
    • as[T] conversion
    • Dataset vs DataFrame
    • serialization overhead
  • Schema & Data Types

    StructType, StructField, data types (StringType, IntegerType, ArrayType, MapType), schema inference vs explicit schema, schema evolution

    Objectives

    • Define and apply schemas to DataFrames
    • Handle complex and nested data types

    Key concepts

    • StructType
    • StructField
    • Spark data types
    • schema inference
    • explicit schema definition
    • nullable fields
    • complex types (ArrayType, MapType)
    • schema evolution
3 · Spark SQL4 topics
  • Spark SQL Queries

    Running SQL queries on DataFrames, temporary views, global views, subqueries, CTEs, and SQL function library

    Objectives

    • Write Spark SQL queries against registered views
    • Choose between SQL and DataFrame API based on use case

    Key concepts

    • createOrReplaceTempView
    • spark.sql()
    • global temp views
    • subqueries
    • CTEs (WITH clause)
    • built-in SQL functions
    • SQL vs DataFrame API
    • catalog API
  • Catalyst Optimizer

    Logical plan, physical plan, optimization rules (predicate pushdown, column pruning, constant folding), Tungsten engine, whole-stage code generation

    Objectives

    • Read and interpret Spark query execution plans
    • Understand how Catalyst optimizes queries

    Key concepts

    • logical plan
    • physical plan
    • predicate pushdown
    • column pruning
    • constant folding
    • Tungsten engine
    • whole-stage codegen
    • explain() output
  • Data Sources & Formats

    Reading/writing Parquet, ORC, JSON, CSV, Avro, JDBC, Delta Lake, partition discovery, schema merging, and data source options

    Objectives

    • Read and write data in various formats
    • Choose appropriate file formats for different use cases

    Key concepts

    • Parquet format
    • ORC format
    • JSON/CSV readers
    • Avro support
    • JDBC data source
    • partition discovery
    • schema merging
    • write modes (append, overwrite, ignore)
  • Window Functions

    Window specifications, ranking functions (row_number, rank, dense_rank), aggregate windows, lead/lag, frame specifications (rows, range)

    Objectives

    • Apply window functions for advanced analytics
    • Define window frames for running calculations

    Key concepts

    • Window specification
    • partitionBy
    • orderBy
    • row_number
    • rank / dense_rank
    • lead / lag
    • frame specification (rows/range)
    • running aggregates
4 · Streaming & MLlib4 topics
  • Structured Streaming

    Streaming DataFrames, readStream/writeStream, output modes (append, complete, update), triggers, watermarks, and event-time processing

    Objectives

    • Build streaming pipelines with Structured Streaming
    • Handle late data with watermarks

    Key concepts

    • readStream
    • writeStream
    • output modes (append, complete, update)
    • triggers
    • watermarks
    • event-time processing
    • late data handling
    • streaming sources and sinks
  • MLlib Fundamentals

    ML algorithms (classification, regression, clustering), feature transformers, Vectors (dense/sparse), LabeledPoint, model training and evaluation

    Objectives

    • Train machine learning models using MLlib
    • Evaluate model performance with appropriate metrics

    Key concepts

    • classification algorithms
    • regression algorithms
    • clustering (KMeans)
    • feature transformers
    • Vectors (dense/sparse)
    • model training
    • model evaluation
    • cross-validation
  • ML Pipelines

    Pipeline API, Transformers and Estimators, Pipeline stages, hyperparameter tuning, ParamGrid, CrossValidator, model persistence

    Objectives

    • Build end-to-end ML pipelines with Spark
    • Tune hyperparameters using CrossValidator

    Key concepts

    • Pipeline API
    • Transformer
    • Estimator
    • Pipeline stages
    • ParamGridBuilder
    • CrossValidator
    • TrainValidationSplit
    • model save/load
  • Stream Processing Patterns

    Stream-stream joins, stream-static joins, deduplication, stateful processing, mapGroupsWithState, flatMapGroupsWithState, checkpointing

    Objectives

    • Implement complex stream processing patterns
    • Design stateful streaming applications with exactly-once guarantees

    Key concepts

    • stream-stream joins
    • stream-static joins
    • deduplication
    • mapGroupsWithState
    • flatMapGroupsWithState
    • stateful processing
    • checkpointing
    • exactly-once semantics
5 · Performance & Tuning3 topics
  • Memory Management

    Unified memory model, execution memory, storage memory, off-heap memory, memory overhead, GC tuning, serialization (Kryo vs Java)

    Objectives

    • Configure Spark memory settings for optimal performance
    • Diagnose and resolve memory-related issues

    Key concepts

    • unified memory model
    • execution memory
    • storage memory
    • off-heap memory
    • spark.memory.fraction
    • Kryo serialization
    • GC tuning
    • memory overhead
  • Shuffles & Partitioning

    Shuffle operations, hash partitioning, range partitioning, data skew, salting technique, broadcast joins, shuffle file management

    Objectives

    • Optimize shuffle operations for large-scale processing
    • Resolve data skew problems with partitioning strategies

    Key concepts

    • shuffle operations
    • hash partitioner
    • range partitioner
    • spark.sql.shuffle.partitions
    • data skew
    • salting technique
    • broadcast join
    • coalesce vs repartition
  • Caching & Persistence

    cache() vs persist(), storage levels (MEMORY_ONLY, MEMORY_AND_DISK, etc.), when to cache, unpersist, checkpoint vs cache, broadcast variables, accumulators

    Objectives

    • Apply caching strategies to optimize iterative workloads
    • Choose appropriate storage levels for different scenarios

    Key concepts

    • cache()
    • persist()
    • storage levels
    • MEMORY_ONLY
    • MEMORY_AND_DISK
    • unpersist()
    • broadcast variables
    • accumulators
6 · Production & Architecture3 topics
  • Cluster Management

    YARN, Kubernetes, Standalone scheduler, dynamic allocation, resource negotiation, executor sizing, spark-submit configurations

    Objectives

    • Configure Spark clusters for production workloads
    • Size executors and drivers appropriately

    Key concepts

    • YARN resource manager
    • Kubernetes operator
    • Standalone scheduler
    • dynamic allocation
    • executor memory/cores
    • driver sizing
    • queue management
    • resource isolation
  • Monitoring & Debugging

    Spark UI, event logs, metrics system, stage/task analysis, identifying bottlenecks, executor logs, OOM troubleshooting, Spark History Server

    Objectives

    • Use Spark UI to identify performance bottlenecks
    • Troubleshoot common Spark failures and OOM errors

    Key concepts

    • Spark UI tabs
    • event logs
    • Spark History Server
    • stage analysis
    • task metrics
    • executor logs
    • OOM troubleshooting
    • DAG visualization
  • Deployment & Architecture Patterns

    Lambda architecture, Kappa architecture, lakehouse architecture, Delta Lake, data quality frameworks, CI/CD for Spark, testing strategies, multi-tenancy

    Objectives

    • Design production data architectures with Spark
    • Implement CI/CD pipelines for Spark applications

    Key concepts

    • Lambda architecture
    • Kappa architecture
    • lakehouse architecture
    • Delta Lake
    • CI/CD for Spark
    • testing strategies
    • multi-tenancy
    • data governance

Levels: Junior, Mid, Senior, and Expert

Explain it in your own words

Graded by AI · no account · 1 question

No options to guess from: you write, the AI grades against the topic's rubric and tells you what level your answer reads at.

0 / 600

Already studying Apache Spark?

The assessment measures topic by topic, and your plan skips what you already know.

Measure my level

Open to read. An account is only needed to be measured.

Frequently asked questions

Where should I start studying Apache Spark?

Start with Spark Fundamentals: Spark Architecture, Spark Ecosystem, and Spark Setup & Configuration. The map orders topics by what they depend on, so each one prepares the next.

How long does a full Apache Spark assessment take?

A full assessment of this path takes about 45 minutes. Study time depends on where you start; a diagnostic shows which topics you can skip.

Do I need an account to use this page?

No. The map, objectives and key concepts are open. You only need an account to be measured, get your gap by topic and receive a plan.