Data Engineer learning path: everything to study, topic by topic

ETL vs ELT architectures, transformation placement, staging strategies, incremental loading, and medallion architecture patterns

Batch job design, scheduling strategies, data partitioning for batch workloads, checkpoint/restart patterns, and MapReduce concepts

DAG design, scheduling, dependency management, retry policies, backfill, SLA monitoring, and orchestrator selection criteria

Idempotent pipeline design, deduplication strategies, upsert patterns, watermark-based processing, and exactly-once batch semantics

Producer-consumer contracts, schema enforcement, backward/forward compatibility, breaking change management, and contract testing

Syncing analytical warehouse data back to operational tools (CRMs, marketing platforms), Census/Hightouch patterns, activation use cases, and sync strategies

Star schema, snowflake schema, fact and dimension tables, grain definition, Kimball methodology, and conformed dimensions

Architecture comparison of data lakes, warehouses, and lakehouse patterns, including medallion architecture and cost-performance trade-offs

Columnar vs row-based formats, compression codecs, schema evolution support, read/write performance, and format selection criteria

Table partitioning strategies, bucket/cluster strategies, partition pruning, small file problem, and partition evolution

Apache Iceberg, Delta Lake, Apache Hudi — ACID transactions on data lakes, time travel, schema evolution, and format comparison

Designing lakehouse architectures using Delta Lake, Apache Iceberg, or Apache Hudi for ACID transactions on data lakes

Tiered storage architectures, lifecycle policies, compression optimization, cold/warm/hot data patterns, and storage cost modeling for data platforms

Window functions, CTEs, recursive queries, lateral joins, advanced aggregations, and analytical SQL patterns

dbt project structure, models, tests, macros, incremental models, materializations, and documentation-as-code

Data quality dimensions, expectation suites, validation pipelines, data profiling, anomaly detection, and quality metrics

SCD Types 1-6, hybrid approaches, temporal data modeling, effective dating, and bitemporal design patterns

Query optimization, execution plans, materialized views, index strategies, join optimization, and warehouse-specific tuning

Kafka architecture, topics, partitions, consumer groups, Kinesis streams, offset management, and message ordering

Stream processing frameworks, windowing strategies, watermarks, event-time vs processing-time, and stateful processing

CDC patterns, Debezium connectors, log-based CDC, snapshot strategies, schema change handling, and CDC pipeline architectures

Exactly-once processing guarantees, transactional producers, idempotent consumers, two-phase commit, and end-to-end guarantees

Real-time feature stores, materialized views for serving, CQRS patterns, speed layer, and serving layer architectures

Event sourcing for analytics, CQRS patterns applied to data platforms, event stores, temporal queries, and rebuilding state from event logs

OLAP engines (ClickHouse, Apache Druid, Pinot), materialized views, pre-aggregation strategies, and sub-second query architectures for analytics

Object storage patterns, cloud data services comparison, data lake storage design, lifecycle policies, and access control

Spark architecture, cluster sizing, memory management, shuffle optimization, dynamic allocation, and Spark on Kubernetes

Cost modeling for data platforms, storage tiering, compute optimization, reserved capacity, and FinOps practices for data teams

CI/CD for data pipelines, testing strategies, environment management, blue-green deployments, and data pipeline versioning

End-to-end data platform design, technology selection, scalability, multi-tenancy, and self-service data infrastructure

Monitoring data pipelines for anomalies, implementing freshness/volume/schema checks, alerting on data quality regressions, and building observability platforms

Building data APIs for internal consumers, GraphQL for data access, API-first analytics, rate limiting, caching strategies, and API versioning for data products

Unit testing data transformations, integration testing pipelines, data mocking frameworks, test data generation, and CI/CD for data quality

Data lineage tracking, metadata catalogs, impact analysis, data discovery, and lineage integration with orchestration tools

PII detection, data masking techniques, tokenization, encryption at rest/transit, and anonymization strategies

Right to erasure implementation, consent management in data pipelines, data retention policies, and cross-border data transfer

Domain-oriented data ownership, data as a product, self-serve data platform, federated governance, and organizational adoption patterns

Data product definition, SLOs for data products, discoverability, interoperability, and data product lifecycle management

Designing and implementing Data Mesh: domain ownership, data products, federated governance, and self-serve infrastructure

Building and operating feature stores (Feast, Tecton) for real-time ML feature serving with point-in-time correctness

Implementing data quality frameworks at scale using Great Expectations, dbt tests, and anomaly detection

Implementing enterprise data governance: data cataloging (Datahub, Amundsen), lineage tracking, classification, and access policies

Fine-grained access control for data lakes, column masking, row-level security, encryption at rest/transit, and multi-tenant data isolation

Metadata stores, active metadata for automation, automated cataloging, metadata-driven pipelines, and business glossary management

Working effectively with data analysts and scientists, defining data SLAs, establishing data contracts between teams, and managing data product ownership