Data Engineer

ETL vs ELT Patterns

Why: You choose where each transformation runs and design incremental loads with a staging step, so a new source is added without reprocessing years of history.

Batch Processing Fundamentals

Why: You design batch jobs with partitioning, scheduling windows and restart points of your own, so a failed run at 3 a.m. resumes cleanly instead of needing a manual rerun.

Dimensional Modeling (Star/Snowflake)

Why: You model a new business area with conformed dimensions, so sales and support reports filter by the same customer and stop disagreeing in meetings.

Data Lake vs Warehouse vs Lakehouse

Why: You propose where a new workload should live and justify it with cost and performance, for example moving cold history out of the warehouse.

Advanced SQL (Window Functions/CTEs)

Why: You rewrite a slow, nested analytical query using window functions or a recursive CTE, and you can show that the result is the same.

dbt & Transformation Layers

Why: You structure a dbt project in layers with tests and incremental models, so a change to a staging model is reviewed, tested and documented before it ships.

Message Queue Fundamentals (Kafka/Kinesis)

Why: You consume from Kafka with the right consumer group and offset handling, so a restart does not skip events or replay a day of orders.

Stream Processing (Flink/Spark Streaming)

Why: You build a windowed aggregation on event time, so late-arriving events from mobile clients land in the right minute and the dashboard stays accurate.

Cloud Data Services (S3/GCS/ADLS)

Why: You design bucket layout, lifecycle policies and access roles for a data lake, so storage cost stays predictable and each team sees only its own zone.

Spark Cluster Management

Why: You size executors and fix a skewed shuffle, so a Spark job that ran out of memory finishes within its window without simply buying a bigger cluster.

Data Lineage & Cataloging

Why: You use lineage to see which dashboards depend on a column before you rename it, and register the table in the catalog so others can find it.

PII Handling & Masking

Why: You detect personal data in a new source and mask or tokenize it before it reaches analysts, so a customer email never appears in a shared table.