Industry
general
Skills
approach
data-understanding
data-wrangling
stream-etl
data-storage
Tools
databricks
spark
Learning Objectives
Understand the key concepts of Structured Streaming and its components for building streaming pipelines.
Learn how to ensure data reliability with state management, checkpointing, and Write-Ahead Log (WAL).
Learn how to use Autoloader to process large datasets with schema evolution support.
Understand trigger modes (micro-batch, continuous) and how they impact streaming performance.
Learn about output modes (append, complete, update)
Overview
Prerequisites
- Basic understanding of streaming data processing and pipelines
- Familiarity with Databricks and cloud data lakes