Stream Processing through Databricks on Azure

4 Scenarios

12 Hours

Industry

e-commerce

Skills

stream-etl

data-understanding

data-storage

data-wrangling

approach

data-quality

Tools

azure

databricks

spark

Learning Objectives

Grasp the basics of real-time data ingestion and processing with streaming tools.

Build a data pipeline that processes and stores streaming data efficiently.

Manage stateful and stateless operations in a streaming environment

Use output modes to manage how and when data is written during streaming.

Implement watermarking to handle late-arriving data in real-time streams.

Perform real-time joins between multiple data streams for deeper analysis.

Understanding the need for real-time data ingestion.
Familiarity with streaming data sources like Event Hubs and methods for ingesting data.
Basic knowledge of Apache Spark