Go back
Design and Implement Reliable ETL pipeline for WeBank using Databricks- Full Version
9 Scenarios
8 Hours 30 Minutes
Intermediate
Industry
banking
Skills
quality
data-understanding
data-storage
data-quality
batch-etl
cloud-management
data-wrangling
stream-etl
Tools
databricks
azure
sql
python
Learning Objectives
Implementing an E2E Data Engineering Architecture
Building Data Governance Layer using Databricks
Designing notifications strategy based on the data in Gold Lake
Handling Incremental data and varying schema
How to ensure Idempotency and Quality of the pipeline
Overview
Prerequisites
- Comprehensive understanding of how ADLS and Azure SQL works
- Knowledge on capturing streaming data using Azure Event-Hub
- Ingestion and Parsing Data using Databricks
- Knowledge on how Delta live table and Unity catalog work in Databricks