Lakehouse Platform Engineering — Lakeflow

Lakeflow Architecture — Declarative Thinking, Pipeline Types & Execution Model

Build your first Lakeflow pipeline — define Bronze, Silver & Gold layers declaratively

Migrate an existing PySpark ETL notebook to a Lakeflow declarative pipeline

Configure pipeline modes — triggered vs continuous — for different SLA requirements

Data Quality with Expectations — Constraints, Warnings, Failures & Quarantine Patterns

Define multi-layer expectations across Bronze, Silver and Gold in a single pipeline

A pipeline passes DQ checks despite null primary keys — debug broken expectation logic

Implement quarantine pattern to isolate invalid records without halting the pipeline

CDC with APPLY CHANGES INTO — SCD Type 1, SCD Type 2 & Out-of-order Event Handling

Implement SCD Type 1 for customer dimension updates using APPLY CHANGES INTO

Implement SCD Type 2 to track full historical changes in product pricing

CDC pipeline processes out-of-order events and corrupts dimension history — diagnose and fix

Streaming Tables in Lakeflow — Auto Loader Integration, Triggers & Watermarking

Build a streaming ingestion pipeline from cloud storage into Lakeflow using Auto Loader

A streaming pipeline produces duplicate records after checkpoint recovery — resolve idempotency

Pipeline watermark is too aggressive causing late-arriving records to be silently dropped — tune it

Pipeline Observability — Event Logs, Data Lineage, Monitoring & Error Handling Patterns

Debug a production pipeline failure using DLT event logs and the lineage graph

Set up alerting and SLA monitoring for a business-critical Lakeflow pipeline

A pipeline error message is generic — implement structured error handling patterns for production