Processing Only What Changed: My Journey with Change Data Feed (CDF)

I've been working on building data pipelines in the retail domain for a while now, and I've noticed something interesting. Let me paint you a picture.

Imagine you're ingesting customer data from a major e-commerce platform. On Day 1, you receive a batch of 10 million customer records. You process it, load it into your data lake, and everything looks great.

Day 2 arrives. Another batch lands. 10 million records again.

But here's the catch - are these really 10 million NEW customers? Absolutely not.

Even the biggest e-commerce platforms don't acquire millions of new customers every single day. What's actually happening? The same existing customers are placing more orders, updating their profiles, changing addresses, or modifying payment methods. Out of those 10 million records, maybe only 50,000 are genuinely new or updated.

With traditional batch processing, the entire dataset often ends up being read and processed every single day. This leads to rising compute costs, increasing processing time, and a lot of unnecessary work, reprocessing millions of records that haven’t changed at all.

This is when I realized: I don't need to process the entire dataset. I just need to process what changed.

How to Achieve Incremental Processing in Databricks

After researching different approaches, I found that there are several ways to process only incremental data in Databricks:

1. Watermarking – Track a timestamp or ID column and filter records greater than the last processed value.

2. Left Anti Join – Compare the current dataset with the previously processed snapshot to identify newly arrived records.

3. Change Data Feed (CDF) – A Delta Lake feature that captures inserts, updates, and deletes directly from Delta tables. It allows you to read only the changed rows without implementing custom comparison logic.

In this blog, we’ll explore the third approach, Change Data Feed (CDF), and understand how it helps simplify incremental processing in Databricks.

What is Change Data Feed?

Change Data Feed (CDF) is a Delta Lake feature that maintains a change log of all modifications made to a table. Once enabled, CDF records:

• What data changed

• When it changed (timestamp)

• What type of change occurred (insert, update, delete)

• Which version of the table does the change belong to

The beauty? You can query this change log using standard SQL to retrieve only the data that changed between two points in time - no full table scans needed.

My Experiment with CDF

I decided to test CDF with a simple scenario: ingesting customer data in batches and processing only the new records. Here's exactly what I did.

Step 1: Initial Data Load

First, I checked what customer data files were available in my cloud storage:

display(dbutils.fs.ls("abfss://test@adlsstoragedata01.dfs.core.windows.net/customersdata/"))

I had two files: customers_1.csv and customers_2.csv.

Step 2: Loading the First Batch

I loaded the first customer file to see what we're working with:

customers_df_1 = spark.read.format("csv").option("header", "true").load("abfss://test@adlsstoragedata01.dfs.core.windows.net/customersdata/customers_1.csv") 
display(customers_df_1)

This file contained 10 customer records.

Step 3: Writing to Delta Table

I wrote this data to a Delta table:

customers_df_1.write.mode("append").saveAsTable("gbmart.bronze.customers")

Result: 10 rows loaded.

Step 4: Enabling Change Data Feed

This is the critical step - enabling CDF on the table:

ALTER TABLE gbmart.bronze.customers 
SET TBLPROPERTIES 
(delta.enableChangeDataFeed = true);

We enable CDF because it allows us to capture only the rows that have changed (inserts, updates, deletes) instead of scanning the entire dataset again.

From this point forward, every change to this table will be tracked automatically. As we proceed, you’ll get a clear understanding of what CDF is, how it works, and why it simplifies incremental processing.

I checked the table history:

DESCRIBE HISTORY gbmart.bronze.customers;

The table now had CDF enabled (you can see this in the operation details).

Step 5: Loading the Second Batch

Now, a new batch of customer data arrived. I loaded the second file:

customers_df_2 = spark.read.format("csv").option("header", "true").load("abfss://test@adlsstoragedata01.dfs.core.windows.net/customersdata/customers_2.csv") 

display(customers_df_2)

This file contained 5 new customer records.

I appended it to the same table:

customers_df_2.write.mode("append").saveAsTable("gbmart.bronze.customers")

Verified the new count:

SELECT COUNT(*) AS total_rows FROM gbmart.bronze.customers;

Result: 15 rows total (10 + 5).

Step 6: The CDF Magic - Processing Only New Data

Now here's the scenario: your downstream pipeline has already processed the first 10 rows, and you don’t want to reprocess them; you only want to process the 5 new rows that just arrived.

The same logic applies at scale: if 10 million rows have already been processed and a new batch of 200,000 rows arrives, you should process only those 200,000 new rows instead of reprocessing the entire dataset.

I checked the table history again:

DESCRIBE HISTORY gbmart.bronze.customers;

I could see multiple versions (0, 1, 2). Version 2 contained the latest changes.

• Version 0: Initial table creation when the first 10 rows from customers_1.csv were loaded

• Version 1: When CDF was enabled using the ALTER TABLE SET TBLPROPERTIES command (no data change, just a metadata operation)

• Version 2: When the second batch of 5 rows from customers_2.csv was appended - this is where our new data lives!

The key query - using CDF to get only the changes since a specific timestamp:

SELECT * FROM table_changes("gbmart.bronze.customers", "2025-12-11T06:57:46.000+00:00")

Result: Only 5 rows returned - exactly the new records that were added!

The table_changes function allows you to query only the rows that have changed in a Delta table between specific versions or timestamps. It returns the incremental changes rather than the full dataset.

The result includes special CDF columns:

• `_change_type`: Type of change (insert, update_preimage, update_postimage, delete)

• `_commit_version`: Delta table version when the change occurred

• `_commit_timestamp`: Exact timestamp of the change

For my downstream processing, I loaded this into a Dataframe:

customers_raw_df = spark.sql(""" 
SELECT * FROM table_changes("gbmart.bronze.customers", "2025-12-11T06:57:46.000+00:00")
""")

And if I wanted just the customer data without CDF metadata:

customers_raw_df.drop("_change_type", "_commit_version", "_commit_timestamp").display()

Perfect! I successfully retrieved only the 5 new rows out of 15 total rows.

Now, with Change Data Feed (CDF) enabled in Delta, we can focus on processing only the new or changed records. I tried one approach in execution and the results were impressive.

The Impact

Think about the savings: Instead of processing 10 million records daily, I'm now processing only 50,000 changed records. That's:

• 99.5% reduction in data scanned

• Massive compute cost savings

• Faster pipeline execution

• Lower cloud storage I/O costs

In a real-world scenario with billions of rows, CDF transforms batch processing pipelines into efficient, incremental processing systems without requiring complex custom logic or application changes.

Key Takeaway

Change Data Feed is a game-changer for incremental data processing in Databricks. With a single line of code to enable it and a simple SQL query to consume changes, you can dramatically reduce processing overhead and costs.

If you're building data pipelines that process the same tables repeatedly, CDF should be one of the first features you enable.