Spark Data Ingestion Optimization: Explicit Schemas vs InferSchema

Data ingestion is the first step in any data engineering pipeline. A simple choice at this step can make the difference between a pipeline that runs in seconds versus one that takes hours in production.

What I Was Trying to Do,

I wanted to get started with Spark optimization while learning data engineering. Data ingestion is the foundation, so that's where I started.

I had a transactions dataset of around 7GB in Azure Data Lake Storage. I used the common approach for reading a CSV file with Spark:

transactions_df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
.load("abfss://wetrustbidata@adlsstoragedata01.dfs.core.windows.net/transactions.csv")

Just let Spark figure out the data types automatically with inferSchema=True. Simple, convenient, and I don't have to think about it.

The Execution Time

When I ran the cell, the execution took longer than expected.

98.26 seconds. Almost 2 minutes just to read a file with a single-node Databricks cluster

This was concerning. Spark is built to handle massive datasets efficiently, so why was a 7GB file taking this long?

That's when it hit me: If 7GB takes 2 minutes, what happens in production when you're dealing with terabytes of data?

Minutes turn into hours. Hours turn into SLA breaches. And you're paying for all that compute time.

Understanding Why This Happens

To understand the performance issue, I looked into how inferSchema=True it works.

When you use inferSchema=True, Spark needs to determine the data types for each column. Here's what happens:

1. First Pass - Schema Inference: Spark reads through your data to understand what type each column should be. Is this column a string? An integer? A date? It samples the data to make these decisions.

2. Second Pass - Actual Data Load: Now that Spark knows the schema, it reads the data again to actually load it into a DataFrame.

Spark is reading your data twice.

To verify this, I checked the Spark Jobs section in Databricks.

See that? (2) Spark Jobs. Two separate jobs - Job 0 and Job 1.

Let me show you what each job is doing:

- Stage 0: This is the schema inference job. It took 1 second and scanned 64 KiB of data to figure out the schema.

- Stage 1: This is the actual data load. It took 1.5 minutes and read 6.8 GiB - the full dataset.

So the total time? 1.5 minutes for loading + overhead for inference = ~2 minutes total.

The problem is clear: Spark has to do two jobs when it should only need to do one.

The Solution: User-Defined Schema

Instead of letting Spark infer the schema, we can define it explicitly up front.

This is called a User-Defined Schema. In Databricks documentation, this approach is also referred to as schema-on-read with explicit schema definition.

The difference:

- inferSchema: Spark scans the data to determine column types

- User-Defined Schema: Spark uses the provided schema and directly loads the data

Here's the implementation. First, define the schema manually:

from pyspark.sql.types import StructType, StructField, StringType, DoubleType, DateType

# Define schema explicitly

transactions_schema = StructType([

    StructField('transaction_id', StringType(), True),

    StructField('AccountID', StringType(), True),

    StructField('TransactionOperation', StringType(), True),

    StructField('transaction_type', StringType(), True),

    StructField('transaction_date', DateType(), True),

    StructField('Amount', DoubleType(), True)

])

Then I loaded the data with this schema:

transactions_df = spark.read.format("csv") \
    .option("header", "true") \
    .schema(transactions_schema) \ .load("abfss://wetrustbidata@adlsstoragedata01.dfs.core.windows.net/transactions.csv")

I created a fresh notebook to get a new Spark session and ran this code.

1.31 seconds.

The execution time dropped from 98 seconds to 1.31 seconds - a 75x performance improvement.

Verifying the Optimization

To verify this optimization, I checked the Spark Jobs section again:

**(1) Spark Jobs**. Just one job this time. Job 2.

And when I looked at the stages:

Only Stage 2 - the data load stage. It took 0.2 seconds and read 64 KiB.

Wait, only 64 KiB? That's because this stage is just the initial scan setup. The actual data reading happens during the task execution, which is completed in just over 1 second total.

The key difference: No Stage for schema inference. Spark skipped that entirely because I already told it what the schema was

The difference is clear:

- No inference job needed - Spark already knows the data types

- Single job - Just load the data, and you're done

- 75x faster - From almost 2 minutes to just over 1 second

What Does This Mean for Production?

This was just 7GB of data. But in production, data engineers work with much larger datasets.

Let's say you're working with a 1TB dataset. If it scales linearly:

- inferSchema: ~14,000 seconds = **~3.9 hours**

- User-Defined Schema: ~187 seconds = **~3 minutes**

You save almost 4 hours. And that's just for one file, one load.

Now imagine you're running this daily. Or even hourly. The time (and cost) adds up fast.

The Cost Impact

In cloud environments like Databricks, you pay for compute time. More time = more money.

With inferSchema=True, you're:

• Paying for schema inference that you don't need

• Scanning data twice instead of once

• Running your cluster 75x longer

For production pipelines that run regularly, this can mean thousands of dollars in wasted compute costs every month.

When Should You Use inferSchema vs User-Defined Schema?

Both approaches have their use cases.

Use inferSchema when:

- You're exploring data and don't know the schema yet

- You're working with small files (< 100MB) where speed doesn't matter

- The schema keeps changing, and you need flexibility

- You're doing a quick, one-off analysis in a notebook

Use User-Defined Schema when:

- You're building production pipelines

- You're working with large files (> 1GB)

- The schema is stable and known

- Performance matters

- You're running scheduled jobs

How to Get the Schema If You Don't Know It

If the schema is unknown, you can use inferSchema once on a small sample to discover it, then use that schema definition in production code.

# Step 1: Run this once to see the schema

sample_df = spark.read.format("csv") \

    .option("header", "true") \

    .option("inferSchema", "true") \

    .load("path/to/your/file.csv")

# Step 2: Print the schema

sample_df.printSchema()

# Step 3: Copy the output and create your StructType definition

# Use that in your production code

Or, if you have documentation about your data source (like a database schema or API documentation), just map those types to Spark types directly.

Key Takeaways

Defining schemas explicitly instead of using inferSchema=True provides significant benefits:

- 75x faster data loading (98 seconds → 1.3 seconds)

- Reduces Spark jobs by 50% (2 jobs → 1 job)

- Saves hours of compute time in production

- Reduces cloud costs significantly

This optimization demonstrates that performance improvements don't always require complex tuning or advanced configurations. In this case, explicitly defining the schema eliminates unnecessary computation.

For production data pipelines, defining schemas explicitly is a best practice that improves performance with every pipeline execution.

When using inferSchema=True, Consider whether explicitly defining the schema would be more efficient for your use case.