Data Engineer Interview Questions With Pipeline Failure Scenarios

Prepare to explain how data remains correct when a pipeline reruns or its inputs change. A successful scheduled run is only one case. Interviews can also explore missing records, duplicates, late events, changing schemas, and recovery.

These original practice questions focus on reasoning. They do not imply that a particular employer asks them or that every pipeline needs the same architecture.

1. A task wrote data and then failed. What happens on retry?

Suppose a fictional task writes 400 records from a 1,000-record input before losing its connection. A blind retry that appends all 1,000 can leave 1,400 records, including 400 duplicates. The exact outcome depends on transaction and sink behavior, so establish those facts first.

Airflow's best-practice guidance emphasizes repeatable task outcomes and stable input partitions. Use that principle to explain an appropriate write strategy: a transaction, atomic partition replacement, or a carefully defined upsert, depending on the storage system.

Do not say "use upsert" without identifying a stable key and how changed records are resolved. Test failure after partial work and rerun the same input. Compare keys and values, not only total row count.

2. How would you handle late-arriving data?

Clarify whether the business cares about event time, ingestion time, or both. A record received today may belong to yesterday's activity.

Discuss the allowed lateness, correction policy, and downstream expectations. A rolling reprocessing window can be useful, but its size must follow the data and requirements. Do not choose an arbitrary window and claim it captures every late event.

For a report, explain whether historical numbers can change and how consumers learn that a partition was revised. For a streaming system, ask what happens beyond the accepted lateness boundary.

3. A source column changes type. How do you respond?

Separate detection from recovery. Validate the incoming schema and identify which consumers depend on the field. Avoid silently coercing invalid values into nulls if that would make a bad dataset look successful.

An illustrative response:

I would preserve the received input, stop or isolate the affected output according to the data contract, and identify whether the change was intentional. I would coordinate the contract update, test the transformation on representative records, and plan any necessary historical correction.

Explain the consequences of stopping versus continuing with quarantined records. Neither choice is universally correct.

4. Which quality checks would you add?

Choose checks that protect the dataset's meaning. Examples include unique business keys, required fields, acceptable value ranges, referential consistency, freshness, and reconciled totals where appropriate.

Use a concrete question: if a daily orders table has the expected row count but every amount is zero, would your checks detect it? Conversely, an unusual volume may reflect a legitimate event. State which checks block publication and which create an alert for investigation.

5. How would you backfill a year of data safely?

Define the affected range, transformation version, input snapshot, and destination. Estimate the load on source and downstream systems. Test a small representative partition, compare the output, then expand with clear checkpoints and a recovery plan.

Keep current processing and historical correction from overwriting each other's work. Explain how you will verify completeness and communicate changed metrics to users. "Rerun everything" is not a sufficient plan without boundaries.

The technical interview preparation guide can help organize practice. For each pipeline story, rehearse the initial symptom, the data at risk, your recovery decision, and the checks that established the result was trustworthy.