Big data interviews probe architecture trade-offs, distributed storage, streaming guarantees, and the ability to diagnose incorrect or expensive pipelines. These 50 questions focus on reasoning rather than memorized marketing terms.

Questions are original study examples, not questions from a live hiring assessment. Adapt your examples to your own work and verify tool behavior using an authorized practice environment.

Distributed data fundamentals

1. What makes a workload a big data problem?

A workload becomes a big data engineering problem when volume, velocity, variety, reliability needs, or processing deadlines exceed what a simple single-node design handles economically. Define the specific constraint before choosing distributed infrastructure.

2. What is horizontal scaling?

Horizontal scaling adds machines or partitions to share data and work. It can improve capacity but introduces coordination, network costs, partition imbalance, and operational complexity that do not arise in the same way on one machine.

3. How is batch processing different from stream processing?

Batch processing works over bounded datasets, often on a schedule. Stream processing handles arriving events continuously or in small increments. Both need failure recovery, a definition of event time, and a policy for late data.

4. What is data partitioning?

Partitioning divides records or files into groups that can be stored or processed independently. Good partition keys balance workload and support common queries; poor keys can create hotspots and expensive shuffles.

5. How are data replication and partitioning different?

Partitioning spreads distinct portions of a dataset across resources. Replication creates copies for resilience or access. More replicas may improve availability but increase storage, consistency coordination, and synchronization costs.

6. What does data locality mean?

Data locality is the advantage of processing near stored data when transferring it is expensive. Modern engines also optimize network shuffles, caching, and compute placement; locality is one factor rather than a universal design rule.

7. What is the difference between a data lake and a warehouse?

A lake commonly stores diverse raw or processed files with flexible schemas. A warehouse structures governed analytical data for querying. The boundary depends on the platform, and good governance matters in either model.

8. What is a lakehouse architecture?

It combines file-based storage with table formats, transactions, metadata, and management features associated with warehouses. Validate its actual query, concurrency, lifecycle, and governance guarantees rather than assuming every lakehouse product behaves alike.

9. What is schema evolution?

Schema evolution describes changes to fields, types, and constraints over time. Safe evolution requires compatibility rules, consumer testing, and documented treatment of old data so downstream jobs do not silently misinterpret records.

10. What is a data contract?

A data contract defines a producer's agreed schema, semantic meaning, quality expectations, and change process. It helps consumers detect breaking changes, but monitoring and ownership are needed to enforce it in practice.

Hadoop and distributed storage

11. What are the roles of HDFS NameNode and DataNode?

The NameNode manages filesystem metadata and mapping of blocks to storage locations. DataNodes store the data blocks and serve reads or writes. Availability design must consider metadata services and actual storage failures separately.

12. Why does HDFS use blocks?

Large files are divided into blocks for placement, replication, and distributed access. Block sizing affects metadata overhead and sequential throughput; many tiny files can burden metadata systems.

13. What is the small files problem?

Huge numbers of tiny files create disproportionate filesystem metadata, open costs, and task scheduling overhead. Compacting files or using suitable table formats can improve processing efficiency.

14. How does HDFS replication contribute to resilience?

Additional block replicas allow data access after some failures. Replication is not a substitute for application-level disaster recovery, consistent backups, or correct handling of accidental deletion and corruption.

15. What is MapReduce?

It is a distributed computation model with mapping and reduction stages. The framework coordinates task execution and shuffle behavior; many modern systems offer higher-level APIs and can reduce unnecessary disk materialization.

16. How is HDFS different from object storage?

HDFS presents a filesystem-oriented model suited to its cluster ecosystem, while object stores expose object APIs and different consistency, listing, and permission semantics. Design connectors and commit behavior for the actual storage provider.

17. What is a shuffle?

A shuffle redistributes data across partitions, often for joins, groupings, or repartitioning. It can become expensive due to network transfers, serialization, memory pressure, and skew.

18. How do you diagnose a skewed join?

Inspect partition size distributions and stage runtimes. Consider filtering, better keys, salting where appropriate, broadcast of genuinely small inputs, or different join logic; validate correctness before optimizing speed.

19. What is a distributed checkpoint?

A checkpoint stores progress or state so computation can resume after failure. Its guarantees depend on the engine, sink, storage semantics, and whether side effects are idempotent.

20. When should a team avoid Hadoop complexity?

If the data fits comfortably in a simpler database or batch process and meets reliability needs, a cluster may add cost and operational risk without providing benefit. Choose tools based on measurable workload constraints.

Spark processing and SQL

21. How do DataFrames differ from RDDs in Spark?

DataFrames expose schema-aware relational operations, allowing Spark's optimizer to reason about execution. RDDs provide lower-level transformations but often lack the same structured optimization opportunities.

22. What is lazy evaluation in Spark?

Many transformations build a logical plan and execute only when an action or write requires a result. This enables optimization but means errors may appear later than the code that declares a transformation.

23. What is an action in Spark?

An action causes execution or collection, such as count, collect, or a data write. Calling collect on a large dataset can exhaust driver memory, so inspect data with bounded methods.

24. Why is repartition different from coalesce?

Repartition redistributes records and can increase or decrease partitions with a shuffle. Coalesce commonly reduces partitions with less movement, but may leave uneven work. Validate the physical plan and workload before choosing.

25. What does caching a DataFrame do?

Caching can reuse intermediate results across actions, reducing recomputation when the data fits available resources. It consumes memory or storage and should be justified by repeated access rather than enabled indiscriminately.

26. How can you inspect Spark query performance?

Review the execution plan and Spark UI for stage duration, shuffle, spills, partition skew, and data-source reads. Compare multiple runs under representative input and cluster conditions.

27. What is a broadcast join?

It distributes a sufficiently small input to executor nodes to avoid a large shuffle for certain joins. It can improve performance, but excessive broadcast data may cause memory pressure.

28. What are window functions?

They calculate values across defined groups and ordering without collapsing each group into one result row. Use them for ranking, moving totals, and per-entity event analysis, with deterministic ordering where required.

29. What is the difference between event time and processing time?

Event time indicates when an event occurred; processing time indicates when a system handled it. Late or out-of-order events make this distinction important for accurate windowed aggregations.

30. What does Spark Structured Streaming provide?

It expresses incremental stream processing through structured APIs and supports checkpointing and event-time logic. Delivery and recovery guarantees depend on the selected mode, query, sink, and surrounding pipeline design.

Kafka, streams, and reliability

31. What is a Kafka topic partition?

A partition is an ordered log of records within a topic. Kafka preserves order within a partition, not across an entire multi-partition topic, so ordering requirements influence key selection.

32. How do Kafka consumer groups work?

Consumers in a group coordinate partition assignments so work can be shared. A partition is typically assigned to one member of a group at a time; adding consumers beyond partition count may not increase parallelism.

33. What is an offset?

An offset identifies a record position within a partition. Consumers track progress using offsets, but a committed offset does not by itself prove that all external side effects completed successfully.

34. How do you approach duplicate events?

Assume retries can produce duplicates unless the end-to-end pipeline proves otherwise. Choose stable event identifiers, idempotent writes, deduplication windows, and checks that match the business semantics.

35. What is idempotency in data pipelines?

An operation is idempotent when repeating it produces the same intended end state. It simplifies recovery after retries, but idempotent sinks do not automatically solve late-event and ordering problems.

36. What is a dead-letter queue?

It stores records that could not be processed within defined retry rules, together with context for diagnosis. Monitor its volume, protect sensitive fields, and implement a reviewed recovery path.

37. What is backpressure?

Backpressure occurs when downstream components cannot process incoming data at the required rate. Teams monitor lag, throughput, memory, and queue growth, then tune capacity or admission rather than hiding data loss.

38. What is a watermark in stream processing?

A watermark expresses progress in event time and guides when a system may finalize or expire state for windowed results. A poorly chosen delay can discard late data or grow state excessively.

39. What does exactly-once mean in practice?

It is an end-to-end claim that depends on source, processing, state, and sink behavior. Do not assume a framework's internal guarantee means every email, API call, or database side effect occurs exactly once.

40. How do you recover a broken streaming pipeline?

Stop or isolate the failing stage, preserve checkpoint and source state, identify the first bad event, and define replay boundaries. Test recovery on representative data to avoid duplicating or silently dropping outcomes.

Data quality, governance, and platform design

41. What is data freshness?

Freshness measures how old data is relative to its expected update cadence. Define it against business requirements, and distinguish source delay from transformation and reporting delay.

42. How do you implement pipeline observability?

Track data volumes, lag, failure rates, schema changes, quality checks, cost, and owner escalation paths. Alerts should explain impact and provide enough context for a useful first investigation.

43. What is data lineage?

Lineage records how fields and datasets flow through sources, transformations, and consumers. It helps evaluate breaking changes, identify incorrect outputs, and explain data provenance.

44. How do you design data retention?

Define business purpose, legal requirements, deletion timelines, storage tiers, and recovery needs. Ensure derived datasets, caches, backups, and logs receive explicit consideration instead of retaining everything indefinitely.

45. How do you protect personal data in a data lake?

Apply least-privilege access, encryption, masking or tokenization where appropriate, audit logging, and verified retention rules. Data discovery and classification are essential before broad analytics access.

46. What is a medallion data architecture?

Bronze, silver, and gold are commonly used names for raw, cleaned, and business-ready layers. They describe organization patterns, not a universal standard or automatic data-quality guarantee.

47. How would you control cloud analytics costs?

Monitor storage lifecycle, query scans, concurrency, cluster utilization, streaming uptime, and data movement. Set budgets and usage alerts, then optimize based on measured query and business value.

48. How can you choose between Spark, SQL warehouse, and a managed stream engine?

Start with data shape, latency, operations burden, governance, integration, and cost requirements. Benchmark a representative workload; avoid selecting a technology solely because it is prominent in interview lists.

49. What is a good data engineering interview system-design answer?

State users and SLAs, estimate scale, choose partition and storage strategies, define ingestion and processing semantics, address quality and security, and explain cost and recovery trade-offs.

50. What is the most important habit for a data engineering candidate?

Be explicit about assumptions and guarantees. Present testable evidence, explain failure modes, and revise your design when data contradicts it, rather than relying on fashionable platform names.

Study with official material

Consult issuer and project-maintained documentation, not leaked or memorized answer sets: Apache Spark SQL documentation, Apache Hadoop HDFS guide.

How to prepare next

Sketch a small ingestion architecture, state assumptions about scale and latency, and explain how you would verify correctness, resiliency, and cost.