Question 1

What is the primary difference between transformations and actions in PySpark?

Accepted Answer

Transformations are lazy operations that produce a new DataFrame from an existing one without triggering immediate execution. In contrast, actions are operations that trigger the actual computation of the Directed Acyclic Graph (DAG) to return a result to the driver or write data to storage. Transformations define the execution plan, while actions represent the execution of that plan.

Question 2

What is an RDD in PySpark and what are its core characteristics?

Accepted Answer

RDD stands for Resilient Distributed Dataset, which is the fundamental data structure of Apache Spark. It is an immutable, distributed collection of objects that can be processed in parallel across a cluster. RDDs are low-level APIs that lack a fixed schema, providing developers with fine-grained control over data computations while supporting lazy evaluation and fault tolerance.

Question 3

What is PySpark and how does it facilitate big data processing?

Accepted Answer

PySpark is the Python API for Apache Spark, an open-source distributed computing system that provides an interface for programming entire clusters with implicit data parallelism and fault tolerance. It allows developers to write Spark applications using Python, which makes large-scale data processing more accessible to data professionals who prefer Python's simplicity. Under the hood, PySpark uses the py4j library to communicate between Python and the Java Virtual Machine (JVM) where Spark runs.

Question 4

What are the primary responsibilities of the Spark Driver node?

Accepted Answer

The Spark Driver is the central coordinator of a Spark application, responsible for running the main() method and orchestrating the entire workflow. It converts user-defined code into a logical plan (DAG), coordinates with the cluster manager to acquire resources, and breaks the job into stages and tasks for distribution. Additionally, the driver monitors task execution on worker nodes and collects results to return to the user.

Question 5

What are the primary differences between RDDs, DataFrames, and Datasets in Spark?

Accepted Answer

RDDs are Spark's core data structure, representing an immutable distributed collection of objects without a fixed schema. DataFrames are distributed collections organized into named columns, similar to a table in a relational database, and are optimized by the Catalyst engine. Datasets provide the benefits of both RDDs and DataFrames by adding compile-time type safety, though they are not supported in PySpark.