Resilient Distributed Dataset
Resilient– If data in memory is lost,it can be recreated
Distributed– Stored in memory across a cluster
Dataset-Initial data can come from a file or created programmatically
An RDD is the basic unit of data in Spark in which all Operations are performed.Simply an immutable distributed collection of objectsRDDs are intermediate results stored in Memory and are Partitioned to be operated on multiple nodes in the ClusterRDDs can be created in two ways: by loading an external dataset, or by distributing a collection of objects
The simplest way to create RDDs is to take an existing collection in your program and pass it to SparkContext’s parallelize() method
parallelize() method in Scala
RDD performs two types of operations:
Set of operations on a RDD that define how they should be transformed
As in relational Algebra ,the application of atransformation to an RDD yields a new RDD
(because RDD are immutable)
Transformations are Lazily evaluated,Which allow for optimizations to take place before execution
It returns a new RDD.
Actions are the operations that return a final value to the driver program or write data to an external storage system.