For this recipe, we will start creating an RDD by generating the data within the PySpark. To create RDDs in Apache Spark, you will need to first install Spark as shown in the previous chapter. You can use the PySpark shell and/or Jupyter notebook to run these code samples.
- Title Page
- Copyright and Credits
- Packt Upsell
- Contributors
- Preface
- Installing and Configuring Spark
- Abstracting Data with RDDs
- Introduction
- Creating RDDs
- Reading data from files
- Overview of RDD transformations
- Getting ready
- How to do it...
- .map(...) transformation
- .filter(...) transformation
- .flatMap(...) transformation
- .distinct() transformation
- .sample(...) transformation
- .join(...) transformation
- .repartition(...) transformation
- .zipWithIndex() transformation
- .reduceByKey(...) transformation
- .sortByKey(...) transformation
- .union(...) transformation
- .mapPartitionsWithIndex(...) transformation
- How it works...
- Overview of RDD actions
- Pitfalls of using RDDs
- Abstracting Data with DataFrames
- Introduction
- Creating DataFrames
- Accessing underlying RDDs
- Performance optimizations
- Inferring the schema using reflection
- Specifying the schema programmatically
- Creating a temporary table
- Using SQL to interact with DataFrames
- Overview of DataFrame transformations
- Getting ready
- How to do it...
- The .select(...) transformation
- The .filter(...) transformation
- The .groupBy(...) transformation
- The .orderBy(...) transformation
- The .withColumn(...) transformation
- The .join(...) transformation
- The .unionAll(...) transformation
- The .distinct(...) transformation
- The .repartition(...) transformation
- The .fillna(...) transformation
- The .dropna(...) transformation
- The .dropDuplicates(...) transformation
- The .summary() and .describe() transformations
- The .freqItems(...) transformation
- See also
- Overview of DataFrame actions
- Preparing Data for Modeling
- Machine Learning with MLlib
- Machine Learning with the ML Module
- Introducing Transformers
- Introducing Estimators
- Introducing Pipelines
- Selecting the most predictable features
- Predicting forest coverage types
- Estimating forest elevation
- Clustering forest cover types
- Tuning hyperparameters
- Extracting features from text
- Discretizing continuous variables
- Standardizing continuous variables
- Topic mining
- Structured Streaming with PySpark
- GraphFrames – Graph Theory with PySpark
Creating RDDs
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.