In the previous recipe, we learned how to infer the schema of a DataFrame using reflection.
In this recipe, we will learn how to specify the schema programmatically.
- Title Page
- Copyright and Credits
- Packt Upsell
- Contributors
- Preface
- Installing and Configuring Spark
- Abstracting Data with RDDs
- Introduction
- Creating RDDs
- Reading data from files
- Overview of RDD transformations
- Getting ready
- How to do it...
- .map(...) transformation
- .filter(...) transformation
- .flatMap(...) transformation
- .distinct() transformation
- .sample(...) transformation
- .join(...) transformation
- .repartition(...) transformation
- .zipWithIndex() transformation
- .reduceByKey(...) transformation
- .sortByKey(...) transformation
- .union(...) transformation
- .mapPartitionsWithIndex(...) transformation
- How it works...
- Overview of RDD actions
- Pitfalls of using RDDs
- Abstracting Data with DataFrames
- Introduction
- Creating DataFrames
- Accessing underlying RDDs
- Performance optimizations
- Inferring the schema using reflection
- Specifying the schema programmatically
- Creating a temporary table
- Using SQL to interact with DataFrames
- Overview of DataFrame transformations
- Getting ready
- How to do it...
- The .select(...) transformation
- The .filter(...) transformation
- The .groupBy(...) transformation
- The .orderBy(...) transformation
- The .withColumn(...) transformation
- The .join(...) transformation
- The .unionAll(...) transformation
- The .distinct(...) transformation
- The .repartition(...) transformation
- The .fillna(...) transformation
- The .dropna(...) transformation
- The .dropDuplicates(...) transformation
- The .summary() and .describe() transformations
- The .freqItems(...) transformation
- See also
- Overview of DataFrame actions
- Preparing Data for Modeling
- Machine Learning with MLlib
- Machine Learning with the ML Module
- Introducing Transformers
- Introducing Estimators
- Introducing Pipelines
- Selecting the most predictable features
- Predicting forest coverage types
- Estimating forest elevation
- Clustering forest cover types
- Tuning hyperparameters
- Extracting features from text
- Discretizing continuous variables
- Standardizing continuous variables
- Topic mining
- Structured Streaming with PySpark
- GraphFrames – Graph Theory with PySpark
Specifying the schema programmatically
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.