As mentioned above, the Dataset is a component of Spark SQL module introduced from the Spark 1.5.0 release. Therefore, all the entry point of all functionally starts from the initialization of Spark SQLContext . Basically, Spark SQL is used for executing SQL queries written either as a basic SQL syntax or HiveQL.

A Dataset object will be returning when running SQL within another programming language. The following code segment will initialize the SQLContext within Spark Context. On the other hand, you might require having the HiveContext initialized for reading a data set from the Hive. You can also create a different context like HiveContext which provides a superset of basic functionalities of SQLContext:

JavaSparkContext sc = new JavaSparkContext("local","DFDemo"); 
SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc); 

[Input File] 
{"name":"Michael"} 
{"name":"Andy", "age":30} 
{"name":"Justin", "age":19} 
[Code]    
Dataset<Row> df = sqlContext.read().json("people.json");    
 
[Output: df.show()] 
+----+-------+ 
| age|   name| 
+----+-------+ 
|null|Michael| 
|  30|   Andy| 
|  19| Justin| 
+----+-------+ 

The code is as follows:]hiveContext.sql("CREATE TEMPORARY TABLE people USING  
org.apache.spark.sql.json OPTIONS ( path "people.json" )"); 
Dataset<Row> results = hiveContext.sql("SELECT * FROM people "); 
results.show(); 

In the previous section, we have described the pre-processing with RDD for a practical machine learning application. Now we will do the same example using DataFrame (DF) API. You will find it very easy to manipulate the SMSSpamCollection Dataset (see at http://www.dt.fee.unicamp.br/~tiago/smsspamcollection/). We will show the same example by tokenizing the spam and ham messages for preparing a training set:

This section will describe how to use SQL queries on DF and different way to create Datasets across the datasets. Mainly running the SQL queries on DataFrame and the creating DataFrame from the JavaBean will be discussed in this section. However, interested readers can refer Spark programing guidelines for SQL operation in [3].

You can create Dataset from a Java Bean; where you don't need to define the schemas programmatically. For example, you can see Plain Old Java Object (POJO) named as Bean in the following code:

public class SMSSpamBean implements Serializable { 
  private String labelString; 
  private String featureString; 
public SMSSpamBean(String labelString, String featureString) { 
    super(); 
    this.labelString = labelString; 
    this.featureString = featureString; 
  } 
  public String getLabelString() { 
    return labelString; 
  } 
  public void setLabelString(String labelString) { 
    this.labelString = labelString; 
  } 
  public String getFeatureString() { 
    return featureString; 
  }  public void setFeatureString(String featureString) {    this.featureString = featureString; 
  }}  

Create DF:

JavaRDD<SMSSpamBean> smsSpamBeanRDD =  rowRDD.map(new Function<Row, SMSSpamBean>() { 
      @Override 
    public SMSSpamBean call(Row r) throws Exception { 
        String[] split = r.getString(0).split("	"); 
        return new SMSSpamBean(split[0],split[1]); 
      }});   
Dataset<Row> SMSSpamDF = spark.sqlContext().createDataFrame(smsSpamBeanRDD, SMSSpamBean.class); 
SMSSpamDF.show();   

The following output is as follows:

Figure 15: Corresponding feature and label string