Getting data from a NumPy array into the DataShape object of Blaze is extremely easy. First, let's create a simple NumPy array: we first load NumPy and then create a matrix with two rows and three columns:

import numpy as np
simpleArray = np.array([
        [1,2,3],
        [4,5,6]
    ])

Now that we have an array, we can abstract it with Blaze's DataShape structure:

simpleData_np = bl.Data(simpleArray)

That's it! Simple enough.

In order to peek inside the structure you can use the .peek() method:

simpleData_np.peek()

You should see an output similar to what is shown in the following screenshot:

You can also use (familiar to those of you versed in pandas' syntax) the .head(...) method.

If you want to retrieve the first column of your DataShape, you can use indexing:

simpleData_np[0]

You should see a table, as shown here:

On the other hand, if you were interested in retrieving a row, all you would have to do (like in NumPy) is transpose your DataShape:

simpleData_np.T[0]

What you will then get is presented in the following figure:

Notice that the name of the column is None. DataShapes, just like pandas' DataFrames, support named columns. Thus, let's specify the names of our fields:

simpleData_np = bl.Data(simpleArray, fields=['a', 'b', 'c'])

Now you can retrieve the data simply by calling the column by its name:

simpleData_np['b']

In return, you will get the following output:

As you can see, defining the fields transposes the NumPy array and, now, each element of the array forms a row, unlike when we first created the simpleData_np.

Since pandas' DataFrame internally uses NumPy data structures, translating a DataFrame to DataShape is effortless.

First, let's create a simple DataFrame. We start by importing pandas:

import pandas as pd

Next, we create a DataFrame:

simpleDf = pd.DataFrame([
        [1,2,3],
        [4,5,6]
    ], columns=['a','b','c'])

We then transform it into a DataShape:

simpleData_df = bl.Data(simpleDf)

You can retrieve data in the same manner as with the DataShape created from the NumPy array. Use the following command:

simpleData_df['a']

Then, it will produce the following output:

A DataShape object can be created directly from a .csv file. In this example, we will use a dataset that consists of 404,536 traffic violations that happened in the Montgomery county of Maryland.

We store the dataset in the ../Data folder locally. However, we modified the dataset slightly so we could store it in the MongoDB: in its original form, with date columns, reading data back from MongoDB caused errors. We filed a bug with Blaze to fix this issue https://github.com/blaze/blaze/issues/1580:

import odo
traffic = bl.Data('../Data/TrafficViolations.csv')

If you do not know the names of the columns in any dataset, you can get these from the DataShape. To get a list of all the fields, you can use the following command:

print(traffic.fields)

Blaze can also read directly from a GZipped archive, saving space:

traffic_gz = bl.Data('../Data/TrafficViolations.csv.gz')

To validate that we get exactly the same data, let's retrieve the first two records from each structure. You can either call the following:

traffic.head(2)

Or you can choose to call:

traffic_gz.head(2)

It produces the same results (columns abbreviated here):

It is easy to notice, however, that it takes significantly more time to retrieve the data from the archived file because Blaze needs to decompress the data.

You can also read from multiple files at one time and create one big dataset. To illustrate this, we have split the original dataset into four GZipped datasets by year of violation (these are stored in the ../Data/Years folder).

Blaze uses odo to handle saving DataShapes to a variety of formats. To save traffic data for traffic violations by year you can call odo like this:

import odo
for year in traffic.Stop_year.distinct().sort():
    odo.odo(traffic[traffic.Stop_year == year], 
        '../Data/Years/TrafficViolations_{0}.csv.gz'
        .format(year))

The preceding instruction saves the data into a GZip archive, but you can save it to any of the formats mentioned earlier. The first argument to the .odo(...) method specifies the input object (in our case, the DataShape with traffic violations that occurred in 2013), the second argument is the output object - the path to the file we want to save the data to. As we are about to learn - storing data is not limited to files only.

To read from multiple files you can use the asterisk character *:

traffic_multiple = bl.Data(
    '../Data/Years/TrafficViolations_*.csv.gz')
traffic_multiple.head(2)

The preceding snippet, once again, will produce a familiar table:

Blaze reading capabilities are not limited to .csv or GZip files only: you can read data from JSON or Excel files (both, .xls and .xlsx), HDFS, or bcolz formatted files.

Blaze can also easily read from SQL databases such as PostgreSQL or SQLite. While SQLite would normally be a local database, the PostgreSQL can be run either locally or on a server.

Blaze, as mentioned earlier, uses odo in the background to handle the communication to and from the databases.

In order to execute the code in this section, you will need two things: a running local instance of a PostgreSQL database, and a locally running MongoDB database.

Before you proceed, we assume that you have a PostgreSQL database up and running at http://localhost:5432/, and MongoDB database running at http://localhost:27017.

We have already loaded the traffic data to both of the databases and stored them in the traffic table (PostgreSQL) or the traffic collection (MongoDB).

traffic_psql = bl.Data(
    'postgresql://{0}:{1}@localhost:5432/drabast::traffic'
    .format('<your_username>', '<your_password>')
)

traffic_2016 = traffic_psql[traffic_psql['Year'] == 2016]
# Drop commands
# odo.drop('sqlite:///traffic_local.sqlite::traffic2016')
# odo.drop('postgresql://{0}:{1}@localhost:5432/drabast::traffic'
.format('<your_username>', '<your_password>'))
# Save to SQLite
odo.odo(traffic_2016,
'sqlite:///traffic_local.sqlite::traffic2016')
# Save to PostgreSQL
odo.odo(traffic_2016,  
    'postgresql://{0}:{1}@localhost:5432/drabast::traffic'
    .format('<your_username>', '<your_password>'))

traffic_sqlt = bl.Data(
    'sqlite:///traffic_local.sqlite::traffic2016'
)

traffic_mongo = bl.Data(
    'mongodb://localhost:27017/packt::traffic'
)