After all of this hard work, we can now create some aggregate statistics on global features.

The pandas value_counts() function allows us to plot the distribution of global features easily. By running the following code, we will get a bar chart output of our Wikipedia dataset:

train.Sub_Page.value_counts().plot(kind='bar')

As a result of running the previous code, we will output a bar chat that ranks the distributions of records within our dataset:

Distribution of records by Wikipedia country page

The preceding plot shows the number of time series available for each subpage. Wikipedia has subpages for different languages, and we can see that our dataset contains pages from the English (en), Japanese (ja), German (de), French (fr), Chinese (zh), Russian (ru), and Spanish (es) Wikipedia sites.

In the bar chart we produced you may have also noted two non-country based Wikipedia sites. Both commons.wikimedia.org and www.mediawiki.org are used to host media files such as images.

Let's run that command again, this time focusing on the type of access:

train.Access.value_counts().plot(kind='bar')

After running this code, we'll then see the following bar chart as the output:

Distribution of records by access type

There are two possible access methods: mobile and desktop. There's also a third option all-access, which combines the statistics for mobile and desktop access.

We can then plot the distribution of records by agent by running the following code:

train.Agent.value_counts().plot(kind='bar')

After running that code, we'll output the following chart:

Distribution of records by agent

There are time series available not only for spider agents, but also for all other types of access. In classic statistical modeling, the next step would be to analyze the effect of each of these global features and build models around them. However, this is not necessary if there's enough data and computing power available.

If that's the case then a neural network is able to discover the effects of the global features itself and create new features based on their interactions. There are only two real considerations that need to be addressed for global features:

To examine the global features, of our dataset, we have to look at a few sample time series in order to get an understanding of the challenges that we may face. In this section, we will plot the views for the English language page of Twenty One Pilots, a musical duo from the USA.

To plot the actual page views together with a 10-day rolling mean. We can do this by running the following code:

idx = 39457

window = 10


data = train.iloc[idx,0:-4]
name = train.iloc[idx,-4]
days = [r for r in range(data.shape[0] )]

fig, ax = plt.subplots(figsize=(10, 7))

plt.ylabel('Views per Page')
plt.xlabel('Day')
plt.title(name)

ax.plot(days,data.values,color='grey')
ax.plot(np.convolve(data, 
                    np.ones((window,))/window, 
                    mode='valid'),color='black')


ax.set_yscale('log')

There is a lot going on in this code snippet, and it is worth going through it step by step. Firstly, we define which row we want to plot. The Twenty One Pilots article is row 39,457 in the training dataset. From there, we then define the window size for the rolling mean.

We separate the page view data and the name from the overall dataset by using the pandas  iloc tool. This allows us to index the data by row and column coordinates. Counting the days rather than displaying all the dates of the measurements makes the plot easier to read, therefore we are going to create a day counter for the X-axis.

Next, we set up the plot and make sure it has the desired size by setting figsize. We also define the axis labels and the title. Next, we plot the actual page views. Our X coordinates are the days, and the Y coordinates are the page views.

To compute the mean, we are going to use a convolve operation, which you might be familiar with as we explored convolutions in Chapter 3, Utilizing Computer Vision. This convolve operation creates a vector of ones divided by the window size, in this case 10. The convolve operation slides the vector over the page view, multiplies 10-page views with 1/10, and then sums the resulting vector up. This creates a rolling mean with a window size 10. We plot this mean in black. Finally, we specify that we want to use a log scale for the Y axis:

Access statistics for the Twenty One Pilots Wikipedia page with a rolling mean

You can see there are some pretty large spikes in the Twenty One Pilots graph we just generated, even though we used a logarithmic axis. On some days, views skyrocket to 10 times what they were just days before. Because of that, it quickly becomes clear that a good model will have to be able to deal with such extreme spikes.

Before we move on, it's worth pointing out that it's also clearly visible that there are global trends, as the page views generally increase over time.

For good measure, let's plot the interest in Twenty One Pilots for all languages. We can do this by running the following code:

fig, ax = plt.subplots(figsize=(10, 7))
plt.ylabel('Views per Page')
plt.xlabel('Day')
plt.title('Twenty One Pilots Popularity')
ax.set_yscale('log')

for country in ['de','en','es','fr','ru']:
    idx= np.where((train['Subject'] == 'Twenty One Pilots') 
                  & (train['Sub_Page'] == '{}.wikipedia.org'.format(country)) & (train['Access'] == 'all-access') & (train['Agent'] == 'all-agents'))
                  
    idx=idx[0][0]
    
    data = train.iloc[idx,0:-4]
    handle = ax.plot(days,data.values,label=country)
    

ax.legend()

In this snippet, we first set up the graph, as before. We then loop over the language codes and find the index of Twenty One Pilots. The index is an array wrapped in a tuple, so we have to extract the integer specifying the actual index. We then extract the page view data from the training dataset and plot the page views.

In the following chart, we can view the output of the code that we've just produced:

Access statistics for Twenty One Pilots by country

There is clearly some correlation between the time series. The English language version of Wikipedia (the top line) is, not surprisingly, by far the most popular. We can also see that the time series in our datasets are clearly not stationary; they change means and standard deviations over time.

A stationary process is one whose unconditional joint probability distribution stays constant over time. In other words, things such as the series mean or standard deviation should stay constant.

However, as you can see, between days 200-250 in the preceding graph, the mean views on the page changes dramatically. This result undermines some of the assumptions many classic modeling approaches make. Yet, financial time series are hardly ever stationary, so it is worthwhile dealing with these problems. By addressing these problems, we become familiar with several useful tools that can help us handle nonstationarity.