In this chapter, you have learned how to use Zeppelin to query parquet files and display some charts. Then, you developed a small program to stream transaction data from a WebSocket to a Kafka topic. Finally, you used Spark Streaming inside Zeppelin to query the data arriving in the Kafka topic in real time.

With all these building blocks in place, you have all the tools to analyze the Bitcoin transaction data in more detail. You could let the BatchProducerApp run for several days or weeks to get some historical data. With the help of Zeppelin and Spark, you could then try to detect patterns and come up with a trading strategy. Finally, you could then use a Spark Streaming flow to detect in real time when some trading signal arises and perform a transaction automatically.

We have produced streaming data on only one topic, but it would be quite straightforward to add other topics covering other currency pairs, such as BTC/EUR or BTC/ETH. You could also create another program that fetches data from another cryptocurrency exchange. This would enable you to create Spark Streaming queries that detect arbitrage opportunities (when the price of a product is cheaper in one market than in another).

The building blocks we implemented in this chapter can be also used in a Lambda architecture. A Lambda architecture is a data processing architecture designed to handle large volumes of data by using both batch processing and streaming methods. Many Lambda architectures involve having different code bases for the batch and streaming layers, but with Spark, this negative point can be greatly reduced. You can find out more about Lambda architectures on this website: http://lambda-architecture.net/.

This completes the final chapter in our book. We hope you enjoyed reading it as much as we enjoyed writing it. You will be empowered by the ins and outs of various concepts of Scala. Scala is a language that is well designed and is not an end by himself, this is the basement to build more interesting concepts, like the type theory and for sure the category theory, I encourage your curiosity to look for more concepts to continually improve the readability and the quality of your code. You will also be able to apply it to solve a variety of real-world problems.