What is the Spark Streaming application data flow?
The following figure provides the data flow between the Spark driver, workers, streaming sources and targets:
It all starts with the Spark Streaming Context, represented by ssc.start() in the preceding figure:
When the Spark Streaming Context starts, the driver will execute a long-running task on the executors (that is, the Spark workers).
The Receiver on the executors (Executor 1 in this diagram) receives a data stream from the Streaming Sources. With the incoming data stream, the receiver divides the stream into blocks and keeps these blocks in memory.
These blocks are also replicated to another executor to avoid data loss.
The block ID information is transmitted to the Block Management Master on the driver.
For every batch interval configured within Spark Streaming Context (commonly this is every 1 second), the driver will launch Spark tasks to process the blocks. Those blocks are then persisted to any number of target data stores, including cloud storage (for example, S3, WASB, and so on), relational data stores (for example, MySQL, PostgreSQL, and so on), and NoSQL stores.
Suffice it to say, there are a lot of moving parts for a streaming application that need to be continually optimized and configured. Most of the documentation for Spark Streaming is more complete in Scala, so, as you are working with the Python APIs, you may sometimes need to reference the Scala version of the documentation instead. If this happens to you, please file a bug and/or fill out a PR if you have a proposed fix (https://issues.apache.org/jira/browse/spark/).