Let's build a simple stateful application that would help us understand the various aspects of a reliable stateful application. We will build an up and down counter that will increment or decrement a counter value depending on the partition it belongs to. We will save the counter value in a ReliableDictionary instance and trace the current state of the dictionary so that we can visualize it in the Diagnostics Event Viewer console.

To begin, using Visual Studio create a new Service Fabric application named StatefulUpDownCounterApplication and add a Stateful Reliable Service to it. Name the service UpDownCounterService:

Let's first create two partitions of this service. For this sample, we will use the named partitioning scheme. Applications that use named partitions usually work with categorized data, for instance an election service that persists vote data by state. Clients of applications that use the named partition scheme need to explicitly specify which partition they want to access at runtime.

To create a named partition, navigate to the ApplicationManifest.xml file and update the contents of the Service node to the following:

    <Service Name="UpDownCounterService"> 
      <StatefulService ServiceTypeName="UpDownCounterServiceType" TargetReplicaSetSize="[UpDownCounterService_TargetReplicaSetSize]" MinReplicaSetSize="[UpDownCounterService_MinReplicaSetSize]"> 
        <NamedPartition> 
          <Partition Name="UpCounter" /> 
          <Partition Name="DownCounter" /> 
        </NamedPartition> 
      </StatefulService> 

Using the preceding configuration Service Fabric will create two partitions named UpCounter and DownCounter for your service.

Next, navigate to the UpDownCounterService class and clear the default code present in the RunAsync method. Let's start placing our counter code inside this method.

First, we need to identify the partition in which our process is getting executed.

The following code block, creates an instance of ReliableDictionary named counter. We use the Partition property of the base class StatefulService to retrieve the partition information.

            var myDictionary = await this.StateManager.GetOrAddAsync<IReliableDictionary<string, long>>("counter"); 
            var myPartitionName = (this.Partition.PartitionInfo as NamedPartitionInformation)?.Name; 

The variable myPartitionName will contain the name of one of the partitions that we had created earlier. Let's consider the case when the value of the myPartitionName variable will be "UpCounter". In this case, we will initialize the dictionary with 0 and keep updating the value by one every five seconds. The code is as follows:

            switch (myPartitionName) 
            { 
                case "UpCounter": 
                    while (true) 
                    { 
                        cancellationToken.ThrowIfCancellationRequested(); 
                        using (var tx = this.StateManager.
                                 CreateTransaction()) 
                        { 
                            var result = await myDictionary.
                                TryGetValueAsync(tx, "Counter"); 
                            ServiceEventSource.Current.ServiceMessage( 
                                this, 
                                "Current Counter Value: {0}", 
                                result.HasValue ? result.Value.
                                 ToString() : "Value does not 
                                                           exist."); 
                            await myDictionary.AddOrUpdateAsync(tx, 
                                "Counter", 0, (key, value) => 
                                                           ++value); 
                            await tx.CommitAsync(); 
                        } 

                        await Task.Delay(TimeSpan.FromSeconds(5), 
                          cancellationToken); 
                    } 
                case "DownCounter": 
... 
            } 

All operations on Reliable Collections happen inside transaction scope. This is because the data in the collections need to be actively replicated to majority of secondary nodes to ensure high availability of state data. A failure to replicate data to majority of replicas would lead to failure of transaction.

Next, we need to implement the down counter. We will write a similar implementation as we did for up counter, except that the counter value will get initialized with 1000 and it will decrement every five seconds. The code for down counter is as follows:

                case "DownCounter": 
                    while (true) 
                    { 
                        cancellationToken.ThrowIfCancellationRequested(); 
                        using (var tx = this.StateManager.
                                 CreateTransaction()) 
                        { 
                            var result = await myDictionary.
                              TryGetValueAsync(tx, "Counter"); 
                            ServiceEventSource.Current.ServiceMessage( 
                                this.Context, 
                                "Current Counter Value: {0}", 
                                result.HasValue ? result.Value.
                                 ToString() : "Value does not 
                                                      exist."); 
                            await myDictionary.AddOrUpdateAsync(tx,  
                             "Counter", 1000, (key, value) => --
                                                              value); 
                            await tx.CommitAsync(); 
                        } 
                        await Task.Delay(TimeSpan.FromSeconds(5), 
                          cancellationToken); 
                    } 

Let's also set the replica count of the service to 3 so that we have multiple secondary replicas of the service. Locate the parameter file Local.5Node.xml in the ApplicationParameters folder. This file is used to supply parameters when you are running your local cluster in five node mode. This mode spins five host processes on your machine to mimic deployment on five nodes. Validate that the minimum number of replicas and target number of replicas is set to three (at least):

<Parameters> 
  <Parameter Name="UpDownCounterService_MinReplicaSetSize" Value="3" />    
  <Parameter Name="UpDownCounterService_TargetReplicaSetSize" Value="3" 
    /> 
</Parameters> 

Next, switch the cluster mode to 5 Node, if not done already, by right clicking the Service Fabric Local Cluster Manager icon in the task bar and selecting the appropriate mode:

Let's quickly deploy our application to the local cluster by pressing F5. Open the Diagnostic Events window and observe the output of the application:

You would notice that even though both the counters referenced the same dictionary named counter, the two counters are progressing independently. That is because the Reliable Collections are not shared across partitions, but only across replicas.

Next, let's kill the primary replicas of both the partitions to see whether the state data is lost. To do so, open the Service Fabric Explorer and find out which node is hosting the primary replica of your service partitions:

In my case _Node_0 and _Node_3 host the primary replicas of the two partitions of the service that we just built. Let's deactivate the nodes and observe the counter value. Click on the ellipsis next to the node to reveal the various node operations. Let's simulate a complete node crash by selecting the Deactivate (remove data) option:

You will notice that as soon as you deactivate the node, an active secondary replica on another node, _Node_1 in my case, gets promoted to primary status and resumes processing without loss of state data which you can verify by looking at the logs accumulated in the Diagnostic Events window: