StatefulSet behaves differently to DaemonSet during Pod recreation. In StatefulSet managed Pods, the Pod name is always consisted to assign an ordered number such as hdfs-datanode-0, hdfs-datanode-1 andhdfs-datanode-2, and if you delete one of them, a new Pod will take over the same Pod name.
To simulate this, let's delete one DataNode (hdfs-datanode-1) to see how StatefulSet recreates a Pod:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hdfs-datanode-0 1/1 Running 0 3m
hdfs-datanode-1 1/1 Running 0 2m
hdfs-datanode-2 1/1 Running 0 2m
hdfs-namenode-0 1/1 Running 0 23m
//delete DataNode-1
$ kubectl delete pod hdfs-datanode-1
pod "hdfs-datanode-1" deleted
//DataNode-1 is Terminating
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hdfs-datanode-0 1/1 Running 0 3m
hdfs-datanode-1 1/1 Terminating 0 3m
hdfs-datanode-2 1/1 Running 0 2m
hdfs-namenode-0 1/1 Running 0 23m
//DataNode-1 is recreating automatically by statefulset
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hdfs-datanode-0 1/1 Running 0 4m
hdfs-datanode-1 0/1 ContainerCreating 0 16s
hdfs-datanode-2 1/1 Running 0 3m
hdfs-namenode-0 1/1 Running 0 24m
//DataNode-1 is recovered
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hdfs-datanode-0 1/1 Running 0 4m
hdfs-datanode-1 1/1 Running 0 22s
hdfs-datanode-2 1/1 Running 0 3m
hdfs-namenode-0 1/1 Running 0 24m
As you see, the same Pod name (hdfs-datanode-1) has been assigned. Approximately after 10 minutes (due to HDFS's heart beat interval), HDFS web console shows that the old Pod has been marked as dead and the new Pod has the in service state, shown as follows:
Note that this is not a perfect ideal case for HDFS, because DataNode-1 lost data and expects to re-sync from other DataNodes. If the data size is bigger, it may take a long time to complete re-sync.
Fortunately, StatefulSets has an capability that preserve a persistent volume while replacing a Pod. Let's see how HDFS DataNode can preserve data during Pod recreation.