You may determine your application as stateless because the datastore function is handled by another pod or system. However, there are some pitfalls in that sometimes; applications actually store important files that you aren't aware of. For example, Grafana (https://grafana.com/grafana) connects time series datasources such as Graphite (https://graphiteapp.org) and InfluxDB (https://www.influxdata.com/time-series-database/), so people may misunderstand that Grafana is a stateless application.

Actually, Grafana itself also uses databases to store user, organization, and dashboard metadata. By default, Grafana uses SQLite3 components and stores the database as /var/lib/grafana/grafana.db. Therefore, when a container is restarted, the Grafana settings will all be reset.

The following example demonstrates how Grafana behaves with an ephemeral volume:

$ cat grafana.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      run: grafana
  template:
    metadata:
      labels:
        run: grafana
    spec:
      containers:
        - image: grafana/grafana
          name: grafana
          ports:
            - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  ports:
    - protocol: TCP
      port: 3000
      nodePort: 30300
  type: NodePort
  selector:
    run: grafana

Next, navigate to the Grafana web console to create the Grafana Organizations named kubernetes org, as follows: 

Then, look at the Grafana directory. Here, there is a database file (/var/lib/grafana/grafana.db) with a timestamp that has been updated after creating a Grafana Organizations:

$ kubectl get pods
NAME                      READY     STATUS    RESTARTS   AGE
grafana-6bf966d7b-7lh89   1/1       Running   0          3m


//access to Grafana container
$ kubectl exec -it grafana-6bf966d7b-7lh89 /bin/bash
grafana@grafana-6bf966d7b-7lh89:/$ ls -l /var/lib/grafana/
total 404
-rw-r--r-- 1 grafana grafana 401408 Sep 20 03:30 grafana.db
drwxrwxrwx 2 grafana grafana   4096 Sep  7 14:59 plugins
drwx------ 4 grafana grafana   4096 Sep 20 03:30 sessions

When the pod is deleted, the Deployment will start a new pod and check whether a Grafana Organizations exists or not:

grafana@grafana-6bf966d7b-7lh89:/$ exit
//delete grafana pod
$ kubectl delete pod grafana-6bf966d7b-7lh89
pod "grafana-6bf966d7b-7lh89" deleted


//Kubernetes Deployment made a new Pod
$ kubectl get pods
NAME                      READY     STATUS    RESTARTS   AGE
grafana-6bf966d7b-wpdmk   1/1       Running   0          9s


//contents has been recreated
$ kubectl exec -it grafana-6bf966d7b-wpdmk /bin/bash
grafana@grafana-6bf966d7b-wpdmk:/$ ls -l /var/lib/grafana
total 400
-rw-r--r-- 1 grafana grafana 401408 Sep 20 03:33 grafana.db
drwxrwxrwx 2 grafana grafana   4096 Sep  7 14:59 plugins

It looks like the sessions directory has disappeared and grafana.db has also been recreated by the Docker image again. If you access the web console, the Grafana organization will also disappear:

How about just attaching a persistent volume to Grafana? You'll soon find that mounting a persistent volume on a pod controlled by a Deployment doesn't scale properly as every new pod attempts to mount the same persistent volume. In most cases, only the first pod can mount the persistent volume. Other pods will try to mount the volume, and they will give up and freeze if they can't. This happens if the persistent volume is capable of only RWO (read write once; only one pod can write).

In the following example, Grafana uses a persistent volume to mount /var/lib/grafana; however, it can't scale because the Google persistent disk is RWO:

$ cat grafana-pv.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      run: grafana
  template:
    metadata:
      labels:
        run: grafana
    spec:
      containers:
        - image: grafana/grafana:3.1.1
          name: grafana
          ports:
            - containerPort: 3000
          volumeMounts:
          - mountPath: /var/lib/grafana
            name: grafana-data
      volumes:
      - name: grafana-data
        gcePersistentDisk:
          pdName: gce-pd-1
          fsType: ext4

$ kubectl create -f grafana-pv.yml 
deployment.apps/grafana created

$ kubectl get pods
NAME  READY  STATUS RESTARTS  AGE
grafana-6cf5467c9d-nw6b7  1/1  Running  0 41s


//can't scale up, becaues 3 Pod can't mount the same volume
$ kubectl scale deploy grafana --replicas=3
The Deployment "grafana" is invalid: spec.template.spec.volumes[0].gcePersistentDisk.readOnly: Invalid value: false: must be true for replicated pods > 1; GCE PD can only be mounted on multiple machines if it is read-only

Even if the persistent volume has the RWX capability (read write many; many pods can mount to read and write simultaneously), such as NFS, it won't complain if multiple pods try to bind the same volume. However, we still need to consider whether multiple application instances can use the same folder/file or not. For example, if it replicates Grafana to two or more pods, it will be conflicted, with multiple Grafana instances that try to write to the same /var/lib/grafana/grafana.db, and then the data could be corrupted, as shown in the following diagram:

In this scenario, Grafana must use backend databases such as MySQL or PostgreSQL, instead of SQLite3, as follows. It allows multiple Grafana instances to read/write Grafana metadata properly:

Because RDBMS basically supports connecting with multiple application instances via a network, this scenario is perfectly suited to being used by multiple pods. Note that Grafana supports using RDBMS as a backend metadata store; however, not all applications support RDBMS.

Therefore, the Kubernetes administrator needs to carefully monitor how an application behaves with volumes, and understand that in some use cases, just using a persistent volume may not help because of issues that might arise when scaling pods.

If multiple pods need to access the centralized volume, then consider using the database as previously shown, if applicable. On the other hand, if multiple pods need an individual volume, consider using StatefulSet.