Resetting the local Docker network database

With the advent of user-defined networks, users are able to define custom network types for their containers. Once defined, these networks persist through system reboots until they are deleted by an administrator. In order for this persistence to work, Docker needs some place to store information related to your user-defined networks. The answer is a database file that's local to the host. In some rare cases, this database can get out of sync with the current state of the containers on the host or become corrupt. This can cause issues related to deleting containers, deleting networks, and starting the Docker service. In this recipe, we'll show you how to remove the database to restore Docker back to its default network configuration.

Getting ready

In this recipe, we'll be using a single Docker network host. It is assumed that Docker is installed and in its default configuration. You'll also need root-level access in order to inspect and change the host's networking and firewall configuration.

How to do it…

Docker stores information related to user-defined networks in a database stored on the local host. This database is written to when networks are defined and read from when the service starts. In the rare case that this database gets out of sync or becomes corrupt, you can delete the database and restart the Docker service in order to reset the Docker user-defined networks and restore the three default network types (bridge, host, and none).

Note

Warning: Deleting this database deletes any and all Docker user-defined networks on the host. It is wise to only do this as a last resort and if you have the capability of recreating the networks that were previously defined. All other troubleshooting options should be pursued before attempting this and you should create a backup of the file before deleting it.

The database is named local-kv.db and is stored in the path /var/lib/network/files/. Accessing and or deleting the file requires root-level access. For the purpose of this example, we'll switch to the root user in order to make browsing this protected directory easier:

user@docker1:~$ sudo su
[sudo] password for user:
root@docker1:/home/user# cd /var/lib/docker/network/files
root@docker1:/var/lib/docker/network/files# ls -al
total 72
drwxr-x--- 2 root root 32768 Aug  9 21:27 .
drwxr-x--- 3 root root  4096 Apr  3 21:04 ..
-rw-r--r-- 1 root root 65536 Aug  9 21:27 local-kv.db
root@docker1:/var/lib/docker/network/files#

To demonstrate what happens when we delete this file, let's first create a new user-defined network and attach a container to it:

root@docker1:~# docker network create -d bridge mybridge
c765f1d24345e4652b137383839aabdd3b01b1441d1d81ad4b4e17229ddca7ac
root@docker1:~# docker run -d --name web1 --net mybridge jonlangemak/web_server_1
24a6497e99de9e114b617b65673a8a50492655e9869dbf7f7930dd7f9f930b5e
root@docker1:~#

Let's now delete the file local-db.kv:

root@docker1:/var/lib/docker/network/files# rm local-kv.db

While this has no immediate effect on the running container, it does prevent us from adding, removing, or starting new containers associated with this user-defined network:

root@docker1:/~# docker run -d --name web2 --net mybridge 
jonlangemak/web_server_2
2ef7e52f44c93412ea7eaa413f523020a65f1a9fa6fd6761ffa6edea157c2623
docker: Error response from daemon: failed to update store for object type *libnetwork.endpointCnt: Key not found in store.
root@docker1:~#

After deleting the boltdb database file, local-kv.db, you'll need to restart the Docker service in order for Docker to recreate it with the default settings:

root@docker1:/var/lib/docker/network/files# cd
root@docker1:~# systemctl restart docker
root@docker1:~# ls /var/lib/docker/network/files
local-kv.db
root@docker1:~# docker network ls
NETWORK ID          NAME                DRIVER
bfd1ba1175a9        none                null
0740840aef37        host                host
97cbc0e116d7        bridge              bridge
root@docker1:/var/lib/docker/network/files#

Now that the file is recreated, you'll once again be able to create user-defined networks. However, any containers that were attached to previously configure user-defined network will now fail to start:

root@docker1:~# docker start web1
Error response from daemon: network mybridge not found
Error: failed to start containers: web1
root@docker1:~#

This is expected behavior since Docker still believes that the container should have an interface on that network:

root@docker1:~# docker inspect web1
…<Additional output removed for brevity>…
            "Networks": {
                "mybridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "c765f1d24345e4652b137383839aabdd3b01b1441d1d81ad4b4e17229ddca7ac",
…<Additional output removed for brevity>…
root@docker1:~#

To remedy this problem, you have two options. First, you can recreate the user-defined network named mybridge using the same configuration options from when it was initially provisioned. If this doesn't work, your only alternative is to delete the container and restart a new instance referencing a newly created or default network.

Note

There has been some discussion on GitHub of newer versions of Docker supporting a --force flag when using the docker network disconnect subcommand. In version 1.10, this parameter exists, but still doesn't like that the user-defined network does not exist. If you're running a newer version, this might be worth trying as well.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset