With the advent of user-defined networks, users are able to define custom network types for their containers. Once defined, these networks persist through system reboots until they are deleted by an administrator. In order for this persistence to work, Docker needs some place to store information related to your user-defined networks. The answer is a database file that's local to the host. In some rare cases, this database can get out of sync with the current state of the containers on the host or become corrupt. This can cause issues related to deleting containers, deleting networks, and starting the Docker service. In this recipe, we'll show you how to remove the database to restore Docker back to its default network configuration.
In this recipe, we'll be using a single Docker network host. It is assumed that Docker is installed and in its default configuration. You'll also need root-level access in order to inspect and change the host's networking and firewall configuration.
Docker stores information related to user-defined networks in a database stored on the local host. This database is written to when networks are defined and read from when the service starts. In the rare case that this database gets out of sync or becomes corrupt, you can delete the database and restart the Docker service in order to reset the Docker user-defined networks and restore the three default network types (bridge, host, and none).
Warning: Deleting this database deletes any and all Docker user-defined networks on the host. It is wise to only do this as a last resort and if you have the capability of recreating the networks that were previously defined. All other troubleshooting options should be pursued before attempting this and you should create a backup of the file before deleting it.
The database is named local-kv.db
and is stored in the path /var/lib/network/files/
. Accessing and or deleting the file requires root-level access. For the purpose of this example, we'll switch to the root user in order to make browsing this protected directory easier:
user@docker1:~$ sudo su [sudo] password for user: root@docker1:/home/user# cd /var/lib/docker/network/files root@docker1:/var/lib/docker/network/files# ls -al total 72 drwxr-x--- 2 root root 32768 Aug 9 21:27 . drwxr-x--- 3 root root 4096 Apr 3 21:04 .. -rw-r--r-- 1 root root 65536 Aug 9 21:27 local-kv.db root@docker1:/var/lib/docker/network/files#
To demonstrate what happens when we delete this file, let's first create a new user-defined network and attach a container to it:
root@docker1:~# docker network create -d bridge mybridge c765f1d24345e4652b137383839aabdd3b01b1441d1d81ad4b4e17229ddca7ac root@docker1:~# docker run -d --name web1 --net mybridge jonlangemak/web_server_1 24a6497e99de9e114b617b65673a8a50492655e9869dbf7f7930dd7f9f930b5e root@docker1:~#
Let's now delete the file local-db.kv
:
root@docker1:/var/lib/docker/network/files# rm local-kv.db
While this has no immediate effect on the running container, it does prevent us from adding, removing, or starting new containers associated with this user-defined network:
root@docker1:/~# docker run -d --name web2 --net mybridge
jonlangemak/web_server_2
2ef7e52f44c93412ea7eaa413f523020a65f1a9fa6fd6761ffa6edea157c2623
docker: Error response from daemon: failed to update store for object type *libnetwork.endpointCnt: Key not found in store.
root@docker1:~#
After deleting the boltdb
database file, local-kv.db
, you'll need to restart the Docker service in order for Docker to recreate it with the default settings:
root@docker1:/var/lib/docker/network/files# cd root@docker1:~# systemctl restart docker root@docker1:~# ls /var/lib/docker/network/files local-kv.db root@docker1:~# docker network ls NETWORK ID NAME DRIVER bfd1ba1175a9 none null 0740840aef37 host host 97cbc0e116d7 bridge bridge root@docker1:/var/lib/docker/network/files#
Now that the file is recreated, you'll once again be able to create user-defined networks. However, any containers that were attached to previously configure user-defined network will now fail to start:
root@docker1:~# docker start web1
Error response from daemon: network mybridge not found
Error: failed to start containers: web1
root@docker1:~#
This is expected behavior since Docker still believes that the container should have an interface on that network:
root@docker1:~# docker inspect web1
…<Additional output removed for brevity>…
"Networks": {
"mybridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "c765f1d24345e4652b137383839aabdd3b01b1441d1d81ad4b4e17229ddca7ac",
…<Additional output removed for brevity>…
root@docker1:~#
To remedy this problem, you have two options. First, you can recreate the user-defined network named mybridge
using the same configuration options from when it was initially provisioned. If this doesn't work, your only alternative is to delete the container and restart a new instance referencing a newly created or default network.
There has been some discussion on GitHub of newer versions of Docker supporting a --force
flag when using the docker network disconnect
subcommand. In version 1.10, this parameter exists, but still doesn't like that the user-defined network does not exist. If you're running a newer version, this might be worth trying as well.