DNS resolution for containers has always been rather straightforward. The container received the same DNS configuration as the host. However, with the advent of user-defined networks and the embedded DNS server, this has now become a little trickier. A common problem in many of the DNS issues I've seen is not understanding how the embedded DNS server works and how to validate that it's working correctly. In this recipe, we'll step through a container DNS configuration to validate which DNS server it is using to resolve specific namespaces.
In this recipe, we'll be using a single Docker host. It is assumed that Docker is installed and in its default configuration. You'll also need root-level access in order to inspect and change the host's networking and firewall configuration.
The standard DNS configuration for Docker without user-defined networks is to simply copy the DNS configuration from the host into the container. In these cases, the DNS resolution is straightforward:
user@docker1:~$ docker run -dP --name web1 jonlangemak/web_server_1 e5735b30ce675d40de8c62fffe28e338a14b03560ce29622f0bb46edf639375f user@docker1:~$ user@docker1:~$ docker exec web1 more /etc/resolv.conf # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN nameserver <your local DNS server> search lab.lab user@docker1:~$ user@docker1:~$ more /etc/resolv.conf nameserver <your local DNS server> search lab.lab user@docker1:~$
In these cases, all DNS requests will go straight to the defined DNS server. This means that our container can resolve any DNS records that our host can:
user@docker1:~$ docker exec -it web1 ping docker2.lab.lab -c 2 PING docker2.lab.lab (10.10.10.102): 48 data bytes 56 bytes from 10.10.10.102: icmp_seq=0 ttl=63 time=0.471 ms 56 bytes from 10.10.10.102: icmp_seq=1 ttl=63 time=0.453 ms --- docker2.lab.lab ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.453/0.462/0.471/0.000 ms user@docker1:~$
Coupled with the fact that Docker will masquerade this traffic to the IP address of the host itself makes this a simple and easily maintainable solution.
However, this gets a little trickier when we start using user-defined networks. This is because user-defined networks provide for container name resolution. That is, one container can resolve the name of another container without the use of static or manual host file entries and linking. This is a great feature, but it can cause some confusion if you don't understand how the container receives its DNS configuration. For instance, let's now create a new user-defined network:
user@docker1:~$ docker network create -d bridge mybridge1 e8afb0e506298e558baf5408053c64c329b8e605d6ad12efbf10e81f538df7b9 user@docker1:~$
Let's now start a new container named web2 on this network:
user@docker1:~$ docker run -dP --name web2 --net mybridge1 jonlangemak/web_server_2 1b38ad04c3c1be7b0f1af28550bf402dcde1515899234e4b09e482da0a560a0a user@docker1:~$
Now if we connect our existing web1 container to this bridge, we should find that web1 can resolve the container web2 by name:
user@docker1:~$ docker network connect mybridge1 web1 user@docker1:~$ docker exec -it web1 ping web2 -c 2 PING web2 (172.18.0.2): 48 data bytes 56 bytes from 172.18.0.2: icmp_seq=0 ttl=64 time=0.100 ms 56 bytes from 172.18.0.2: icmp_seq=1 ttl=64 time=0.086 ms --- web2 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.086/0.093/0.100/0.000 ms user@docker1:~$
The problem here is that in order to facilitate this, Docker had to change the DNS configuration of the web1 container. In doing so, it injects the embedded DNS server in the middle of a containers DNS request. So before, when we were talking directly to the hosts DNS server, we are now talking to the embedded DNS server:
user@docker1:~$ docker exec -t web1 more /etc/resolv.conf
search lab.lab
nameserver 127.0.0.11
options ndots:0
user@docker1:~$This is required for DNS resolution to containers to work, but it has an interesting side effect. The embedded DNS server reads the host's /etc/resolv.conf file and uses any name servers defined in that file as forwarders for the embedded DNS server. The net effect of this is that you don't notice the embedded DNS server since it's still forwarding requests it can't answer to the host's DNS server. However, it only programs these forwarders if they are defined. If they don't exist or are set to 127.0.0.1, then Docker programs the forwarders to be Google's public DNS server (8.8.8.8 and 8.4.4.4).
Although this makes good sense, there are rare circumstances in which your local DNS server happens to be 127.0.0.1. For instance, you happen to be running some type of local DNS resolver on the same host or using a DNS forwarder application such as DNSMasq. In those cases, there are some complications that can be caused by Docker forwarding the container's DNS requests off to the aforementioned external DNS servers instead of the one locally defined. Namely, internal DNS zones will no longer be resolvable:
user@docker1:~$ docker exec -it web1 ping docker2.lab.lab ping: unknown host user@docker1:~$
In these scenarios, there are a couple of ways to address this. You can either run the container with a specific DNS server by passing the DNS flag at container runtime:
user@docker1:~$ docker run -dP --name web2 --net mybridge1 --dns <your local DNS server> jonlangemak/web_server_2
Otherwise, you can set the DNS server at the Docker service level, which the embedded DNS server will then use as the forwarder:
ExecStart=/usr/bin/dockerd --dns=<your local DNS server>
In either case, if you're having container resolution issues, always check and see what the container has configured in its /etc/resolv.conf file. If it's 127.0.0.11, that indicates you're using the Docker embedded DNS server. If you are, and you're still having issues, make sure that you validate the host DNS configuration to determine what the embedded DNS server is consuming for a forwarder. If there isn't one defined or it's 127.0.0.1, then make sure that you tell the Docker service what DNS server it should be passing to containers in one of the two ways defined earlier.