2016-05-09

Running dockerd on VM so containers can be reached from other VMs

Recently I needed this kind of setup for some testing so wanted to share. This way, all your libvirt guests can talk directly to all your docker containers and vice versa. All nicely isolated on one system. All involved pieces are RHEL7.
Intentional schema of docker containers running in libvirt/KVM guest, all on one network
schema of docker containers running in libvirt/KVM guest, all on one network
It is not perfect (I'm weak at networking), so you can get IP assigned to your container by dockerd conflicting with some VM IP. This is because docker assigns IPs from defined range (sequentially) and VMs have random IPs from same range assigned by libvirtd. Also I have seen some disconnects from Docker VM when starting containers there and sshing to container from docker VM was also lagging.
Libvirt is just a default configuration with its default network.
On one of the guests I have installed Docker (on RHEL7 it is in rhel-7-server-extras-rpms repository) and changed it's configuration to use (to-be created) custom bridge:
[root@docker1 ~]# grep ^OPTIONS /etc/sysconfig/docker
OPTIONS='--selinux-enabled -b=bridge0'
As I already started Docker, I wanted to remove it's default docker0 bridge it created, so simply:
[root@docker1 ~]# ip link set docker0 down   # first bring it down
[root@docker1 ~]# brctl delbr docker0   # delete it (brctl is in bridge-utils package)
Now to create new bridge which will get "public" IP (in a scope of libvirt's network) assigned:
[root@docker1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 
DEVICE="eth0"
BRIDGE="bridge0"
HWADDR="52:54:00:13:76:b5"
ONBOOT="yes"
[root@docker1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bridge0 
DEVICE=bridge0
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes
DELAY=0
[root@docker1 ~]# service network restart
[root@docker1 ~]# service docker restart
This way containers get IPs from same range virtual machines.