During some load testing work we had a big problems with too big disk consumption when on Devicemapper direct LVM - our use-case of very many small (cpu/ram-vise) containers is not good for this storage driver. We have experimented and compared Docker 1.10 (docker-1.10.3-59.el7.x86_64) with Devicemapper direct LVM vs. OverlayFS (with XFS as a "hosting" FS) and there are some random numbers in this post. Docker also have page comparing different storage drivers. Red Hat documentation warns about OverlayFS stability, so we will see how it goes later in real load.
Configuring Docker for Devicemapper direct LVM
You need LVM volume group with free space for this.systemctl stop docker :>/etc/sysconfig/docker-storage rm -rf /var/lib/docker/* echo "VG='volume_group_with_free_space_for_docker'" >/etc/sysconfig/docker-storage-setup docker-storage-setup systemctl start docker
Configuring Docker for OverlayFS
Docker's documentation advises to use separate partition for OverlayFS as this storage driver consumes lots of inodes ("overlay2" should be better for that, but afaict this comes with Docker 1.12). We will see how it goes as we did not do any tunings when creating filesystem there. How to create XFS filesystem for OverlayFS and Changing Storage Configuration and "Overlay Graph Driver" below:systemctl stop docker sed -i '/OPTIONS=/s/--selinux-enabled//' /etc/sysconfig/docker :>/etc/sysconfig/docker-storage rm -rf /var/lib/docker/* lvcreate --name docker --extents 100%FREE volume_group_with_free_space_for_docker mkfs -t xfs -n ftype=1 /dev/volume_group_with_free_space_for_docker/docker echo "/dev/volume_group_with_free_space_for_docker/docker /var/lib/docker xfs defaults 0 0" >>/etc/fstab mount /var/lib/docker echo "STORAGE_DRIVER='overlay'" >/etc/sysconfig/docker-storage-setup docker-storage-setup systemctl start docker
Comparing Devicemapper direct LVM vs. OverlayFS
This is the container we are using.
Starting containers
# time for i in $(seq 10); do docker run -h "con$i.example.com" -d r7perfsat; done Devicemapper direct LVM: 38.832s OverlayFS: 7.530s
Inspecting container sizes
# time docker inspect --size -f "SizeRw={{.SizeRw}} SizeRootFs={{.SizeRootFs}}" some_container
Devicemapper direct LVM: 18.254s
SizeRw=2917 SizeRootFs=1633247414
OverlayFS: 2.694s
SizeRw=2921 SizeRootFs=1633247406
Note: not sure if above is a relevant test case.
Stopping containers
# time docker stop 10_containers_we_have_created_before Devicemapper direct LVM: 4.888s OverlayFS: 3.266s
Removing stopped containers
# time docker rm 10_containers_we_have_created_before Devicemapper direct LVM: 2.289s OverlayFS: 0.206s
Registering containers
Running ansible playbook which registers via subscription-manager to Satellite 6 and installs Katello Agent and does few quick changes to the container on 5 of these containers in parallel:
Devicemapper direct LVM: 2m54.169s OverlayFS: 2m45.890s
Note that during this load on the docker host seems much smaller on OverlayFS enable docker host, but that night be because of lots of other containers there are running on the host.
Downgrading few packages in containers
Running ansible playbook which downgrades few most small packages on 5 of these containers in parallel:
Devicemapper direct LVM: 1m34.035s OverlayFS: 1m1.685s
At first glance, OverlayFS seems faster.
