Docker 1.10 with Devicemapper direct LVM vs. OverlayFS

During some load testing work we had a big problems with too big disk consumption when on Devicemapper direct LVM - our use-case of very many small (cpu/ram-vise) containers is not good for this storage driver. We have experimented and compared Docker 1.10 (docker-1.10.3-59.el7.x86_64) with Devicemapper direct LVM vs. OverlayFS (with XFS as a "hosting" FS) and there are some random numbers in this post. Docker also have page comparing different storage drivers. Red Hat documentation warns about OverlayFS stability, so we will see how it goes later in real load.

Configuring Docker for Devicemapper direct LVM

You need LVM volume group with free space for this.

systemctl stop docker
:>/etc/sysconfig/docker-storage
rm -rf /var/lib/docker/*
echo "VG='volume_group_with_free_space_for_docker'" >/etc/sysconfig/docker-storage-setup
docker-storage-setup
systemctl start docker

Configuring Docker for OverlayFS

Docker's documentation advises to use separate partition for OverlayFS as this storage driver consumes lots of inodes ("overlay2" should be better for that, but afaict this comes with Docker 1.12). We will see how it goes as we did not do any tunings when creating filesystem there. How to create XFS filesystem for OverlayFS and Changing Storage Configuration and "Overlay Graph Driver" below:

systemctl stop docker
sed -i '/OPTIONS=/s/--selinux-enabled//' /etc/sysconfig/docker
:>/etc/sysconfig/docker-storage
rm -rf /var/lib/docker/*
lvcreate --name docker --extents 100%FREE volume_group_with_free_space_for_docker
mkfs -t xfs -n ftype=1 /dev/volume_group_with_free_space_for_docker/docker
echo "/dev/volume_group_with_free_space_for_docker/docker /var/lib/docker xfs defaults 0 0" >>/etc/fstab
mount /var/lib/docker
echo "STORAGE_DRIVER='overlay'" >/etc/sysconfig/docker-storage-setup
docker-storage-setup
systemctl start docker

Comparing Devicemapper direct LVM vs. OverlayFS

This is the container we are using.

Starting containers

# time for i in $(seq 10); do docker run -h "con$i.example.com" -d r7perfsat; done
Devicemapper direct LVM: 38.832s
OverlayFS: 7.530s

Inspecting container sizes

# time docker inspect --size -f "SizeRw={{.SizeRw}} SizeRootFs={{.SizeRootFs}}" some_container
Devicemapper direct LVM: 18.254s
  SizeRw=2917 SizeRootFs=1633247414
OverlayFS: 2.694s
  SizeRw=2921 SizeRootFs=1633247406

Note: not sure if above is a relevant test case.

Stopping containers

# time docker stop 10_containers_we_have_created_before
Devicemapper direct LVM: 4.888s
OverlayFS: 3.266s

Removing stopped containers

# time docker rm 10_containers_we_have_created_before
Devicemapper direct LVM: 2.289s
OverlayFS: 0.206s

Registering containers

Running ansible playbook which registers via subscription-manager to Satellite 6 and installs Katello Agent and does few quick changes to the container on 5 of these containers in parallel:

Devicemapper direct LVM: 2m54.169s
OverlayFS: 2m45.890s

Note that during this load on the docker host seems much smaller on OverlayFS enable docker host, but that night be because of lots of other containers there are running on the host.

Downgrading few packages in containers

Running ansible playbook which downgrades few most small packages on 5 of these containers in parallel:

Devicemapper direct LVM: 1m34.035s
OverlayFS: 1m1.685s

At first glance, OverlayFS seems faster.