Working local DNS for your libvirtd guests

Update 2017-12-25: possibly better way: Definitive solution to libvirt guest naming

This is basically just a copy&paste of commands from this great post: [Howto] Automated DNS resolution for KVM/libvirt guests with a local domain and Automatic DNS updates from libvirt guests which already saved me a lots of typing. So with my favorite domain:

Make libvirtd's dnsmasq to act as authoritative nameserver for example.com domain:

# virsh net-dumpxml default
<network>
  <name>default</name>
  <uuid>2ed15952-d1c0-4819-bde5-c8f7278ce3ac</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:a4:40:a7'/>
  <domain name='example.com' localOnly='yes'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

And restart that network:

# virsh net-edit default   # do the edits here
# virsh net-destroy default
# virsh net-start default

Now configure NetworkManager to start its own dnsmasq which acts like your local caching nameserver and forwards all requests for example.com domain to 192.168.122.1 nameserver (which is libvirtd's dnsmasq):

# cat /etc/NetworkManager/conf.d/localdns.conf
[main]
dns=dnsmasq
# cat /etc/NetworkManager/dnsmasq.d/libvirt_dnsmasq.conf
server=/example.com/192.168.122.1

And restart NetworkManager:

# systemctl restart NetworkManager

Now if I have guest with hostname set (check HOSTNAME=... in /etc/sysconfig/network on RHEL6 and below or hostnamectl set-hostname ... on RHEL7) to "satellite.example.com", I can ping it from both virtualization host and another guests on that host by hostname. If you have some old OS release on the guest (like RHEL 6.5 from what I have tried, 6.8 do not need this), set hostname with DHCP_HOSTNAME=... in /etc/sysconfig/network-scripts/ifcfg-eth0 (on the guest) to make this to work.

DNS and "next-server" in DHCP configuration on libvirt's dnsmasq

I was playing with Satellite and re-provisioning client registered to it. This is awkward when you do it remotely and on real hardware - for me it is difficult to setup (if you want DNS and DHCP) and when client fails during re-provisioning, you either have to have physical access to it, or client have to have some kind of remote management console. Using libvirt is, on the other hand, very straightforward and you can get DNS and DHCP for free.

# virsh net-edit --network default
<network>
  <name>default</name>
  <uuid>970b7e2e-88d1-4100-8a2a-8db36c911d4c</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:f1:e9:9a'/>
  <dns>
    <host ip='192.168.122.46'>
      <hostname>sat-emb.example.com</hostname>
    </host>
    <host ip='192.168.122.170'>
      <hostname>proxy.example.com</hostname>
    </host>
    <host ip='192.168.122.25'>
      <hostname>client.example.com</hostname>
    </host>
  </dns>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
      <bootp file='/pxelinux.0' server='192.168.122.46'/>
    </dhcp>
  </ip>
</network>

<network><dns> configures hostname and their IPs form domain name resolution.

<network><ip><dhcp><bootp> allows me to set server which serves as PXE network boot server and file it clients should request. In my case, 192.168.122.46 is a Satellite with tftp running and configured.

NOTE: I have noticed that guests can not translate outer world hostnames to IPs - it looked like dnsmasq on the virtualization host is not forwarding requests it can not resolve to DNS servers from /etc/resolv.conf. Adding "<dns><forwarder addr="ip.of.another.nameserver" domain="internal.network.com"/>..." and restarting the network did not helped. At the end I have discovered there is forgotten no-resolv option in /var/lib/libvirt/dnsmasq/default.conf. When I have removed it and restarted network to regenerate config, it worked. I have probably forgotten it there in some previous adventures. From dnsmasq manual page:

       -R, --no-resolv
              Don't read /etc/resolv.conf. Get upstream servers
              only from the command line or the dnsmasq
              configuration file.

Running dockerd on VM so containers can be reached from other VMs

Recently I needed this kind of setup for some testing so wanted to share. This way, all your libvirt guests can talk directly to all your docker containers and vice versa. All nicely isolated on one system. All involved pieces are RHEL7.

schema of docker containers running in libvirt/KVM guest, all on one network

It is not perfect (I'm weak at networking), so you can get IP assigned to your container by dockerd conflicting with some VM IP. This is because docker assigns IPs from defined range (sequentially) and VMs have random IPs from same range assigned by libvirtd. Also I have seen some disconnects from Docker VM when starting containers there and sshing to container from docker VM was also lagging.
Libvirt is just a default configuration with its default network.
On one of the guests I have installed Docker (on RHEL7 it is in rhel-7-server-extras-rpms repository) and changed it's configuration to use (to-be created) custom bridge:

[root@docker1 ~]# grep ^OPTIONS /etc/sysconfig/docker
OPTIONS='--selinux-enabled -b=bridge0'

As I already started Docker, I wanted to remove it's default docker0 bridge it created, so simply:

[root@docker1 ~]# ip link set docker0 down   # first bring it down
[root@docker1 ~]# brctl delbr docker0   # delete it (brctl is in bridge-utils package)

Now to create new bridge which will get "public" IP (in a scope of libvirt's network) assigned:

[root@docker1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 
DEVICE="eth0"
BRIDGE="bridge0"
HWADDR="52:54:00:13:76:b5"
ONBOOT="yes"
[root@docker1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bridge0 
DEVICE=bridge0
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes
DELAY=0
[root@docker1 ~]# service network restart
[root@docker1 ~]# service docker restart

This way containers get IPs from same range virtual machines.