Recovering RHEL in emergency mode in AWS EC2

OK. I'm sure everybody knows that, but I did not. When you have AWS EC2 instance, say c3.8xlarge, you will get 2 x 320 GB SSD storage. Thats nice, isn't it? But, besides you have to manually attach that storage when launching the machine (4. Add Storage -> Add New Volume -> Volume Type: Instance Store 0 and repeat for Instance Store 1), it gets purged every time you stop and start your instance. I did not knew that so I have created LVM on these and added physical volume to the /etc/fstab to be auto mounted on next boot:

# pvcreate --yes /dev/xvdb
# pvcreate --yes /dev/xvdc
# vgcreate mygroup /dev/xvdb /dev/xvdc
# lvcreate --size 1GB --name myvol mygroup
# echo "/dev/mapper/mygroup-myvol /mnt/test xfs defaults 0 0" >>/etc/fstab

Now, here comes the problem: on instance stop and start, RHEL notices there is now no logical volume and skips into emergency mode? Now what? Let's recover using second machine. Executive summary:

1. Take some working machine
2. Detach root device volume from broken machine
3. Attach it to working machine
4. From the working machine mount, fix fstab and umount
5. Detach
6. Attach (do not ask me why, but as a Device, I had to use /dev/sda1 instead of /dev/sda)

Because mine broken and working machines were created from same RHEL 7.2 image, when I have attempted to mount I got this:

# mount /dev/xvdf2 /mnt/tmp/
mount: wrong fs type, bad option, bad superblock on /dev/xvdf2,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
# dmesg | tail
[  265.755327] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
[  288.106763] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
[  568.495065] Adjusting xen more than 11% (9437184 vs 9311354)
[  583.752252] blkfront: xvdf: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled;
[  583.766118]  xvdf: xvdf1 xvdf2
[  662.933196] XFS (xvdf2): Filesystem has duplicate UUID 379de64d-ea11-4f5b-ae6a-0aa50ff7b24d - can't mount
[  752.706161] XFS (xvdf2): Filesystem has duplicate UUID 379de64d-ea11-4f5b-ae6a-0aa50ff7b24d - can't mount
[  842.806648] XFS (xvdf2): Filesystem has duplicate UUID 379de64d-ea11-4f5b-ae6a-0aa50ff7b24d - can't mount
[  879.618806] XFS (xvdf): Invalid superblock magic number
[  884.951716] XFS (xvdf2): Filesystem has duplicate UUID 379de64d-ea11-4f5b-ae6a-0aa50ff7b24d - can't mount

So I had to mount with -o nouuid option.

Enabling 10Gb networking on RHEL7 in AWS cloud (some instances types)

Even when you order recent RHEL 7.2 in some beefy Amazon EC2 instance type (like "c4.8xlarge") which have "enhanced networking capabilities", you still have to do some manual steps described in Amazon's docs (note there are actually two ways - based on the instance type you will choose). Basically you need newer network driver than what is in default installation. First check your current driver:

We would like to have "ixgbevf" here:

# ethtool -i eth0 | grep '^driver'
driver: vif

And here we would like to have at least "2.14.2" (ignore my actual version):

# modinfo ixgbevf | grep '^version'
version:        2.12.1-k-rh7.3

So lets go on (as root on the instance) - we will take newest version in Intel Ethernet Drivers and Utilities:

curl -o ixgbevf-3.2.2.tar.gz 'http://netcologne.dl.sourceforge.net/project/e1000/ixgbevf%20stable/3.2.2/ixgbevf-3.2.2.tar.gz'   # URL might be different for you, follow the download button on the SourceForge site
yum -y install kernel-devel gcc rpm-build   # we will need these to compile
rpmbuild -tb ixgbevf-3.2.2.tar.gz   # specfile is inside of the tarball
rpm -ivh /root/rpmbuild/RPMS/x86_64/ixgbevf-3.2.2-1.x86_64.rpm   # install resulting rpm
cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.$(date +%m-%d-%H%M%S).ORIG   # backup current initrd
dracut -f -v   # rebuild initrd so it contains our module
shutdown -h now   # stop the instance

Now we will need to run one command via aws command line tool. If you do not have it installed, like me (I'm on Fedora 24), you can install it with:

mkdir aws
cd aws
virtualenv .
. bin/activate
pip install aws
aws configure

And now we can finally run the command which enables some fancy attribute for your instance (they say in the docs this can not be undone):

aws ec2 modify-instance-attribute --instance-id <instanceID> --sriov-net-support simple

Now start the instance. Warning: instance had different public IP (and DNS hostname) now in my case, so do not blindly attempt to connect to previous hostname. Lets check what we have on the system now:

# modinfo ixgbevf | grep '^version'
version:        3.2.2
# ethtool -i eth0 | grep -e '^driver' -e '^version'
driver: ixgbevf
version: 3.2.2

This looks good, although I have not tested real performance yet. Going to turn off that expensive machine now :-)