# xhost local:rootstart Fedora 27 container with some extra vars and mounts:
# docker run \and then in the container install all needed and run the browser:
--network host \
-e DISPLAY=:0.0 \
-v /tmp/.X11-unix:/tmp/.X11-unix \
-v /root/.Xauthority:/root/.Xauthority:rw \
-ti fedora:27 /bin/bash
# dnf -y install firefox xorg-x11-xauth icedtea-webNow I'm able to open the console, hooray!
# firefox
This is quite simple, there is a lot of docs out there, so just to put it on one place I do not need to look for it next time I want to install this "full stack solution":
# subscription-manager repos --enable rhel-7-server-extras-rpms # yum install podman
# lvcreate data_perf54 --size 25G --name docker_registry # mkfs.xfs /dev/mapper/data_xyz-docker_registry # tail -n 1 /etc/fstab /dev/mapper/data_xyz-docker_registry /var/lib/registry xfs defaults 0 0 # mount /var/lib/registry # podman run --privileged -d --name registry-srv -p 5000:5000 -v /var/lib/registry:/var/lib/registry registry:2
# cat /etc/systemd/system/registry-srv-container.service [Unit] Description=Docker registry container [Service] Restart=always ExecStart=/usr/bin/podman start -a registry-srv ExecStop=/usr/bin/podman stop -t 30 registry-srv [Install] WantedBy=local.target # systemctl enable registry-srv-container.service # systemctl restart registry-srv-container.service # systemctl status registry-srv-container.service
# grep 'registries.insecure' -A 1 /etc/containers/registries.conf [registries.insecure] registries = ['your_hostname:5000'] # podman pull busybox # podman tag docker.io/library/busybox $( hostname ):5000/busybox # podman push $( hostname ):5000/busybox
# curl -s "http://$( hostname ):5000/v2/_catalog?n=100" | json_reformat
{
"repositories": [
"busybox"
]
}
I had a task to add an assert to Ansible playbook to check that root / filesystem supports d_type (i.e. "directory entry type" and is important for Docker / Podman). Here is the result:
- name: Read root filesystem type and device
set_fact:
root_fstype: "{{ ansible_mounts | selectattr('mount', 'equalto', '/') | map(attribute='fstype') | join(',') }}"
root_device: "{{ ansible_mounts | selectattr('mount', 'equalto', '/') | map(attribute='device') | join(',') }}"
- name: If root filesystem is xfs, get more info from it
command:
xfs_info "{{ root_device }}"
register: xfs_info
ignore_errors: true
when: "root_fstype != 'ext4'"
- name: Check that root filesystem supports directory entry type (aka d_type)
assert:
that:
- "root_fstype == 'ext4' or ( root_fstype == 'xfs' and 'ftype=1' in xfs_info.stdout )"
First, we extract (more info on how we get one value from a list of dicts based on another value) root filesystem type (e.g. "ext4" or "xfs") and device (e.g. /dev/mapper/centos_something-root) from Ansible facts obtained by setup module (use ansible -u root -i inventory.ini -m setup all to see all the facts). Then we load additional info by xfs_info utility if the fs type is "xfs". And last step is finally to assert for d_type support: "ext4" is a clear win, when we got "xfs", "ftype=1" in xfs_info output is needed.
$ cat Singularity
Bootstrap: docker
From: fedora:latest
%help
This is a container for ... project
See https://gitlab.com/...
Email ...
%labels
Homepage https://gitlab.com/...
Author ...
Maintainer ...
Version 0.1
%files
/home/where/is/your/project /projectX
%post
dnf -y install python2-biopython python2-numpy python2-tabulate python2-scikit-learn pymol mono-core python2-unittest2 python2-svgwrite python2-requests
chown -R 1000:1000 /projectX # probably not important
%test
cd /projectX
python -m unittest discover
%environment
export LC_ALL=C
%runscript
exec /projectX/worker.sh
Majority of above is not needed: e.g. "%help" have completely free form, keys in "%labels" does not seem to be codified, "%test" which is ran as a last step of a build process is also optional. To build it:
$ sudo singularity build --writable projectXy.simg Singularity # *.simg is a native format of singularity-2.4.6 $ sudo singularity build --writable projectX.img projectX.simg # where the project is supposed to run, there is 2.3.2 which needs older *.img, so convert *.simg into itMine original idea was to have the project in writable container (so the option "--writable" above), but that would require me to run it as root again (or I'm missing something), so I have ended up with the solution of running the container in read-only mode and mounting mine project into it to have a read-write-able directory where I can generate the data:
$ echo "cd /projectX; ./worker.sh" \
| singularity exec --bind projectX/:/projectX projectX.img bash
So far it looks like it just works.
For some time I'm running publicly available web application Brno People team uses to determine technical interests of employee candidates. The app was running on the OpenShift v.2, but that was discontinued and I had to port it to OpenShift v.3. I was postponing the task for multiple moths and got to the state when v.2 instances were finally discontinued. It turned out that porting is not that hard. This is what I have done.
Note that I'm using Red Hat's Employee account, so some paths might be different when OpenShift is being used "normally" (you will see something like ....starter-us-east-1.openshift.com instead of mine ....rh-us-east-1.openshift.com).
Because I need to put some private data into the image, I want the image to be accessible only from mine OpenShift Online account. Anyway, I have created Dockerfile based on Fedora Dockerfile template for Python (is it official?) like this:
FROM fedora MAINTAINER Jan Hutar <jhutar@redhat.com> RUN dnf -y update && dnf clean all RUN dnf -y install subversion python mod_wsgi && dnf clean all RUN ... VOLUME ["/xyz/data/"] WORKDIR /xyz EXPOSE 8080 USER root CMD ["python", "/xyz/application"]
TODO for the container is: move to Python 3 (so I do not need to install python2 and dependencies, figure out how to have the private data available to the container but not being part of it (it is quite big directory structure), go through these nice General Container Image Guidelines and explore this Image Metadata thingy.
Once I had that, I needed to login to the OpenShift's registry, build my image locally, test it and push it:
sudo docker build --tag selftest . sudo docker run -ti --publish 8080:8080 --volume $( pwd )/data/:/xyz/data/ xyz # now I can check if all looks sane with `firefox http://localhost:8080` oc whoami -t # this shows token I can use below sudo docker login -u <username> -p <token> registry.rh-us-east-1.openshift.com sudo docker tag xyz registry.rh-us-east-1.openshift.com/xyz/xyz sudo docker push registry.rh-us-east-1.openshift.com/xyz/xyz
Now I have used Console on https://console.rh-us-east-1.openshift.com/ to create new application, then added a deployment to that application with Add to Project -> Deploy Image and selected (well, I could use cli tool oc for that):
Next step looks logical, but I got stuck on it for some time, but OpenShift folks helped me (thanks Jiří!). I just need to be aware of OpenShift Online Restrictions.
So, because I wanted persistent storage and because my account uses Amazon EC2, I can not use "Shared Access (RWX)" storage type (useful when new pod is starting while old pod is still running), I had to change process of new pods start to first stop old and the start new: Applications -> Deployments -> my deployment -> Actions -> Edit -> Strategy Type: Recreate. I have created a storage with "RWO (Read-Write-Once)" access mode, added it to the deployment (... -> Actions -> Add Storage) and made sure that that storage is the only one attached to the deployment (... -> Actions -> Edit YAML and check that keys spec.template.spec.containers.env.volumeMounts and spec.template.spec.volumes only contain one volume you have just attached). In my case, there is this in the YAML definition:
[...]
spec:
containers:
- env:
[...]
volumeMounts:
- mountPath: /xyz/data
name: volume-jt8t6
[...]
volumes:
- name: volume-jt8t6
persistentVolumeClaim:
claimName: xyz-storage-claim
[...]
When working with this, I have also used ... -> Actions -> Pause Rollouts. It is also possible to configure environment variable for a deployment in ... -> Actions -> Edit -> Environment Variables which is useful to pass passwords and stuff into your app (so I do not need to store them in the image). In the app I use something like import os; SMTP_SERVER_PASSWORD = os.getenv('XYZ_SMTP_SERVER_PASSWORD', default='') to read that.
To make the app available from outside world, I have created a route in Applications -> Routes -> Create Route. It created domain like http://xyz-route-xyz.6923.rh-us-east-1.openshiftapps.com for me.
Now, looks like everything works for me and I'm kinda surprised how easy it was. I plan to get nicer domain and configure its CNAME DNS record and to explore monitoring possibilities OpenShift have. I'll see how it goes.
We have container build from this Dockerfile, running RHEL7 with oldish docker-1.10.3-59.el7.x86_64. Containers are started with:
# for i in $( seq 500 ); do
docker run -h "$( hostname -s )container$i.example.com" -d --tmpfs /tmp --tmpfs /run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --ulimit nofile=10000:10000 r7perfsat
done
and we have set limits for a docker service on a docker host:
# cat /etc/systemd/system/docker.service.d/limits.conf [Service] LimitNOFILE=10485760 LimitNPROC=10485760
but we have still seen issues with "Too many open files" inside the container. It could happen when installing package with yum (resulting into corrupted rpm database, rm -rf /var/lib/rpm/__db.00*; rpm --rebuilddb; saved it though) and when enabling service (our containers have systemd in them on purpose):
# systemctl restart osad Error: Too many open files # echo $? 0
Because I was stupid, I have not checked journal (in the container) in the moment when I have spotted the failure for the first time:
Dec 21 10:18:54 b08-h19-r620container247.example.com journalctl[39]: Failed to create inotify watch: Too many open files Dec 21 10:18:54 b08-h19-r620container247.example.com systemd[1]: systemd-journal-flush.service: main process exited, code=exited, status=1/FAILURE Dec 21 10:18:54 b08-h19-r620container247.example.com systemd[1]: inotify_init1() failed: Too many open files Dec 21 10:18:54 b08-h19-r620container247.example.com systemd[1]: inotify_init1() failed: Too many open files
At the end I have ran into some issue and very last comment there had a think I have not seen before. At the end I have ended up with:
# cat /etc/sysctl.d/40-max-user-watches.conf fs.inotify.max_user_instances=8192 fs.inotify.max_user_watches=1048576
Default on a different machine is:
# sysctl -a 2>&1 | grep fs.inotify.max_user_ fs.inotify.max_user_instances = 128 fs.inotify.max_user_watches = 8192
Looks like increasing fs.inotify.max_user_instances helped and our containers are stable.
During some load testing work we had a big problems with too big disk consumption when on Devicemapper direct LVM - our use-case of very many small (cpu/ram-vise) containers is not good for this storage driver. We have experimented and compared Docker 1.10 (docker-1.10.3-59.el7.x86_64) with Devicemapper direct LVM vs. OverlayFS (with XFS as a "hosting" FS) and there are some random numbers in this post. Docker also have page comparing different storage drivers. Red Hat documentation warns about OverlayFS stability, so we will see how it goes later in real load.
systemctl stop docker :>/etc/sysconfig/docker-storage rm -rf /var/lib/docker/* echo "VG='volume_group_with_free_space_for_docker'" >/etc/sysconfig/docker-storage-setup docker-storage-setup systemctl start docker
systemctl stop docker sed -i '/OPTIONS=/s/--selinux-enabled//' /etc/sysconfig/docker :>/etc/sysconfig/docker-storage rm -rf /var/lib/docker/* lvcreate --name docker --extents 100%FREE volume_group_with_free_space_for_docker mkfs -t xfs -n ftype=1 /dev/volume_group_with_free_space_for_docker/docker echo "/dev/volume_group_with_free_space_for_docker/docker /var/lib/docker xfs defaults 0 0" >>/etc/fstab mount /var/lib/docker echo "STORAGE_DRIVER='overlay'" >/etc/sysconfig/docker-storage-setup docker-storage-setup systemctl start docker
This is the container we are using.
# time for i in $(seq 10); do docker run -h "con$i.example.com" -d r7perfsat; done Devicemapper direct LVM: 38.832s OverlayFS: 7.530s
# time docker inspect --size -f "SizeRw={{.SizeRw}} SizeRootFs={{.SizeRootFs}}" some_container
Devicemapper direct LVM: 18.254s
SizeRw=2917 SizeRootFs=1633247414
OverlayFS: 2.694s
SizeRw=2921 SizeRootFs=1633247406
Note: not sure if above is a relevant test case.
# time docker stop 10_containers_we_have_created_before Devicemapper direct LVM: 4.888s OverlayFS: 3.266s
# time docker rm 10_containers_we_have_created_before Devicemapper direct LVM: 2.289s OverlayFS: 0.206s
Running ansible playbook which registers via subscription-manager to Satellite 6 and installs Katello Agent and does few quick changes to the container on 5 of these containers in parallel:
Devicemapper direct LVM: 2m54.169s OverlayFS: 2m45.890s
Note that during this load on the docker host seems much smaller on OverlayFS enable docker host, but that night be because of lots of other containers there are running on the host.
Running ansible playbook which downgrades few most small packages on 5 of these containers in parallel:
Devicemapper direct LVM: 1m34.035s OverlayFS: 1m1.685s
At first glance, OverlayFS seems faster.
 |
| schema of docker containers running in libvirt/KVM guest, all on one network |
[root@docker1 ~]# grep ^OPTIONS /etc/sysconfig/docker
OPTIONS='--selinux-enabled -b=bridge0'Now to create new bridge which will get "public" IP (in a scope of libvirt's network) assigned:[root@docker1 ~]# ip link set docker0 down # first bring it down [root@docker1 ~]# brctl delbr docker0 # delete it (brctl is in bridge-utils package)
[root@docker1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
BRIDGE="bridge0"
HWADDR="52:54:00:13:76:b5"
ONBOOT="yes"
[root@docker1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bridge0
DEVICE=bridge0
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes
DELAY=0
[root@docker1 ~]# service network restart
[root@docker1 ~]# service docker restart