2018-04-18

Creating Singularity container

For some project I had to create a Singularity container (because of the environment where it needs to run). Singularity is a container technology used by scientists. It turned out that it is very simple. Although there is not recent enough Singularity package in the normal Fedora repos, they have nice guide on how to build your own packages and that worked for me on a first try. First "Singularity" build file (similar to "Dockerfile" file) - I have based it on a recent Fedora docker image:
$ cat Singularity 
Bootstrap: docker
From: fedora:latest

%help
    This is a container for ... project
    See https://gitlab.com/...
    Email ...

%labels
    Homepage https://gitlab.com/...
    Author ...
    Maintainer ...
    Version 0.1

%files
    /home/where/is/your/project /projectX

%post
    dnf -y install python2-biopython python2-numpy python2-tabulate python2-scikit-learn pymol mono-core python2-unittest2 python2-svgwrite python2-requests
    chown -R 1000:1000 /projectX   # probably not important

%test
    cd /projectX
    python -m unittest discover

%environment
    export LC_ALL=C

%runscript
    exec /projectX/worker.sh
Majority of above is not needed: e.g. "%help" have completely free form, keys in "%labels" does not seem to be codified, "%test" which is ran as a last step of a build process is also optional. To build it:
$ sudo singularity build --writable projectXy.simg Singularity   # *.simg is a native format of singularity-2.4.6
$ sudo singularity build --writable projectX.img projectX.simg   # where the project is supposed to run, there is 2.3.2 which needs older *.img, so convert *.simg into it
Mine original idea was to have the project in writable container (so the option "--writable" above), but that would require me to run it as root again (or I'm missing something), so I have ended up with the solution of running the container in read-only mode and mounting mine project into it to have a read-write-able directory where I can generate the data:
$ echo "cd /projectX; ./worker.sh" \
      | singularity exec --bind projectX/:/projectX projectX.img bash
So far it looks like it just works.

2018-02-08

TaskWarior and listing tasks completed in specified time range

Besides mountain of inefficient papers (good thing about paper is you can lose it easily), I'm using TaskWarrior to manage my TODOs. Today I'm creating list of things I have finished in last (fiscal) year, so its reporting and filtering capabilities come handy. But as not too advanced user, it took me some time to discover how to list tasks completed in specified time range:

$ task end.after:2017-03-01 end.before:2018-03-01 completed

ID UUID     Created    Completed  Age  P Project         Tags R Due        Description                                                                                                                                                        
[...]                                                                                                                   
 - 5248839d 2017-01-24 2017-03-20 1.0y H                                   Prepare 'Investigating differences in code coverage reports' presentation
[...]

2018-02-01

Running wsgi application in OpenShift v.3 for the first time

For some time I'm running publicly available web application Brno People team uses to determine technical interests of employee candidates. The app was running on the OpenShift v.2, but that was discontinued and I had to port it to OpenShift v.3. I was postponing the task for multiple moths and got to the state when v.2 instances were finally discontinued. It turned out that porting is not that hard. This is what I have done.

Note that I'm using Red Hat's Employee account, so some paths might be different when OpenShift is being used "normally" (you will see something like ....starter-us-east-1.openshift.com instead of mine ....rh-us-east-1.openshift.com).

Because I need to put some private data into the image, I want the image to be accessible only from mine OpenShift Online account. Anyway, I have created Dockerfile based on Fedora Dockerfile template for Python (is it official?) like this:

FROM fedora
MAINTAINER Jan Hutar <jhutar@redhat.com>
RUN dnf -y update && dnf clean all
RUN dnf -y install subversion python mod_wsgi && dnf clean all
RUN ...
VOLUME ["/xyz/data/"]
WORKDIR /xyz
EXPOSE 8080
USER root
CMD ["python", "/xyz/application"]

TODO for the container is: move to Python 3 (so I do not need to install python2 and dependencies, figure out how to have the private data available to the container but not being part of it (it is quite big directory structure), go through these nice General Container Image Guidelines and explore this Image Metadata thingy.

Once I had that, I needed to login to the OpenShift's registry, build my image locally, test it and push it:

sudo docker build --tag selftest .
sudo docker run -ti --publish 8080:8080 --volume $( pwd )/data/:/xyz/data/ xyz   # now I can check if all looks sane with `firefox http://localhost:8080`
oc whoami -t   # this shows token I can use below
sudo docker login -u <username> -p <token> registry.rh-us-east-1.openshift.com
sudo docker tag xyz registry.rh-us-east-1.openshift.com/xyz/xyz
sudo docker push registry.rh-us-east-1.openshift.com/xyz/xyz

Now I have used Console on https://console.rh-us-east-1.openshift.com/ to create new application, then added a deployment to that application with Add to Project -> Deploy Image and selected (well, I could use cli tool oc for that):

  • surprisingly you do not choose "Image Name" here
  • but you choose "Image Stream Tag" with:
    • Namespace: selftest
    • Image Stream: selftest
    • Tag: latest

Next step looks logical, but I got stuck on it for some time, but OpenShift folks helped me (thanks Jiří!). I just need to be aware of OpenShift Online Restrictions.

So, because I wanted persistent storage and because my account uses Amazon EC2, I can not use "Shared Access (RWX)" storage type (useful when new pod is starting while old pod is still running), I had to change process of new pods start to first stop old and the start new: Applications -> Deployments -> my deployment -> Actions -> Edit -> Strategy Type: Recreate. I have created a storage with "RWO (Read-Write-Once)" access mode, added it to the deployment (... -> Actions -> Add Storage) and made sure that that storage is the only one attached to the deployment (... -> Actions -> Edit YAML and check that keys spec.template.spec.containers.env.volumeMounts and spec.template.spec.volumes only contain one volume you have just attached). In my case, there is this in the YAML definition:

[...]
    spec:
      containers:
        - env:
          [...]
          volumeMounts:
            - mountPath: /xyz/data
              name: volume-jt8t6
      [...]
      volumes:
        - name: volume-jt8t6
          persistentVolumeClaim:
            claimName: xyz-storage-claim
[...]

When working with this, I have also used ... -> Actions -> Pause Rollouts. It is also possible to configure environment variable for a deployment in ... -> Actions -> Edit -> Environment Variables which is useful to pass passwords and stuff into your app (so I do not need to store them in the image). In the app I use something like import os; SMTP_SERVER_PASSWORD = os.getenv('XYZ_SMTP_SERVER_PASSWORD', default='') to read that.

To make the app available from outside world, I have created a route in Applications -> Routes -> Create Route. It created domain like http://xyz-route-xyz.6923.rh-us-east-1.openshiftapps.com for me.

Now, looks like everything works for me and I'm kinda surprised how easy it was. I plan to get nicer domain and configure its CNAME DNS record and to explore monitoring possibilities OpenShift have. I'll see how it goes.