Difference in bash's $@ and $* and how it is expanded

I keep forgetting about this and I'm always confused what is happening but it is not that difficult. Example:

$ function measurement_add() {     python -c "import sys; print sys.argv[1:]" $@; }
$ measurement_add "Hello world" 1
['Hello', 'world', '1']
$ function measurement_add() {     python -c "import sys; print sys.argv[1:]" $*; }
$ measurement_add "Hello world" 1
['Hello', 'world', '1']
$ function measurement_add() {     python -c "import sys; print sys.argv[1:]" "$@"; }
$ measurement_add "Hello world" 1
['Hello world', '1']
$ function measurement_add() {     python -c "import sys; print sys.argv[1:]" "$*"; }
$ measurement_add "Hello world" 1
['Hello world 1']
Looking into man bash into Special Parameters section:
       *      Expands to the positional parameters, starting from  one.   When
              the  expansion  is  not  within  double  quotes, each positional
              parameter expands to a separate word.  In contexts where  it  is
              performed, those words are subject to further word splitting and
              pathname expansion.  When the  expansion  occurs  within  double
              quotes,  it  expands  to  a  single  word with the value of each
              parameter separated by the first character of  the  IFS  special
              variable.   That  is, "$*" is equivalent to "$1c$2c...", where c
              is the first character of the value of the IFS variable.  If IFS
              is  unset,  the  parameters  are separated by spaces.  If IFS is
              null, the parameters are joined without intervening separators.
       @      Expands to the positional parameters, starting from  one.   When
              the  expansion  occurs  within  double  quotes,  each  parameter
              expands to a separate word.  That is, "$@" is equivalent to "$1"
              "$2"  ...   If the double-quoted expansion occurs within a word,
              the expansion of the first parameter is joined with  the  begin‐
              ning  part  of  the original word, and the expansion of the last
              parameter is joined with the last part  of  the  original  word.
              When  there  are no positional parameters, "$@" and $@ expand to
              nothing (i.e., they are removed).


Hide sidebar when viewing message in NeoMutt

When I need to copy&paste from the email I'm reading in the Mutt, I had to enter edit mode (press 'e'), so sidebar gets out of my way. So I was thinking: would it be possible to hide sidebar when you enter reading mode (i.e. "pager") and then show it again when you leave pager mode back to screen with list of your emails (i.e. "index" mode)?

message-hook ~A "set sidebar_visible = no"
macro pager q "set sidebar_visible = yes"

So, this way you set sidebar invisible when you are viewing message (that "~A" is a mutt pattern to specify "any message") and then when I press "q" to close the pager, it sets sidebar to be visible again and exits the pager.


Creating Singularity container

For some project I had to create a Singularity container (because of the environment where it needs to run). Singularity is a container technology used by scientists. It turned out that it is very simple. Although there is not recent enough Singularity package in the normal Fedora repos, they have nice guide on how to build your own packages and that worked for me on a first try. First "Singularity" build file (similar to "Dockerfile" file) - I have based it on a recent Fedora docker image:
$ cat Singularity 
Bootstrap: docker
From: fedora:latest

    This is a container for ... project
    See https://gitlab.com/...
    Email ...

    Homepage https://gitlab.com/...
    Author ...
    Maintainer ...
    Version 0.1

    /home/where/is/your/project /projectX

    dnf -y install python2-biopython python2-numpy python2-tabulate python2-scikit-learn pymol mono-core python2-unittest2 python2-svgwrite python2-requests
    chown -R 1000:1000 /projectX   # probably not important

    cd /projectX
    python -m unittest discover

    export LC_ALL=C

    exec /projectX/worker.sh
Majority of above is not needed: e.g. "%help" have completely free form, keys in "%labels" does not seem to be codified, "%test" which is ran as a last step of a build process is also optional. To build it:
$ sudo singularity build --writable projectXy.simg Singularity   # *.simg is a native format of singularity-2.4.6
$ sudo singularity build --writable projectX.img projectX.simg   # where the project is supposed to run, there is 2.3.2 which needs older *.img, so convert *.simg into it
Mine original idea was to have the project in writable container (so the option "--writable" above), but that would require me to run it as root again (or I'm missing something), so I have ended up with the solution of running the container in read-only mode and mounting mine project into it to have a read-write-able directory where I can generate the data:
$ echo "cd /projectX; ./worker.sh" \
      | singularity exec --bind projectX/:/projectX projectX.img bash
So far it looks like it just works.


TaskWarior and listing tasks completed in specified time range

Besides mountain of inefficient papers (good thing about paper is you can lose it easily), I'm using TaskWarrior to manage my TODOs. Today I'm creating list of things I have finished in last (fiscal) year, so its reporting and filtering capabilities come handy. But as not too advanced user, it took me some time to discover how to list tasks completed in specified time range:

$ task end.after:2017-03-01 end.before:2018-03-01 completed

ID UUID     Created    Completed  Age  P Project         Tags R Due        Description                                                                                                                                                        
 - 5248839d 2017-01-24 2017-03-20 1.0y H                                   Prepare 'Investigating differences in code coverage reports' presentation


Running wsgi application in OpenShift v.3 for the first time

For some time I'm running publicly available web application Brno People team uses to determine technical interests of employee candidates. The app was running on the OpenShift v.2, but that was discontinued and I had to port it to OpenShift v.3. I was postponing the task for multiple moths and got to the state when v.2 instances were finally discontinued. It turned out that porting is not that hard. This is what I have done.

Note that I'm using Red Hat's Employee account, so some paths might be different when OpenShift is being used "normally" (you will see something like ....starter-us-east-1.openshift.com instead of mine ....rh-us-east-1.openshift.com).

Because I need to put some private data into the image, I want the image to be accessible only from mine OpenShift Online account. Anyway, I have created Dockerfile based on Fedora Dockerfile template for Python (is it official?) like this:

FROM fedora
MAINTAINER Jan Hutar <jhutar@redhat.com>
RUN dnf -y update && dnf clean all
RUN dnf -y install subversion python mod_wsgi && dnf clean all
RUN ...
VOLUME ["/xyz/data/"]
USER root
CMD ["python", "/xyz/application"]

TODO for the container is: move to Python 3 (so I do not need to install python2 and dependencies, figure out how to have the private data available to the container but not being part of it (it is quite big directory structure), go through these nice General Container Image Guidelines and explore this Image Metadata thingy.

Once I had that, I needed to login to the OpenShift's registry, build my image locally, test it and push it:

sudo docker build --tag selftest .
sudo docker run -ti --publish 8080:8080 --volume $( pwd )/data/:/xyz/data/ xyz   # now I can check if all looks sane with `firefox http://localhost:8080`
oc whoami -t   # this shows token I can use below
sudo docker login -u <username> -p <token> registry.rh-us-east-1.openshift.com
sudo docker tag xyz registry.rh-us-east-1.openshift.com/xyz/xyz
sudo docker push registry.rh-us-east-1.openshift.com/xyz/xyz

Now I have used Console on https://console.rh-us-east-1.openshift.com/ to create new application, then added a deployment to that application with Add to Project -> Deploy Image and selected (well, I could use cli tool oc for that):

  • surprisingly you do not choose "Image Name" here
  • but you choose "Image Stream Tag" with:
    • Namespace: selftest
    • Image Stream: selftest
    • Tag: latest

Next step looks logical, but I got stuck on it for some time, but OpenShift folks helped me (thanks Jiří!). I just need to be aware of OpenShift Online Restrictions.

So, because I wanted persistent storage and because my account uses Amazon EC2, I can not use "Shared Access (RWX)" storage type (useful when new pod is starting while old pod is still running), I had to change process of new pods start to first stop old and the start new: Applications -> Deployments -> my deployment -> Actions -> Edit -> Strategy Type: Recreate. I have created a storage with "RWO (Read-Write-Once)" access mode, added it to the deployment (... -> Actions -> Add Storage) and made sure that that storage is the only one attached to the deployment (... -> Actions -> Edit YAML and check that keys spec.template.spec.containers.env.volumeMounts and spec.template.spec.volumes only contain one volume you have just attached). In my case, there is this in the YAML definition:

        - env:
            - mountPath: /xyz/data
              name: volume-jt8t6
        - name: volume-jt8t6
            claimName: xyz-storage-claim

When working with this, I have also used ... -> Actions -> Pause Rollouts. It is also possible to configure environment variable for a deployment in ... -> Actions -> Edit -> Environment Variables which is useful to pass passwords and stuff into your app (so I do not need to store them in the image). In the app I use something like import os; SMTP_SERVER_PASSWORD = os.getenv('XYZ_SMTP_SERVER_PASSWORD', default='') to read that.

To make the app available from outside world, I have created a route in Applications -> Routes -> Create Route. It created domain like http://xyz-route-xyz.6923.rh-us-east-1.openshiftapps.com for me.

Now, looks like everything works for me and I'm kinda surprised how easy it was. I plan to get nicer domain and configure its CNAME DNS record and to explore monitoring possibilities OpenShift have. I'll see how it goes.


Monitoring Satellite 5 with PCP (Performance Co-Pilot)

During some performance testing we have done, I have used PCP to monitor basic stats about Red Hat Satellite 5 (could be applied to Spacewalk). I was unable to make it sufficient, but maybe somebody could fix and enhance it. I have taken lots from lzap. First of all, install PCP (PostgreSQL and Apache PMDA lives in RHEL Optional repo as of now, in CentOS7 it seems to be directly in base repo):
subscription-manager repos --enable rhel-6-server-optional-rpms
yum -y install pcp pcp-pmda-postgresql pcp-pmda-apache
subscription-manager repos --disable rhel-6-server-optional-rpms
Now start services:
chkconfig pmcd on
chkconfig pmlogger on
service pmcd restart
service pmlogger restart
Install PostgreSQL and Apache monitoring plugins
cd /var/lib/pcp/pmdas/postgresql
./Install   # select "c(ollector)" when it asks
cd /var/lib/pcp/pmdas/apache
echo -e "<Location /server-status>\n  SetHandler server-status\n  Allow from all\n</Location>\nExtendedStatus On" >>/etc/httpd/conf/httpd.conf
service httpd restart
# Configure hot proc
cat >/var/lib/pcp/pmdas/proc/hotproc.conf <<EOF
> #pmdahotproc
> Version 1.0
> fname == "java" || fname == "httpd"
And because I have Graphite/Grafana setup available, I was pumping selected metrices there (from RHEL6 which is with SysV):
# tail -n 1 /etc/rc.local
pcp2graphite --graphite-host carbon.example.com --prefix "pcp-jhutar." --host localhost - kernel.all.load mem.util.used mem.util.swapCached filesys.full network.interface.out.bytes network.interface.in.bytes disk.dm.read disk.dm.write apache.requests_per_sec apache.bytes_per_sec apache.busy_servers apache.idle_servers postgresql.stat.all_tables.idx_scan postgresql.stat.all_tables.seq_scan postgresql.stat.database.tup_inserted postgresql.stat.database.tup_returned postgresql.stat.database.tup_deleted postgresql.stat.database.tup_fetched postgresql.stat.database.tup_updated filesys.full hotproc.memory.rss &

Problems I had with this

For some reasons I have not investigated closely, after some time PostgreSQL data were not visible in Grafana. Also I was unable to get hotproc data available in Grafana. Also I was experimenting with PCP's emulation of Graphite and its Grafana, but PCP's Graphite lack filters which makes its usage hard and not practical for anything beyond simple stats.


"Error: Too many open files" when inside Docker container

Does not work: various ulimit settings for daemon

We have container build from this Dockerfile, running RHEL7 with oldish docker-1.10.3-59.el7.x86_64. Containers are started with:

# for i in $( seq 500 ); do
      docker run -h "$( hostname -s )container$i.example.com" -d --tmpfs /tmp --tmpfs /run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --ulimit nofile=10000:10000 r7perfsat

and we have set limits for a docker service on a docker host:

# cat /etc/systemd/system/docker.service.d/limits.conf

but we have still seen issues with "Too many open files" inside the container. It could happen when installing package with yum (resulting into corrupted rpm database, rm -rf /var/lib/rpm/__db.00*; rpm --rebuilddb; saved it though) and when enabling service (our containers have systemd in them on purpose):

# systemctl restart osad
Error: Too many open files
# echo $?

Because I was stupid, I have not checked journal (in the container) in the moment when I have spotted the failure for the first time:

Dec 21 10:18:54 b08-h19-r620container247.example.com journalctl[39]: Failed to create inotify watch: Too many open files
Dec 21 10:18:54 b08-h19-r620container247.example.com systemd[1]: systemd-journal-flush.service: main process exited, code=exited, status=1/FAILURE
Dec 21 10:18:54 b08-h19-r620container247.example.com systemd[1]: inotify_init1() failed: Too many open files
Dec 21 10:18:54 b08-h19-r620container247.example.com systemd[1]: inotify_init1() failed: Too many open files

Does work: fs.inotify.max_user_instances

At the end I have ran into some issue and very last comment there had a think I have not seen before. At the end I have ended up with:

# cat /etc/sysctl.d/40-max-user-watches.conf

Default on a different machine is:

# sysctl -a 2>&1 | grep fs.inotify.max_user_
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192

Looks like increasing fs.inotify.max_user_instances helped and our containers are stable.