How to access oldish Dell DRAC console? Install old Firefox and Java in Docker container

Sometimes I need to access DRAC console (i.e. "remote screen") of some older Dell system - in this case it is PowerEdge R610 and "About" says "Integrated Dell Remote Access Controller 6 - Enterprise, Version 1.98, © 2008-2011 Dell Inc.". I have tried bunch of browsers on my not super recent Fedora 30 and it failed. One solution I have found is to access the console with Firefox and IcedTea plugin from Fedora 27. VM feels too heavy for this usecase, so just allow connections to my X server:

# xhost local:root

start Fedora 27 container with some extra vars and mounts:

# docker run \
        --network host \
        -e DISPLAY=:0.0 \
        -v /tmp/.X11-unix:/tmp/.X11-unix \
        -v /root/.Xauthority:/root/.Xauthority:rw \
        -ti fedora:27 /bin/bash

and then in the container install all needed and run the browser:

# dnf -y install firefox xorg-x11-xauth icedtea-web
# firefox

Now I'm able to open the console, hooray!

Dumping my notes on Jenkins shared library use in declarative pipeline

Recently I have tasked myself to work on how we organize our Jenkins jobs code. We are using Jenkins declarative pipeline (officially that is "simplified and opinionated syntax on top of the Pipeline sub-systems", but basically it is "nicely structured, but they forbid you to use mostly anything fancy in you Groovy code").

In our case we have lots (and it is slowly growing) of jobs which are running tests from different directories in a same way, but with different parameters. Also we have another set of jobs that are checking something and in some case they trigger the test jobs. This all loudly calls for sharing the code, so I wanted to take a look at how to do it.

Here is list of tabs I'm closing now once I'm done :)

• Shared libraries - Create another git repo with vars/ directory and share the code there
• Defining Declarative Pipelines - You can define function that implements whole declarative pipeline (you can also define custom step, but you can not define shared stage)
• Groovy Maps - If you want to pass lots of parameters to the function, passing a Map is very handy and easy to read - in Python equivalent would be passing a dict I think
• Cannot define variable in pipeline stage - In declarative pipeline you can not define variable directly, use this workaround
• sh: Shell Script - Jenkins help for sh step
• Catching both output and error code - spoiler: try { ... } catch ( ... ) { ... } is also forbidden in declarative pipeline, use script { ... } again
• when - this is way how to skip a stage in declarative pipeline based on something
• Writing file in groovy - This would probably work, but Jenkins was giving me lots of security related warnings
• Library variable in jenkins shared library - how to get variables from vars/someVariables.groovy in vars/someFunction.groovy

Different Numpy results on different systems

Recently mine wife in her compute intensive project noticed strange issue when same input date and same code produced different output on different hosts in the cloud she is using. She tracked it down to this simple python & numpy test case:

#!/usr/bin/env python

import numpy

a = [[0.67115835, -0.74131401], [0.74131401, 0.67115835]]
b = [[-4.95494273, -1.77170756, ...], [1.87737564, 4.99951546, ...]]
c = numpy.matmul(a, b)
print(c)

On one host it was returning (correct result):

[[-4.71727605 -4.89530718 -4.71727605 -4.89530718 -4.71727605 -4.89530718
...

On different host it was returning (wrong result):

[[ 0.34761728  0.12429531  0.34761728  0.12429531 -0.13170853 -0.35074431
...

We have been googling a bit and found some tips:

• Same Python code, same data, different results on different machines: There is a suggestion to use export MKL_CBWR=AUTO - this did not helped
• Different results on different computers when using Scipy stack compiled with MKL: Some more tips are here: export MKL_CBWR=AVX; export OMP_NUM_THREADS=1 or with export MKL_CBWR=SSE4_2 - this did not helped again (anyway, our Numpy does not seem to be build with this Intel Math Kernel Library - see ldd output below)
• How can I make sure svd having same result on different machine: Here this is mentioned: export NUMEXPR_NUM_THREADS=1; export OPENBLAS_NUM_THREADS=1 - no luck, but finally: export OPENBLAS_CORETYPE=prescott - bingo!

According to OpenBLAS ussage instructions (OpenBLAS is "an optimized BLAS (Basic Linear Algebra Subprograms) library" if you have same knowleadge about it as I do), OPENBLAS_CORETYPE is environment variable which control the kernel selection. Looking at Prescott CPU description, it was launched in 2000, so is probably a safe default. Some more details about our setup:

Numpy in our setup is linked with these libraries:

ldd $( rpm -ql python3-numpy | grep '\.so$' ) | grep -v '\.so:$' | sed 's/([0-9a-zx]\+)/(...)/' | sort -u
	/lib64/ld-linux-x86-64.so.2 (...)
	libc.so.6 => /lib64/libc.so.6 (...)
	libdl.so.2 => /lib64/libdl.so.2 (...)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (...)
	libgfortran.so.5 => /lib64/libgfortran.so.5 (...)
	libm.so.6 => /lib64/libm.so.6 (...)
	libopenblasp.so.0 => /lib64/libopenblasp.so.0 (...)
	libpthread.so.0 => /lib64/libpthread.so.0 (...)
	libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...)
	libquadmath.so.0 => /lib64/libquadmath.so.0 (...)
	libutil.so.1 => /lib64/libutil.so.1 (...)
	linux-vdso.so.1 (...)

The code is packaged in Singularity containers and is running on Metacentrum cloud. Two machines we have hit were - the one with correct result:

Singularity> tail -n 28 /proc/cpuinfo

processor       : 15
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel Xeon E3-12xx v2 (Ivy Bridge)
stepping        : 9
microcode       : 0x1
cpu MHz         : 2199.998
cache size      : 16384 KB
physical id     : 15
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 15
initial apicid  : 15
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt
+tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm kaiser fsgsbase smep erms xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 4399.99
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Singularity> uname -a
Linux [hostname] 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21) x86_64 x86_64 x86_64 GNU/Linux

The other host - the one with wrong results:

Singularity> tail -n 28 /proc/cpuinfo

processor       : 63
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
stepping        : 4
microcode       : 0x200004d
cpu MHz         : 2399.392
cache size      : 22528 KB
physical id     : 1
siblings        : 32
core id         : 15
cpu cores       : 16
apicid          : 63
initial apicid  : 63
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology
+nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault
+epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt
+clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips        : 4201.71
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

Singularity> uname -a
Linux [hostname] 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 x86_64 x86_64 GNU/Linux

Packages in the container are:

• python3-3.7.4-1.fc30.x86_64
• python3-numpy-1.16.4-2.fc30.x86_64

If you want to try, full test case is here:

import numpy

a = [[0.67115835,-0.74131401],[0.74131401,0.67115835]]
b = [[-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936],[1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674]]
c = numpy.matmul(a,b)
print(c)

Breaking voice in Google Meet (Hangouts) with Firefox

Lot of ppl kept telling me how much distortions are there when I'm talking over Google Meet - not that everithing I say is gold, but sometimes I just want to get answer to my question :-) I'm using some (cheap) KOSS headset which connects via USB and integrates its own sound card.

After some digging this is what I did:

1. Enabled Echo/Noise-Cancellation module on PulseAudio (PulseAudio is a sound system in Linux - it is a proxy for sound applications) and disabled automatic analog gain control: Enable Echo/Noise-Cancellation (AFAICT this means PulseAudion won't attempt to automatically increase volume of the mic when I'm quiet) - you might not need this step as to me it is hard to believe sound server would have this bad results (note that you run pulseaudio -k as normal user - that "$" - and I had to restart Firefox I have been using to play some sound to hear the difference)
2. Then I have disabled automatic gain control and friends in Firefox: Disable WebRTC audio post processing
3. Hearing what you are recording was very useful: Echo test (to stop it, just use $ pactl unload-module module-loopback)

Note I'm on fedora-release-30-6.noarch, firefox-76.0-2.fc30.x86_64 and pulseaudio-12.2-9.fc30.x86_64.

Running insecure registry via Podman, starting on reboot

This is quite simple, there is a lot of docs out there, so just to put it on one place I do not need to look for it next time I want to install this "full stack solution":

• Setup Docker Container Registry with Podman & Let’s Encrypt SSL
• Starting Containers with systemd

Install Podman

# subscription-manager repos --enable rhel-7-server-extras-rpms
# yum install podman

Start and configure registry

# lvcreate data_perf54 --size 25G --name docker_registry
# mkfs.xfs /dev/mapper/data_xyz-docker_registry
# tail -n 1 /etc/fstab
/dev/mapper/data_xyz-docker_registry /var/lib/registry xfs defaults 0 0
# mount /var/lib/registry
# podman run --privileged -d --name registry-srv -p 5000:5000 -v /var/lib/registry:/var/lib/registry registry:2

Surviving reboot

# cat /etc/systemd/system/registry-srv-container.service
[Unit]
Description=Docker registry container

[Service]
Restart=always
ExecStart=/usr/bin/podman start -a registry-srv
ExecStop=/usr/bin/podman stop -t 30 registry-srv

[Install]
WantedBy=local.target
# systemctl enable registry-srv-container.service
# systemctl restart registry-srv-container.service
# systemctl status registry-srv-container.service

Push to it

# grep 'registries.insecure' -A 1 /etc/containers/registries.conf 
[registries.insecure]
registries = ['your_hostname:5000']
# podman pull busybox
# podman tag docker.io/library/busybox $( hostname ):5000/busybox
# podman push $( hostname ):5000/busybox

See registry's API

# curl -s "http://$( hostname ):5000/v2/_catalog?n=100" | json_reformat 
{
    "repositories": [
        "busybox"
    ]
}

Changing Slack status from command-line

I'm used to track mine working time with a custom script and I have keyboard shortcut for start and stop actions. One thing the script does is that when I "stop" my work, it sets mine IRC nick to "jhutar_afk" and when I "start" my work it sets the nick back to plain "jhutar". This is e.g. handy taking a launch break or going for some errands. The same is possible with Slack:

I have started with How to set a Slack status from other apps article. It is very easy.

1. Create a legacy token (I know, it is legacy - need to investigate how to use current way :-/) and put it to the variable token='xoxp-0000000000-000000000000-000000000000-00000000000000000000000000000000'
2. Construct a json to send: profile='{"status_text": "Away from keyboard", "status_emoji": ":tea:"}'
3. Send that to the API: curl -X POST https://slack.com/api/users.profile.set --silent --data "profile=$profile" --data "token=$token"

This way I can set various statuses. Then I have realized that what I really want is to set mine presence (that green or empty/black dot next to your nick). That is easy as well:

1. Again (see above), prepare your token token='xoxp-0000000000-000000000000-000000000000-00000000000000000000000000000000'
2. Use the Slack API: curl -X POST https://slack.com/api/users.setPresence --silent --data "presence=away" --data "token=$token" (to set yourself away) or use presence=auto (for a normal mode when Slack decides based on your activity)

Given how long I was avoiding to actually add it to my script, it was very easy at the end :-)

My Prometheus@OpenShift cheat-sheet

Prometheus is monitoring solution in OpenShift and I'm reading some basic out of it after some performance tests. Here are the queries I'm using:

Get CPU consumption by pods xyz...:

sum(pod_name:container_cpu_usage:sum{pod_name=~'xyz.*',namespace='qa'})

Now for memory usage (these "POD" and "''" container names seems to be doubling the value):

sum(container_memory_usage_bytes{namespace='qa', pod_name=~'xyz.*', container_name!='POD', container_name!=''})

Also see these nice examples on how to construct query.

To Querying Prometheus via API, I have used range query and this Python code:

assert start is not None and end is not None, \
    "We need timerange to approach Prometheus"

# Get data from Prometheus
token = 'your `oc whoami -t`'
url = 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query_range'   # I'm running this inside the cluster, so I can use internal hostname
headers = {
    'Authorization': f'Bearer {token}',
    'Content-Type': 'application/json',
}
params = {
    'query': monitoring_query,   # this will be some query from above
    'step': monitoring_step,   # using 60 seconds here
    'start': start.strftime('%s'),
    'end': end.strftime('%s'),
}
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)   # security is hard ;)
response = requests.get(url, headers=headers, params=params, verify=False)

# Check that what we got back seems OK
response.raise_for_status()
json_response = response.json()
assert json_response['status'] == 'success'
assert 'data' in json_response
assert 'result' in json_response['data']
assert len(json_response['data']['result']) == 1
assert 'values' in json_response['data']['result'][0]

data = [float(i[1]) for i in json_response['data']['result'][0]['values']]