Note to explore Spec Kit and its ecosystem

I recently attended a great workshop where I got to try GitHub's Spec Kit, a nice tool for Spec-Driven Development that translates specifications directly into working implementations. To dive deeper into its capabilities, I am bookmarking a few interesting community extensions and a preset to experiment with later:

• Brownfield Bootstrap: Helps adopt the Spec Kit workflow in already existing codebases by auto-discovering your architecture, tech stack, and conventions.
• Checkpoint: Instructs the AI agent to make organized, incremental commits throughout the development workflow, preventing a single massive commit at the end.
• Agent Assign: Improves implementation quality by automatically routing specific tasks to specialized sub-agents (like frontend or test experts) rather than relying on one generalist agent.
• Fiction Book Writing: A creative preset that adapts the Spec Kit software workflow for storytelling, turning features into story elements and tasks into scene-by-scene drafting.

Looking forward to testing how these tools augment the standard Spec Kit experience.

Regex matching using previously seen string in Procmail

OK, I was asking Gemini (using Gemini CLI) about a syntax of Procmail rule I wanted to use and it came up with nice and reusable test, so thought I store it here for later as a refference, because I think it can be used nicely to debug any rule.

I'm trying to match emails from Atlasian Jira cloud that contain new comments so I can keep them (and drop almost everithing else). I wanted mine regexp that looks into email body to be bit more specific, because plain commented on is not very uniq string. Gemini continues:

---

I recently needed to solve a tricky Procmail filtering problem. I wanted to extract a sender's name from a header and use it immediately in a regular expression to match a specific pattern in the email body.

The Question

I have an email like this:

From: "Jane Doe (Jira)" <jira@example.com>
Date: Tue, 24 Mar 2026 03:33:20 +0000
To: John Smith <jsmith@example.com>
Subject: [JIRA] (PROJ-1234) Investigate issue

Jane Doe commented on PROJ-1234:
---------------------------------
Identified the root cause of the crash...

I wanted to take the name from the "From" header and use it in a regex on the body of the email like Jane Doe commented on PROJ-1234:. Is it possible?

The Answer

Yes, this is absolutely possible using Procmail's MATCH variable capture and variable interpolation modifiers. To do this safely—especially when the variable is immediately followed by other text—you use the following pattern:

:0:$PMDIR/notmuch.lock
* ^From: ".* \(Jira\)" <jira@example.com>
* ^From: "\/[^(]+
* $ B ?? $\MATCH()commented on [A-Z0-9]+-[0-9]+:?$
| notmuch insert $tags_keep

The Key Components:

• The Capture (\/): In the condition * ^From: "\/[^(]+, the \/ token tells Procmail to start capturing whatever matches the rest of the line into the MATCH variable. Here, "[^(]+ captures the name after opening double quotes up to the first parenthesis (e.g., "Jane Doe ").
• Variable Expansion ($): The $ at the very beginning of the condition line (* $ ...) is critical. It tells Procmail to evaluate variables like $MATCH before processing the regex.
• Match in email body instead of headers (B ??): The B ?? at the beginning of the condition line (* $ B ?? ...) tells Procmail to look for the pattern in email body instead of headers.
• Safe Escaping ($\): Using $\MATCH instead of just $MATCH tells Procmail to automatically escape any special regex characters (like . or +) found inside the name so they are treated as literal text.
• Variable Boundary (()): Since we want to follow the variable immediately with the word "commented", we use (). Just like in Bash where you might use ${MATCH}commented, Procmail needs a boundary. () is an empty regex group that separates the variable name from the following text without affecting the match.

Validation Script

To verify this works as expected, I used the following test script. It creates a mock environment, a sample email, and runs Procmail in verbose mode to confirm the match.

#!/bin/bash

# Setup test environment
TEST_DIR="/tmp/procmail_test"
mkdir -p "$TEST_DIR"
cd "$TEST_DIR"

# 1. Create a sample Jira email
cat << 'EOF' > test.eml
From: "Jane Doe (Jira)" <jira@example.com>
Date: Tue, 24 Mar 2026 03:33:20 +0000
To: John Smith <jsmith@example.com>
Subject: [JIRA] (PROJ-1234) Investigate issue

Jane Doe commented on PROJ-1234:
---------------------------------
Identified the root cause of the crash...
EOF

# 2. Create the Procmail recipe
cat << 'EOF' > test.rc
MAILDIR=/tmp/procmail_test
DEFAULT=/tmp/procmail_test/default
LOGFILE=/tmp/procmail_test/procmail.log
VERBOSE=yes

:0
* ^From: ".* \(Jira\)" <jira@example.com>
* ^From: "\/[^(]+
* $ B ?? $\MATCH()commented on [A-Z0-9]+-[0-9]+:?$
/tmp/procmail_test/matched
EOF

# 3. Run Procmail and check the logs
procmail ./test.rc < test.eml
echo "--- Procmail Log ---"
cat procmail.log
echo "--- Delivery Result ---"
ls -l matched

The verbose log confirms that Procmail correctly expands the variable and matches the body content:

procmail: Match on "^From: "\/[^(]+"
procmail: Match on "()Jane Doe commented on [A-Z0-9]+-[0-9]+:?$"
procmail: Assigning "LASTFOLDER=/tmp/procmail_test/matched"

Bash while read line without subshell and arrays with IFS

Just a few notes about script where writing it took me far more time than it shoud :)

First, while read line without subshell. So I was attempting to do something like this:

count=0
export IFS=$'/n'
cat file.log | while read line; do
  let count+=1
done
echo $count

But because of the pipe, while runs in a subshell, so whatever I do to the count variable there is lost once I exit the loop, so echo prints 0 at the end. Using answer here I used this:

count=0
trap 'rm -rf $TMPFIFODIR' EXIT
TMPFIFODIR=$( mktemp -d )
mkfifo $TMPFIFODIR/mypipe
cat file.log > $TMPFIFODIR/mypipe &
export IFS=$'/n'
while read line; do
  let count+=1
done < $TMPFIFODIR/mypipe
echo $count

Now the second problem. Inside the loop I was using this to get first number from the line (because I think it is bit cheaper to do it this way than using sed or something):

numbers=( ${line//[!0-9]/ } )
count="${numbers[0]}"

This code did not worked for me and I was getting string with numbers and spaces in count - something that converting to array (these brackets) should take care about. It took me quite a bit of time to realize that when I'm exporting IFS to only newline, I'm breaking this. So I ended with just setting IFS for that read like this:

while IFS=$'\n' read line; do
  ...
done < $TMPFIFODIR/mypipe

Troubles pasting ritch-text content from CLI to Confluence

Using some custom script I'm generating some markdown (because it is easy to write) content, maybe it is status report or something. fOR EXAMPLE this:

$ cat /tmp/report.md
# Monday
* Watching cat videos
* Fixing what I broke at Fry

Now you want to paste it to, say, Google Document. First step is to convert to HTML:

$ cat /tmp/report.md | multimarkdown
<h1 id="monday">Monday</h1>

<ul>
<li>Watching cat videos</li>
<li>Fixing what I broke at Fry</li>
</ul>

Or using Pandoc:

$ cat /tmp/report.md | pandoc --from=markdown --to=html
<h1 id="monday">Monday</h1>
<ul>
<li>Watching cat videos</li>
<li>Fixing what I broke at Fry</li>
</ul>

Now to transfer it, very convinient way I was using is to copy it to the clipboard (and then just paste in in the editor with Ctrl+V):

cat /tmp/report.md | multimarkdown | xclip -sel clip -t "text/html"

When I needed to paste into Altasian Confluence WYSIWYG editor, it did not worked for me for some reason - only plain text was copied and the formatting was lost. But copying from normal web page worked. What is the difference? Thanks to this great answer, I have examined how clipboard looks like when I copy snippet from a web browser and noticed this:

$ # Selected and copied something from the web browser
$ xclip -o -selection clipboard -t TARGETS
TIMESTAMP
TARGETS
MULTIPLE
SAVE_TARGETS
text/html
text/_moz_htmlcontext
text/_moz_htmlinfo
UTF8_STRING
COMPOUND_TEXT
TEXT
STRING
text/plain;charset=utf-8
text/plain
text/x-moz-url-priv
$ xclip -o -selection clipboard -t text/html
<meta http-equiv="content-type" content="text/html; charset=utf-8"><ol>
<li>copy something from you web browser</li>
<li>investigate available types</li>
</ol>

So looks like I need that <meta http-equiv="content-type" content="text/html; charset=utf-8"><ol> string, so let's add it:

$ (echo '<meta http-equiv="content-type" content="text/html; charset=utf-8">'; cat /tmp/report.md | multimarkdown) | xclip -sel clip -t "text/html"

And here we go, pasting to Conflence works now!

Note: For some reason I do not understand, it seems to work the old way now once I pasted the new way for a first time :-/ Maybe above is not needed at all, YMMW.

First steps with Fedora IoT on Raspberry Pi 4

This is just a quick description, brain dump, of how we started measuring temperature and moisture in various rooms in our flat. This first part describes setting up a server that will collect and present the data.

Here I was basically just following How to install Fedora IoT on Raspberry Pi 4 post.

First, I installed prerequisites on my Fedora workstation:

$ sudo dnf install gnupg2 arm-image-installer

Now download images and Fedora GPG key so I can verify the signature of the downloaded image:

$ wget https://download.fedoraproject.org/pub/alt/iot/39/IoT/aarch64/images/Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
$ wget https://download.fedoraproject.org/pub/alt/iot/39/IoT/aarch64/images/Fedora-IoT-39-aarch64-20231103.1-CHECKSUM
$ wget https://fedoraproject.org/fedora.gpg

Now verify the signature and check the downloaded key fingerprint matches what Fedora team publishes on a page with list of current GPG keys fingerprints.

$ gpgv --keyring ./fedora.gpg Fedora-IoT-39-aarch64-20231103.1-CHECKSUM
gpgv: Signature made Mon 06 Nov 2023 03:03:23 PM CET
gpgv:                using RSA key E8F23996F23218640CB44CBE75CF5AC418B8E74C
gpgv: Good signature from "Fedora (39) <fedora-39-primary@fedoraproject.org>"

$ gpg --show-keys fedora.gpg | grep -C 1 E8F23996F23218640CB44CBE75CF5AC418B8E74C
pub   rsa4096 2022-08-09 [SCE]
      E8F23996F23218640CB44CBE75CF5AC418B8E74C
uid                      Fedora (39) <fedora-39-primary@fedoraproject.org>

Also check checksum of downloaded image:

$ sha256sum -c Fedora-IoT-39-aarch64-20231103.1-CHECKSUM
Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz: OK
sha256sum: WARNING: 17 lines are improperly formatted

I guess that warning in the output is there because the checksum file also contains GPG signature and sha256sum utility dislikes it, so I did not worried about it:

$ cat Fedora-IoT-39-aarch64-20231103.1-CHECKSUM
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

# Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz: 712162312 bytes
SHA256 (Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz) = bb10ed4469f6ac1448162503b68f84e96f8e8410e5c8c9a4a56b5406bf13dff2
-----BEGIN PGP SIGNATURE-----

iQI[...]
-----END PGP SIGNATURE-----

Now I put SD card into the USB reader and connected it. It shows nicely in lsblk output as /dev/sda:

$ lsblk
NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                                             8:0    1  29.7G  0 disk  
├─sda1                                          8:1    1   512M  0 part  
└─sda2                                          8:2    1   4.4G  0 part  
zram0                                         252:0    0     8G  0 disk  [SWAP]
nvme0n1                                       259:0    0 476.9G  0 disk  
├─nvme0n1p1                                   259:1    0     1G  0 part  /boot
├─nvme0n1p2                                   259:2    0    32G  0 part  [SWAP]
└─nvme0n1p3                                   259:3    0 443.9G  0 part  
  └─luks-c9494ef2-8c28-4817-befb-8ac43ff79ee3 253:0    0 443.9G  0 crypt /home
                                                                         /

So now I should have everything needed to write Fedora IoT image to the card:

$ sudo arm-image-installer --image Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz --media /dev/sda --addkey /home/jhutar/.ssh/id_rsa.pub --norootpass --resizefs --target=rpi4 -y
[sudo] password for jhutar:

=====================================================
= Selected Image:
= Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
= Selected Media : /dev/sda
= U-Boot Target : rpi4
= Root Password will be removed.
= Root partition will be resized
= SSH Public Key /home/jhutar/.ssh/id_rsa.pub will be added.
=====================================================

*****************************************************
*****************************************************
******** WARNING! ALL DATA WILL BE DESTROYED ********
*****************************************************
*****************************************************
= Writing:
= Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
= To: /dev/sda ....
4282384384 bytes (4.3 GB, 4.0 GiB) copied, 243 s, 17.6 MB/s
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 243.92 s, 17.6 MB/s
= Writing image complete!
= Resizing /dev/sda ....
Checking that no-one is using this disk right now ... OK

Disk /dev/sda: 29.72 GiB, 31914983424 bytes, 62333952 sectors
Disk model: UHSII uSD Reader
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc1748067

Old situation:

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 1028095 1026048 501M 6 FAT16
/dev/sda2 1028096 3125247 2097152 1G 83 Linux
/dev/sda3 3125248 8388607 5263360 2.5G 83 Linux

/dev/sda3:
New situation:
Disklabel type: dos
Disk identifier: 0xc1748067

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 1028095 1026048 501M 6 FAT16
/dev/sda2 1028096 3125247 2097152 1G 83 Linux
/dev/sda3 3125248 62333951 59208704 28.2G 83 Linux

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
e2fsck 1.46.5 (30-Dec-2021)
/dev/sda3 has unsupported feature(s): FEATURE_C12
e2fsck: Get a newer version of e2fsck!

root: ********** WARNING: Filesystem still has errors **********

resize2fs 1.46.5 (30-Dec-2021)
Please run 'e2fsck -f /dev/sda3' first.

= Raspberry Pi 4 Uboot is already in place, no changes needed.
= Removing the root password.
= Adding SSH key to authorized keys.

= Installation Complete! Insert into the rpi4 and boot.

There are some errors there, right? Well, I ignored them. RPi booted nicely, I was able to setup everything (more on that in some later blog) but then I have ran out of storage. Only then I noticed root filesystem was not extended (exactly as the error message says).

After some online search I figured I need e2fsprogs-1.47.0 or newer and (at the time?) it was only available in Fedora 39. So I upgraded and now I was able to write the image just fine:

$ sudo arm-image-installer --image Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz --media /dev/sda --addkey /home/jhutar/.ssh/id_rsa.pub --norootpass --resizefs --target=rpi4 -y
[sudo] password for jhutar:

=====================================================
= Selected Image:                                 
= Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
= Selected Media : /dev/sda
= U-Boot Target : rpi4
= Root Password will be removed.
= Root partition will be resized
= SSH Public Key /home/jhutar/.ssh/id_rsa.pub will be added.
=====================================================
 
*****************************************************
*****************************************************
******** WARNING! ALL DATA WILL BE DESTROYED ********
*****************************************************
*****************************************************
= Writing:
= Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
= To: /dev/sda ....
4282384384 bytes (4.3 GB, 4.0 GiB) copied, 245 s, 17.5 MB/s
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 245.85 s, 17.5 MB/s
= Writing image complete!
= Resizing /dev/sda ....
Checking that no-one is using this disk right now ... OK

Disk /dev/sda: 29.72 GiB, 31914983424 bytes, 62333952 sectors
Disk model: UHSII uSD Reader
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc1748067

Old situation:

Device     Boot   Start     End Sectors  Size Id Type
/dev/sda1  *       2048 1028095 1026048  501M  6 FAT16
/dev/sda2       1028096 3125247 2097152    1G 83 Linux
/dev/sda3       3125248 8388607 5263360  2.5G 83 Linux

/dev/sda3:
New situation:
Disklabel type: dos
Disk identifier: 0xc1748067

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sda1  *       2048  1028095  1026048  501M  6 FAT16
/dev/sda2       1028096  3125247  2097152    1G 83 Linux
/dev/sda3       3125248 62333951 59208704 28.2G 83 Linux

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
root: 32041/164640 files (0.6% non-contiguous), 449099/657920 blocks
resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/sda3 to 7401088 (4k) blocks.
The filesystem on /dev/sda3 is now 7401088 (4k) blocks long.

= Raspberry Pi 4 Uboot is already in place, no changes needed.
= Removing the root password.
= Adding SSH key to authorized keys.

= Installation Complete! Insert into the rpi4 and boot.

Stick the card into RPi, connect power and ethernet cable and voila, I'm now able to SSH to RPi. I got the IP from my router management console from DHCP leases section.

Kinda SQL "join" in Prometheus

I'm using Prometheus query language, PromQL, quite a bit these days. But all I do are very simple queries like sum(...) or rate(...[5m]) on a OpenShift cluster I work with.

For few weeks now, mine inner me was bothered with one slightly more complex stuff. To filter one metric by label from different metric - something like JOIN in SQL world. Specifically, I wanted to see number of pods running on each cluster node with "worker" role.

We have (I'm on OpenShift 4.13) kube_node_role{role="worker"} (AFAICT this is what we call "vector" in PromQL) that have these labels:

Name            container             endpoint    job                 namespace             node                     prometheus                role    service             Value
kube_node_role  kube-rbac-proxy-main  https-main  kube-state-metrics  openshift-monitoring  ip-1-2-3-4.ec2.internal  openshift-monitoring/k8s  worker  kube-state-metrics  1
kube_node_role  kube-rbac-proxy-main  https-main  kube-state-metrics  openshift-monitoring  ip-1-2-3-5.ec2.internal  openshift-monitoring/k8s  worker  kube-state-metrics  1
[...]

And we have kube_pod_info with these labels:

Name           container             created_by_kind  created_by_name  endpoint    host_ip        host_network  job                 namespace       node                     pod                                               pod_ip       priority_class           prometheus                service             uid                                   Value
kube_pod_info  kube-rbac-proxy-main  <none>           <none>           https-main  10.201.24.232  false         kube-state-metrics  openshift-etcd  ip-1-2-3-6.ec2.internal  etcd-guard-ip-10-201-24-232.ec2.internal          10.128.2.14  system-cluster-critical  openshift-monitoring/k8s  kube-state-metrics  a2eec7b0-9f29-42b4-853d-6919d963ffa1  1
kube_pod_info  kube-rbac-proxy-main  <none>           <none>           https-main  10.201.24.232  false         kube-state-metrics  openshift-etcd  ip-1-2-3-6.ec2.internal  revision-pruner-13-ip-10-201-24-232.ec2.internal  10.128.2.4   system-node-critical     openshift-monitoring/k8s  kube-state-metrics  df5cdd67-b0f5-4896-b0b0-85095a9f3122  1

We will use on(...) and group_left(...) PromQL operators. I had some issues understanding what these do, so here is mine interpretation:

* because values are always 1 in these vectors, it is safe to multiply these.

on(...) allows me to define common label(s) that should be used to match two different vectors.

group_left(...) ... thinking, thinking, nah. I forgot mine mental model here :-/

And this is the final query I used:

sum(
    kube_pod_info{} * on(node) group_left(role) kube_node_role{role="worker"}
) by(node)

These links helped me a lot:

• How to have labels for machine roles
• Prometheus Cheat Sheet - How to Join Multiple Metrics (Vector Matching)
• PromQL operators

Tekton notes

Some time ago I was tasked to create a pipeline in Tekton and here comes some my notes I would like to know few days back :-)

1.  It is not that hard. It is just a fancy way how split your shell automation script :-)
2. Tasks are not that useful on it's own (I think), you have to stack them into a Pipeline, but Tekton Getting started with Tasks is nice start. Once you need more details, see Tasks.
3. Pipelines are the core thing and starting with Getting Started with Pipelines helped me a lot. Later I was looking into Pipelines as well.
4. Blog post Building in Kubernetes Using Tekton was also very helpful. Also used my company's CI/CD guide here ad there.
5. Tekton Hub is full of tasks (and more) and I was able to easilly see documentation for them and more importantly the actual YAML behind them - having a practical examples of how the tasks could look like behind simple hello-world tasks was very helpful. E.g. see kubernetes-actions and git-clone or git-cli.
   
6. To test things, I have used Kind as "Getting started" guide suggested and Tekton installed there really easily.
7. Creating user on Kind to be able to follow some Tekton how-tos out there that are building container using Tekton was beyond mine possibilities. I did not needed to build images, so I'm good.
8. To be able to talk to the app running in Kind cluster, I used Ingress NGINX and it's rewrite rule annotation as mine app did not liked extra data in URI. Mine specific example: perfcale-demo-app-ingress.yaml.
   
9. Results are quite simple concept. You just configure them in the task and in the script you redirect the value (their size is quite limited) to filename stored in some variable.
10. When something does not make sense, you can always add a step to your task with sleep 1000 and kubectl exec -ti pod/... -- bash.
11. Every pipeline run name have to be unique. It would be boring to create new ones with kubectl apply -f ... on each of the attempts I have done without some script, but having generateName in pipeline run metadata and using kubectl create -f ... saved my day.
    

At the end mine pipeline worked like this:

1. Clones required repos:
1. Demo application: perfscale-demo-app
2. YAMLs and misc: my-tekton-perfscale-experiment
3. Results repo: my-tekton-perfscale-experiment-results
2. Deploys the demo application (no need to build images as it is done by quay.io)
   
1. It is a simple bank-like application exposing REST API
2. There is a Locust framework based perf test included with the application that stresses the API and measures RPS
3. Application consist of one pod with PostgreSQL and another one with application itself and Gunicorn application server
3. Populates test data into the application (code for it is built in into the demo application for ease of use)
   
4. Runs the Locust framework based perf test from demo application's repository, but wrapped in thin OPL helper that stores the test results in nice JSON
5. Runs a script that loads historical results for the same test with same parameters and determines if new result is PASS or FAIL
6. Adds a new result into results repository and pushes it to GitHub
7. Deletes a demo app deployment

 The commands I have used most when working on the pipeline were:

• kubectl apply --filename pipeline.yaml - to apply changes I have done to the pipeline
  
• kubectl create --filename pipeline-run.yaml - to create new pipeline run with random suffix
  
• tkn pipelinerun logs --follow --last --all --prefix - to follow logs of the current pipeline run
• tkn pipelinerun delete --all --force - to remove all previous pipeline runs
  

Use Google Chat webhook API to send message to channel

Sending message to the Google Chat (chat.google.com, recently integrated to mail.google.com/chat/) is surprisingly simple with their webhook API. Just took me some time to figure out a data structure to send (although it it very simple as I found on Incoming webhook with Python page):

curl -X POST -H "Content-Type: application/json; charset=UTF-8" --data '{"text": "Hello @jhutar, how are you?"}' "https://chat.googleapis.com/v1/spaces/.../messages?key=...&token=..."
{
  "name": "spaces/.../messages/...",
  "sender": {
    "name": "users/...",
    "displayName": "Jenkins incomming webhook",
    "avatarUrl": "",
    "email": "",
    "domainId": "",
    "type": "BOT",
    "isAnonymous": false
  },
  "text": "Hello @jhutar",
  "cards": [],
  "previewText": "",
  "annotations": [],
  "thread": {
    "name": "spaces/.../threads/..."
  },
  "space": {
    "name": "spaces/...",
    "type": "ROOM",
    "singleUserBotDm": false,
    "threaded": true,
    "displayName": "Name of the channel"
  },
  "fallbackText": "",
  "argumentText": "Hello @jhutar, how are you?",
  "attachment": [],
  "createTime": "2021-10-11T22:07:39.490063Z",
  "lastUpdateTime": "2021-10-11T22:07:39.490063Z"
}

Using redirect() on https:// site handled by Flask -> Gunicorn -> Nginx redirects me to http

And this might be hard to notice as we usually configure Nginx to also redirect all http requests to https, so at the end you end up on correct link, but going through http is not nice and it can also break CORS as I was told.

There are two parts of the problem.

First, Nginx need to set certain headers when proxying application running in Gunicorn (e.g. see them in Deploying Gunicornbehind Nginx):

proxy_set_header    Host                $host;
proxy_set_header    X-Real-IP           $remote_addr;
proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
proxy_set_header    X-Forwarded-Proto   $scheme;
proxy_set_header    X-Forwarded-Host    $http_host;
proxy_pass  http://my_app;

Second, Flask app needs to know to use content of these headers to overwrite normal request metadata (it is called Proxy Fix and brought to us by Werkzung which is a Flask's dependency):

from flask import Flask
from werkzeug.middleware.proxy_fix import ProxyFix

app = Flask(__name__, instance_relative_config=True)
app.wsgi_app = ProxyFix(app.wsgi_app, x_for=1, x_proto=1, x_host=1)

Obligatory note: see the docs linked above as these numbers are actually important from security point of view.

Accessing Red Hat OpenShift Streams for Apache Kafka from Python

Recently Red Hat launched a way how to get managed Kafka instance and you can get one for 2 days for free. There is a limit for 1 MB per second. So far I was only using Kafka without any auth and without any encription, so here is what I had to do to make it work - typing here so I do not need to reinvent once I forgot it :-) I'm using python-kafka.

I have created a cluster and under it's "Connection" menu item I got bootstrap server jhutar--c-jc--gksg-rukm-fu-a.bf2.kafka-stage.rhcloud.com:443. It also advised me to create a service account, so I created one and it generated "Client ID" like srvc-acct-00000000-0000-0000-0000-000000000000 and "Client secret" like 00000000-0000-0000-0000-000000000000. Although "SASL/OAUTHBEARER" authentication method is recommended, as of now it is too complicated for my poor head, so I used "SASL/PLAIN" where you just use "Client ID" as username and "Client secret" as password. To create a topic, there is UI as well

To create producer and consumer:

producer = KafkaProducer(
    bootstrap_servers='jhutar--c-jc--gksg-rukm-fu-a.bf2.kafka-stage.rhcloud.com:443',
    sasl_plain_username='srvc-acct-00000000-0000-0000-0000-000000000000',
    sasl_plain_password='00000000-0000-0000-0000-000000000000',
    security_protocol='SASL_SSL',
    sasl_mechanism='PLAIN',
)

And consumer needs same parameters:

consumer = KafkaConsumer(
    '<topic>',
    bootstrap_servers='jhutar--c-jc--gksg-rukm-fu-a.bf2.kafka-stage.rhcloud.com:443',
    sasl_plain_username='srvc-acct-00000000-0000-0000-0000-000000000000',
    sasl_plain_password='00000000-0000-0000-0000-000000000000',
    security_protocol='SASL_SSL',
    sasl_mechanism='PLAIN',
)

How to access oldish Dell DRAC console? Install old Firefox and Java in Docker container

Sometimes I need to access DRAC console (i.e. "remote screen") of some older Dell system - in this case it is PowerEdge R610 and "About" says "Integrated Dell Remote Access Controller 6 - Enterprise, Version 1.98, © 2008-2011 Dell Inc.". I have tried bunch of browsers on my not super recent Fedora 30 and it failed. One solution I have found is to access the console with Firefox and IcedTea plugin from Fedora 27. VM feels too heavy for this usecase, so just allow connections to my X server:

# xhost local:root

start Fedora 27 container with some extra vars and mounts:

# docker run \
        --network host \
        -e DISPLAY=:0.0 \
        -v /tmp/.X11-unix:/tmp/.X11-unix \
        -v /root/.Xauthority:/root/.Xauthority:rw \
        -ti fedora:27 /bin/bash

and then in the container install all needed and run the browser:

# dnf -y install firefox xorg-x11-xauth icedtea-web
# firefox

Now I'm able to open the console, hooray!

Dumping my notes on Jenkins shared library use in declarative pipeline

Recently I have tasked myself to work on how we organize our Jenkins jobs code. We are using Jenkins declarative pipeline (officially that is "simplified and opinionated syntax on top of the Pipeline sub-systems", but basically it is "nicely structured, but they forbid you to use mostly anything fancy in you Groovy code").

In our case we have lots (and it is slowly growing) of jobs which are running tests from different directories in a same way, but with different parameters. Also we have another set of jobs that are checking something and in some case they trigger the test jobs. This all loudly calls for sharing the code, so I wanted to take a look at how to do it.

Here is list of tabs I'm closing now once I'm done :)

• Shared libraries - Create another git repo with vars/ directory and share the code there
• Defining Declarative Pipelines - You can define function that implements whole declarative pipeline (you can also define custom step, but you can not define shared stage)
• Groovy Maps - If you want to pass lots of parameters to the function, passing a Map is very handy and easy to read - in Python equivalent would be passing a dict I think
• Cannot define variable in pipeline stage - In declarative pipeline you can not define variable directly, use this workaround
• sh: Shell Script - Jenkins help for sh step
• Catching both output and error code - spoiler: try { ... } catch ( ... ) { ... } is also forbidden in declarative pipeline, use script { ... } again
• when - this is way how to skip a stage in declarative pipeline based on something
• Writing file in groovy - This would probably work, but Jenkins was giving me lots of security related warnings
• Library variable in jenkins shared library - how to get variables from vars/someVariables.groovy in vars/someFunction.groovy

Different Numpy results on different systems

Recently mine wife in her compute intensive project noticed strange issue when same input date and same code produced different output on different hosts in the cloud she is using. She tracked it down to this simple python & numpy test case:

#!/usr/bin/env python

import numpy

a = [[0.67115835, -0.74131401], [0.74131401, 0.67115835]]
b = [[-4.95494273, -1.77170756, ...], [1.87737564, 4.99951546, ...]]
c = numpy.matmul(a, b)
print(c)

On one host it was returning (correct result):

[[-4.71727605 -4.89530718 -4.71727605 -4.89530718 -4.71727605 -4.89530718
...

On different host it was returning (wrong result):

[[ 0.34761728  0.12429531  0.34761728  0.12429531 -0.13170853 -0.35074431
...

We have been googling a bit and found some tips:

• Same Python code, same data, different results on different machines: There is a suggestion to use export MKL_CBWR=AUTO - this did not helped
• Different results on different computers when using Scipy stack compiled with MKL: Some more tips are here: export MKL_CBWR=AVX; export OMP_NUM_THREADS=1 or with export MKL_CBWR=SSE4_2 - this did not helped again (anyway, our Numpy does not seem to be build with this Intel Math Kernel Library - see ldd output below)
• How can I make sure svd having same result on different machine: Here this is mentioned: export NUMEXPR_NUM_THREADS=1; export OPENBLAS_NUM_THREADS=1 - no luck, but finally: export OPENBLAS_CORETYPE=prescott - bingo!

According to OpenBLAS ussage instructions (OpenBLAS is "an optimized BLAS (Basic Linear Algebra Subprograms) library" if you have same knowleadge about it as I do), OPENBLAS_CORETYPE is environment variable which control the kernel selection. Looking at Prescott CPU description, it was launched in 2000, so is probably a safe default. Some more details about our setup:

Numpy in our setup is linked with these libraries:

ldd $( rpm -ql python3-numpy | grep '\.so$' ) | grep -v '\.so:$' | sed 's/([0-9a-zx]\+)/(...)/' | sort -u
	/lib64/ld-linux-x86-64.so.2 (...)
	libc.so.6 => /lib64/libc.so.6 (...)
	libdl.so.2 => /lib64/libdl.so.2 (...)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (...)
	libgfortran.so.5 => /lib64/libgfortran.so.5 (...)
	libm.so.6 => /lib64/libm.so.6 (...)
	libopenblasp.so.0 => /lib64/libopenblasp.so.0 (...)
	libpthread.so.0 => /lib64/libpthread.so.0 (...)
	libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...)
	libquadmath.so.0 => /lib64/libquadmath.so.0 (...)
	libutil.so.1 => /lib64/libutil.so.1 (...)
	linux-vdso.so.1 (...)

The code is packaged in Singularity containers and is running on Metacentrum cloud. Two machines we have hit were - the one with correct result:

Singularity> tail -n 28 /proc/cpuinfo

processor       : 15
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel Xeon E3-12xx v2 (Ivy Bridge)
stepping        : 9
microcode       : 0x1
cpu MHz         : 2199.998
cache size      : 16384 KB
physical id     : 15
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 15
initial apicid  : 15
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt
+tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm kaiser fsgsbase smep erms xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 4399.99
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Singularity> uname -a
Linux [hostname] 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21) x86_64 x86_64 x86_64 GNU/Linux

The other host - the one with wrong results:

Singularity> tail -n 28 /proc/cpuinfo

processor       : 63
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
stepping        : 4
microcode       : 0x200004d
cpu MHz         : 2399.392
cache size      : 22528 KB
physical id     : 1
siblings        : 32
core id         : 15
cpu cores       : 16
apicid          : 63
initial apicid  : 63
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology
+nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault
+epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt
+clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips        : 4201.71
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

Singularity> uname -a
Linux [hostname] 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 x86_64 x86_64 GNU/Linux

Packages in the container are:

• python3-3.7.4-1.fc30.x86_64
• python3-numpy-1.16.4-2.fc30.x86_64

If you want to try, full test case is here:

import numpy

a = [[0.67115835,-0.74131401],[0.74131401,0.67115835]]
b = [[-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936],[1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674]]
c = numpy.matmul(a,b)
print(c)

Breaking voice in Google Meet (Hangouts) with Firefox

Lot of ppl kept telling me how much distortions are there when I'm talking over Google Meet - not that everithing I say is gold, but sometimes I just want to get answer to my question :-) I'm using some (cheap) KOSS headset which connects via USB and integrates its own sound card.

After some digging this is what I did:

1. Enabled Echo/Noise-Cancellation module on PulseAudio (PulseAudio is a sound system in Linux - it is a proxy for sound applications) and disabled automatic analog gain control: Enable Echo/Noise-Cancellation (AFAICT this means PulseAudion won't attempt to automatically increase volume of the mic when I'm quiet) - you might not need this step as to me it is hard to believe sound server would have this bad results (note that you run pulseaudio -k as normal user - that "$" - and I had to restart Firefox I have been using to play some sound to hear the difference)
2. Then I have disabled automatic gain control and friends in Firefox: Disable WebRTC audio post processing
3. Hearing what you are recording was very useful: Echo test (to stop it, just use $ pactl unload-module module-loopback)

Note I'm on fedora-release-30-6.noarch, firefox-76.0-2.fc30.x86_64 and pulseaudio-12.2-9.fc30.x86_64.

Running insecure registry via Podman, starting on reboot

This is quite simple, there is a lot of docs out there, so just to put it on one place I do not need to look for it next time I want to install this "full stack solution":

• Setup Docker Container Registry with Podman & Let’s Encrypt SSL
• Starting Containers with systemd

Install Podman

# subscription-manager repos --enable rhel-7-server-extras-rpms
# yum install podman

Start and configure registry

# lvcreate data_perf54 --size 25G --name docker_registry
# mkfs.xfs /dev/mapper/data_xyz-docker_registry
# tail -n 1 /etc/fstab
/dev/mapper/data_xyz-docker_registry /var/lib/registry xfs defaults 0 0
# mount /var/lib/registry
# podman run --privileged -d --name registry-srv -p 5000:5000 -v /var/lib/registry:/var/lib/registry registry:2

Surviving reboot

# cat /etc/systemd/system/registry-srv-container.service
[Unit]
Description=Docker registry container

[Service]
Restart=always
ExecStart=/usr/bin/podman start -a registry-srv
ExecStop=/usr/bin/podman stop -t 30 registry-srv

[Install]
WantedBy=local.target
# systemctl enable registry-srv-container.service
# systemctl restart registry-srv-container.service
# systemctl status registry-srv-container.service

Push to it

# grep 'registries.insecure' -A 1 /etc/containers/registries.conf 
[registries.insecure]
registries = ['your_hostname:5000']
# podman pull busybox
# podman tag docker.io/library/busybox $( hostname ):5000/busybox
# podman push $( hostname ):5000/busybox

See registry's API

# curl -s "http://$( hostname ):5000/v2/_catalog?n=100" | json_reformat 
{
    "repositories": [
        "busybox"
    ]
}

Changing Slack status from command-line

I'm used to track mine working time with a custom script and I have keyboard shortcut for start and stop actions. One thing the script does is that when I "stop" my work, it sets mine IRC nick to "jhutar_afk" and when I "start" my work it sets the nick back to plain "jhutar". This is e.g. handy taking a launch break or going for some errands. The same is possible with Slack:

I have started with How to set a Slack status from other apps article. It is very easy.

1. Create a legacy token (I know, it is legacy - need to investigate how to use current way :-/) and put it to the variable token='xoxp-0000000000-000000000000-000000000000-00000000000000000000000000000000'
2. Construct a json to send: profile='{"status_text": "Away from keyboard", "status_emoji": ":tea:"}'
3. Send that to the API: curl -X POST https://slack.com/api/users.profile.set --silent --data "profile=$profile" --data "token=$token"

This way I can set various statuses. Then I have realized that what I really want is to set mine presence (that green or empty/black dot next to your nick). That is easy as well:

1. Again (see above), prepare your token token='xoxp-0000000000-000000000000-000000000000-00000000000000000000000000000000'
2. Use the Slack API: curl -X POST https://slack.com/api/users.setPresence --silent --data "presence=away" --data "token=$token" (to set yourself away) or use presence=auto (for a normal mode when Slack decides based on your activity)

Given how long I was avoiding to actually add it to my script, it was very easy at the end :-)

My Prometheus@OpenShift cheat-sheet

Prometheus is monitoring solution in OpenShift and I'm reading some basic out of it after some performance tests. Here are the queries I'm using:

Get CPU consumption by pods xyz...:

sum(pod_name:container_cpu_usage:sum{pod_name=~'xyz.*',namespace='qa'})

Now for memory usage (these "POD" and "''" container names seems to be doubling the value):

sum(container_memory_usage_bytes{namespace='qa', pod_name=~'xyz.*', container_name!='POD', container_name!=''})

Also see these nice examples on how to construct query.

To Querying Prometheus via API, I have used range query and this Python code:

assert start is not None and end is not None, \
    "We need timerange to approach Prometheus"

# Get data from Prometheus
token = 'your `oc whoami -t`'
url = 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query_range'   # I'm running this inside the cluster, so I can use internal hostname
headers = {
    'Authorization': f'Bearer {token}',
    'Content-Type': 'application/json',
}
params = {
    'query': monitoring_query,   # this will be some query from above
    'step': monitoring_step,   # using 60 seconds here
    'start': start.strftime('%s'),
    'end': end.strftime('%s'),
}
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)   # security is hard ;)
response = requests.get(url, headers=headers, params=params, verify=False)

# Check that what we got back seems OK
response.raise_for_status()
json_response = response.json()
assert json_response['status'] == 'success'
assert 'data' in json_response
assert 'result' in json_response['data']
assert len(json_response['data']['result']) == 1
assert 'values' in json_response['data']['result'][0]

data = [float(i[1]) for i in json_response['data']['result'][0]['values']]

Checking if filesystem supports d_type via Ansible

I had a task to add an assert to Ansible playbook to check that root / filesystem supports d_type (i.e. "directory entry type" and is important for Docker / Podman). Here is the result:

    - name: Read root filesystem type and device
      set_fact:
        root_fstype: "{{ ansible_mounts | selectattr('mount', 'equalto', '/') | map(attribute='fstype') | join(',') }}"
        root_device: "{{ ansible_mounts | selectattr('mount', 'equalto', '/') | map(attribute='device') | join(',') }}"
    - name: If root filesystem is xfs, get more info from it
      command:
        xfs_info "{{ root_device }}"
      register: xfs_info
      ignore_errors: true
      when: "root_fstype != 'ext4'"
    - name: Check that root filesystem supports directory entry type (aka d_type)
      assert:
        that:
          - "root_fstype == 'ext4' or ( root_fstype == 'xfs' and 'ftype=1' in xfs_info.stdout )"

First, we extract (more info on how we get one value from a list of dicts based on another value) root filesystem type (e.g. "ext4" or "xfs") and device (e.g. /dev/mapper/centos_something-root) from Ansible facts obtained by setup module (use ansible -u root -i inventory.ini -m setup all to see all the facts). Then we load additional info by xfs_info utility if the fs type is "xfs". And last step is finally to assert for d_type support: "ext4" is a clear win, when we got "xfs", "ftype=1" in xfs_info output is needed.

Local variable in bash

Just a quick explanation on how local works in Bash I have been sending to somebody. Have this code:

$ cat /tmp/aaa 
function without_local() {
    variable1='hello'
    echo "Function without_local: $variable1"
}
function with_local() {
    local variable2='world'
    echo "Function with_local: $variable2"
}

echo "(1) variable1='$variable1'; variable2='$variable2'"
without_local
echo "(2) variable1='$variable1'; variable2='$variable2'"
with_local
echo "(3) variable1='$variable1'; variable2='$variable2'"

And run it and notice that variable1 behaves as global, variable2 wont leave function's context:

$ bash /tmp/aaa 
(1) variable1=''; variable2=''
Function without_local: hello
(2) variable1='hello'; variable2=''
Function with_local: world
(3) variable1='hello'; variable2=''

Difference in bash's $@ and $* and how it is expanded

I keep forgetting about this and I'm always confused what is happening but it is not that difficult. Example:

$ function measurement_add() {     python -c "import sys; print sys.argv[1:]" $@; }
$ measurement_add "Hello world" 1
['Hello', 'world', '1']
$ function measurement_add() {     python -c "import sys; print sys.argv[1:]" $*; }
$ measurement_add "Hello world" 1
['Hello', 'world', '1']
$ function measurement_add() {     python -c "import sys; print sys.argv[1:]" "$@"; }
$ measurement_add "Hello world" 1
['Hello world', '1']
$ function measurement_add() {     python -c "import sys; print sys.argv[1:]" "$*"; }
$ measurement_add "Hello world" 1
['Hello world 1']

Looking into man bash into Special Parameters section:

       *      Expands to the positional parameters, starting from  one.   When
              the  expansion  is  not  within  double  quotes, each positional
              parameter expands to a separate word.  In contexts where  it  is
              performed, those words are subject to further word splitting and
              pathname expansion.  When the  expansion  occurs  within  double
              quotes,  it  expands  to  a  single  word with the value of each
              parameter separated by the first character of  the  IFS  special
              variable.   That  is, "$*" is equivalent to "$1c$2c...", where c
              is the first character of the value of the IFS variable.  If IFS
              is  unset,  the  parameters  are separated by spaces.  If IFS is
              null, the parameters are joined without intervening separators.
       @      Expands to the positional parameters, starting from  one.   When
              the  expansion  occurs  within  double  quotes,  each  parameter
              expands to a separate word.  That is, "$@" is equivalent to "$1"
              "$2"  ...   If the double-quoted expansion occurs within a word,
              the expansion of the first parameter is joined with  the  begin‐
              ning  part  of  the original word, and the expansion of the last
              parameter is joined with the last part  of  the  original  word.
              When  there  are no positional parameters, "$@" and $@ expand to
              nothing (i.e., they are removed).