First steps with Fedora IoT on Raspberry Pi 4

This is just a quick description, brain dump, of how we started measuring temperature and moisture in various rooms in our flat. This first part describes setting up a server that will collect and present the data.

Here I was basically just following How to install Fedora IoT on Raspberry Pi 4 post.

First, I installed prerequisites on my Fedora workstation:

$ sudo dnf install gnupg2 arm-image-installer

Now download images and Fedora GPG key so I can verify the signature of the downloaded image:

$ wget https://download.fedoraproject.org/pub/alt/iot/39/IoT/aarch64/images/Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
$ wget https://download.fedoraproject.org/pub/alt/iot/39/IoT/aarch64/images/Fedora-IoT-39-aarch64-20231103.1-CHECKSUM
$ wget https://fedoraproject.org/fedora.gpg

Now verify the signature and check the downloaded key fingerprint matches what Fedora team publishes on a page with list of current GPG keys fingerprints.

$ gpgv --keyring ./fedora.gpg Fedora-IoT-39-aarch64-20231103.1-CHECKSUM
gpgv: Signature made Mon 06 Nov 2023 03:03:23 PM CET
gpgv:                using RSA key E8F23996F23218640CB44CBE75CF5AC418B8E74C
gpgv: Good signature from "Fedora (39) <fedora-39-primary@fedoraproject.org>"

$ gpg --show-keys fedora.gpg | grep -C 1 E8F23996F23218640CB44CBE75CF5AC418B8E74C
pub   rsa4096 2022-08-09 [SCE]
      E8F23996F23218640CB44CBE75CF5AC418B8E74C
uid                      Fedora (39) <fedora-39-primary@fedoraproject.org>

Also check checksum of downloaded image:

$ sha256sum -c Fedora-IoT-39-aarch64-20231103.1-CHECKSUM
Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz: OK
sha256sum: WARNING: 17 lines are improperly formatted

I guess that warning in the output is there because the checksum file also contains GPG signature and sha256sum utility dislikes it, so I did not worried about it:

$ cat Fedora-IoT-39-aarch64-20231103.1-CHECKSUM
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

# Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz: 712162312 bytes
SHA256 (Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz) = bb10ed4469f6ac1448162503b68f84e96f8e8410e5c8c9a4a56b5406bf13dff2
-----BEGIN PGP SIGNATURE-----

iQI[...]
-----END PGP SIGNATURE-----

Now I put SD card into the USB reader and connected it. It shows nicely in lsblk output as /dev/sda:

$ lsblk
NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                                             8:0    1  29.7G  0 disk  
├─sda1                                          8:1    1   512M  0 part  
└─sda2                                          8:2    1   4.4G  0 part  
zram0                                         252:0    0     8G  0 disk  [SWAP]
nvme0n1                                       259:0    0 476.9G  0 disk  
├─nvme0n1p1                                   259:1    0     1G  0 part  /boot
├─nvme0n1p2                                   259:2    0    32G  0 part  [SWAP]
└─nvme0n1p3                                   259:3    0 443.9G  0 part  
  └─luks-c9494ef2-8c28-4817-befb-8ac43ff79ee3 253:0    0 443.9G  0 crypt /home
                                                                         /

So now I should have everything needed to write Fedora IoT image to the card:

$ sudo arm-image-installer --image Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz --media /dev/sda --addkey /home/jhutar/.ssh/id_rsa.pub --norootpass --resizefs --target=rpi4 -y
[sudo] password for jhutar:

=====================================================
= Selected Image:
= Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
= Selected Media : /dev/sda
= U-Boot Target : rpi4
= Root Password will be removed.
= Root partition will be resized
= SSH Public Key /home/jhutar/.ssh/id_rsa.pub will be added.
=====================================================

*****************************************************
*****************************************************
******** WARNING! ALL DATA WILL BE DESTROYED ********
*****************************************************
*****************************************************
= Writing:
= Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
= To: /dev/sda ....
4282384384 bytes (4.3 GB, 4.0 GiB) copied, 243 s, 17.6 MB/s
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 243.92 s, 17.6 MB/s
= Writing image complete!
= Resizing /dev/sda ....
Checking that no-one is using this disk right now ... OK

Disk /dev/sda: 29.72 GiB, 31914983424 bytes, 62333952 sectors
Disk model: UHSII uSD Reader
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc1748067

Old situation:

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 1028095 1026048 501M 6 FAT16
/dev/sda2 1028096 3125247 2097152 1G 83 Linux
/dev/sda3 3125248 8388607 5263360 2.5G 83 Linux

/dev/sda3:
New situation:
Disklabel type: dos
Disk identifier: 0xc1748067

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 1028095 1026048 501M 6 FAT16
/dev/sda2 1028096 3125247 2097152 1G 83 Linux
/dev/sda3 3125248 62333951 59208704 28.2G 83 Linux

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
e2fsck 1.46.5 (30-Dec-2021)
/dev/sda3 has unsupported feature(s): FEATURE_C12
e2fsck: Get a newer version of e2fsck!

root: ********** WARNING: Filesystem still has errors **********

resize2fs 1.46.5 (30-Dec-2021)
Please run 'e2fsck -f /dev/sda3' first.

= Raspberry Pi 4 Uboot is already in place, no changes needed.
= Removing the root password.
= Adding SSH key to authorized keys.

= Installation Complete! Insert into the rpi4 and boot.

There are some errors there, right? Well, I ignored them. RPi booted nicely, I was able to setup everything (more on that in some later blog) but then I have ran out of storage. Only then I noticed root filesystem was not extended (exactly as the error message says).

After some online search I figured I need e2fsprogs-1.47.0 or newer and (at the time?) it was only available in Fedora 39. So I upgraded and now I was able to write the image just fine:

$ sudo arm-image-installer --image Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz --media /dev/sda --addkey /home/jhutar/.ssh/id_rsa.pub --norootpass --resizefs --target=rpi4 -y
[sudo] password for jhutar:

=====================================================
= Selected Image:                                 
= Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
= Selected Media : /dev/sda
= U-Boot Target : rpi4
= Root Password will be removed.
= Root partition will be resized
= SSH Public Key /home/jhutar/.ssh/id_rsa.pub will be added.
=====================================================
 
*****************************************************
*****************************************************
******** WARNING! ALL DATA WILL BE DESTROYED ********
*****************************************************
*****************************************************
= Writing:
= Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
= To: /dev/sda ....
4282384384 bytes (4.3 GB, 4.0 GiB) copied, 245 s, 17.5 MB/s
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 245.85 s, 17.5 MB/s
= Writing image complete!
= Resizing /dev/sda ....
Checking that no-one is using this disk right now ... OK

Disk /dev/sda: 29.72 GiB, 31914983424 bytes, 62333952 sectors
Disk model: UHSII uSD Reader
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc1748067

Old situation:

Device     Boot   Start     End Sectors  Size Id Type
/dev/sda1  *       2048 1028095 1026048  501M  6 FAT16
/dev/sda2       1028096 3125247 2097152    1G 83 Linux
/dev/sda3       3125248 8388607 5263360  2.5G 83 Linux

/dev/sda3:
New situation:
Disklabel type: dos
Disk identifier: 0xc1748067

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sda1  *       2048  1028095  1026048  501M  6 FAT16
/dev/sda2       1028096  3125247  2097152    1G 83 Linux
/dev/sda3       3125248 62333951 59208704 28.2G 83 Linux

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
root: 32041/164640 files (0.6% non-contiguous), 449099/657920 blocks
resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/sda3 to 7401088 (4k) blocks.
The filesystem on /dev/sda3 is now 7401088 (4k) blocks long.

= Raspberry Pi 4 Uboot is already in place, no changes needed.
= Removing the root password.
= Adding SSH key to authorized keys.

= Installation Complete! Insert into the rpi4 and boot.

Stick the card into RPi, connect power and ethernet cable and voila, I'm now able to SSH to RPi. I got the IP from my router management console from DHCP leases section.

Kinda SQL "join" in Prometheus

I'm using Prometheus query language, PromQL, quite a bit these days. But all I do are very simple queries like sum(...) or rate(...[5m]) on a OpenShift cluster I work with.

For few weeks now, mine inner me was bothered with one slightly more complex stuff. To filter one metric by label from different metric - something like JOIN in SQL world. Specifically, I wanted to see number of pods running on each cluster node with "worker" role.

We have (I'm on OpenShift 4.13) kube_node_role{role="worker"} (AFAICT this is what we call "vector" in PromQL) that have these labels:

Name            container             endpoint    job                 namespace             node                     prometheus                role    service             Value
kube_node_role  kube-rbac-proxy-main  https-main  kube-state-metrics  openshift-monitoring  ip-1-2-3-4.ec2.internal  openshift-monitoring/k8s  worker  kube-state-metrics  1
kube_node_role  kube-rbac-proxy-main  https-main  kube-state-metrics  openshift-monitoring  ip-1-2-3-5.ec2.internal  openshift-monitoring/k8s  worker  kube-state-metrics  1
[...]

And we have kube_pod_info with these labels:

Name           container             created_by_kind  created_by_name  endpoint    host_ip        host_network  job                 namespace       node                     pod                                               pod_ip       priority_class           prometheus                service             uid                                   Value
kube_pod_info  kube-rbac-proxy-main  <none>           <none>           https-main  10.201.24.232  false         kube-state-metrics  openshift-etcd  ip-1-2-3-6.ec2.internal  etcd-guard-ip-10-201-24-232.ec2.internal          10.128.2.14  system-cluster-critical  openshift-monitoring/k8s  kube-state-metrics  a2eec7b0-9f29-42b4-853d-6919d963ffa1  1
kube_pod_info  kube-rbac-proxy-main  <none>           <none>           https-main  10.201.24.232  false         kube-state-metrics  openshift-etcd  ip-1-2-3-6.ec2.internal  revision-pruner-13-ip-10-201-24-232.ec2.internal  10.128.2.4   system-node-critical     openshift-monitoring/k8s  kube-state-metrics  df5cdd67-b0f5-4896-b0b0-85095a9f3122  1

We will use on(...) and group_left(...) PromQL operators. I had some issues understanding what these do, so here is mine interpretation:

* because values are always 1 in these vectors, it is safe to multiply these.

on(...) allows me to define common label(s) that should be used to match two different vectors.

group_left(...) ... thinking, thinking, nah. I forgot mine mental model here :-/

And this is the final query I used:

sum(
    kube_pod_info{} * on(node) group_left(role) kube_node_role{role="worker"}
) by(node)

These links helped me a lot:

• How to have labels for machine roles
• Prometheus Cheat Sheet - How to Join Multiple Metrics (Vector Matching)
• PromQL operators