First steps with Fedora IoT on Raspberry Pi 4

This is just a quick description, brain dump, of how we started measuring temperature and moisture in various rooms in our flat. This first part describes setting up a server that will collect and present the data.

Here I was basically just following How to install Fedora IoT on Raspberry Pi 4 post.

First, I installed prerequisites on my Fedora workstation:

$ sudo dnf install gnupg2 arm-image-installer

Now download images and Fedora GPG key so I can verify the signature of the downloaded image:

$ wget https://download.fedoraproject.org/pub/alt/iot/39/IoT/aarch64/images/Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
$ wget https://download.fedoraproject.org/pub/alt/iot/39/IoT/aarch64/images/Fedora-IoT-39-aarch64-20231103.1-CHECKSUM
$ wget https://fedoraproject.org/fedora.gpg

Now verify the signature and check the downloaded key fingerprint matches what Fedora team publishes on a page with list of current GPG keys fingerprints.

$ gpgv --keyring ./fedora.gpg Fedora-IoT-39-aarch64-20231103.1-CHECKSUM
gpgv: Signature made Mon 06 Nov 2023 03:03:23 PM CET
gpgv:                using RSA key E8F23996F23218640CB44CBE75CF5AC418B8E74C
gpgv: Good signature from "Fedora (39) <fedora-39-primary@fedoraproject.org>"

$ gpg --show-keys fedora.gpg | grep -C 1 E8F23996F23218640CB44CBE75CF5AC418B8E74C
pub   rsa4096 2022-08-09 [SCE]
      E8F23996F23218640CB44CBE75CF5AC418B8E74C
uid                      Fedora (39) <fedora-39-primary@fedoraproject.org>

Also check checksum of downloaded image:

$ sha256sum -c Fedora-IoT-39-aarch64-20231103.1-CHECKSUM
Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz: OK
sha256sum: WARNING: 17 lines are improperly formatted

I guess that warning in the output is there because the checksum file also contains GPG signature and sha256sum utility dislikes it, so I did not worried about it:

$ cat Fedora-IoT-39-aarch64-20231103.1-CHECKSUM
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

# Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz: 712162312 bytes
SHA256 (Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz) = bb10ed4469f6ac1448162503b68f84e96f8e8410e5c8c9a4a56b5406bf13dff2
-----BEGIN PGP SIGNATURE-----

iQI[...]
-----END PGP SIGNATURE-----

Now I put SD card into the USB reader and connected it. It shows nicely in lsblk output as /dev/sda:

$ lsblk
NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                                             8:0    1  29.7G  0 disk  
├─sda1                                          8:1    1   512M  0 part  
└─sda2                                          8:2    1   4.4G  0 part  
zram0                                         252:0    0     8G  0 disk  [SWAP]
nvme0n1                                       259:0    0 476.9G  0 disk  
├─nvme0n1p1                                   259:1    0     1G  0 part  /boot
├─nvme0n1p2                                   259:2    0    32G  0 part  [SWAP]
└─nvme0n1p3                                   259:3    0 443.9G  0 part  
  └─luks-c9494ef2-8c28-4817-befb-8ac43ff79ee3 253:0    0 443.9G  0 crypt /home
                                                                         /

So now I should have everything needed to write Fedora IoT image to the card:

$ sudo arm-image-installer --image Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz --media /dev/sda --addkey /home/jhutar/.ssh/id_rsa.pub --norootpass --resizefs --target=rpi4 -y
[sudo] password for jhutar:

=====================================================
= Selected Image:
= Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
= Selected Media : /dev/sda
= U-Boot Target : rpi4
= Root Password will be removed.
= Root partition will be resized
= SSH Public Key /home/jhutar/.ssh/id_rsa.pub will be added.
=====================================================

*****************************************************
*****************************************************
******** WARNING! ALL DATA WILL BE DESTROYED ********
*****************************************************
*****************************************************
= Writing:
= Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
= To: /dev/sda ....
4282384384 bytes (4.3 GB, 4.0 GiB) copied, 243 s, 17.6 MB/s
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 243.92 s, 17.6 MB/s
= Writing image complete!
= Resizing /dev/sda ....
Checking that no-one is using this disk right now ... OK

Disk /dev/sda: 29.72 GiB, 31914983424 bytes, 62333952 sectors
Disk model: UHSII uSD Reader
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc1748067

Old situation:

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 1028095 1026048 501M 6 FAT16
/dev/sda2 1028096 3125247 2097152 1G 83 Linux
/dev/sda3 3125248 8388607 5263360 2.5G 83 Linux

/dev/sda3:
New situation:
Disklabel type: dos
Disk identifier: 0xc1748067

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 1028095 1026048 501M 6 FAT16
/dev/sda2 1028096 3125247 2097152 1G 83 Linux
/dev/sda3 3125248 62333951 59208704 28.2G 83 Linux

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
e2fsck 1.46.5 (30-Dec-2021)
/dev/sda3 has unsupported feature(s): FEATURE_C12
e2fsck: Get a newer version of e2fsck!

root: ********** WARNING: Filesystem still has errors **********

resize2fs 1.46.5 (30-Dec-2021)
Please run 'e2fsck -f /dev/sda3' first.

= Raspberry Pi 4 Uboot is already in place, no changes needed.
= Removing the root password.
= Adding SSH key to authorized keys.

= Installation Complete! Insert into the rpi4 and boot.

There are some errors there, right? Well, I ignored them. RPi booted nicely, I was able to setup everything (more on that in some later blog) but then I have ran out of storage. Only then I noticed root filesystem was not extended (exactly as the error message says).

After some online search I figured I need e2fsprogs-1.47.0 or newer and (at the time?) it was only available in Fedora 39. So I upgraded and now I was able to write the image just fine:

$ sudo arm-image-installer --image Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz --media /dev/sda --addkey /home/jhutar/.ssh/id_rsa.pub --norootpass --resizefs --target=rpi4 -y
[sudo] password for jhutar:

=====================================================
= Selected Image:                                 
= Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
= Selected Media : /dev/sda
= U-Boot Target : rpi4
= Root Password will be removed.
= Root partition will be resized
= SSH Public Key /home/jhutar/.ssh/id_rsa.pub will be added.
=====================================================
 
*****************************************************
*****************************************************
******** WARNING! ALL DATA WILL BE DESTROYED ********
*****************************************************
*****************************************************
= Writing:
= Fedora-IoT-39.20231103.1-20231103.1.aarch64.raw.xz
= To: /dev/sda ....
4282384384 bytes (4.3 GB, 4.0 GiB) copied, 245 s, 17.5 MB/s
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 245.85 s, 17.5 MB/s
= Writing image complete!
= Resizing /dev/sda ....
Checking that no-one is using this disk right now ... OK

Disk /dev/sda: 29.72 GiB, 31914983424 bytes, 62333952 sectors
Disk model: UHSII uSD Reader
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc1748067

Old situation:

Device     Boot   Start     End Sectors  Size Id Type
/dev/sda1  *       2048 1028095 1026048  501M  6 FAT16
/dev/sda2       1028096 3125247 2097152    1G 83 Linux
/dev/sda3       3125248 8388607 5263360  2.5G 83 Linux

/dev/sda3:
New situation:
Disklabel type: dos
Disk identifier: 0xc1748067

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sda1  *       2048  1028095  1026048  501M  6 FAT16
/dev/sda2       1028096  3125247  2097152    1G 83 Linux
/dev/sda3       3125248 62333951 59208704 28.2G 83 Linux

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
root: 32041/164640 files (0.6% non-contiguous), 449099/657920 blocks
resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/sda3 to 7401088 (4k) blocks.
The filesystem on /dev/sda3 is now 7401088 (4k) blocks long.

= Raspberry Pi 4 Uboot is already in place, no changes needed.
= Removing the root password.
= Adding SSH key to authorized keys.

= Installation Complete! Insert into the rpi4 and boot.

Stick the card into RPi, connect power and ethernet cable and voila, I'm now able to SSH to RPi. I got the IP from my router management console from DHCP leases section.

Different Numpy results on different systems

Recently mine wife in her compute intensive project noticed strange issue when same input date and same code produced different output on different hosts in the cloud she is using. She tracked it down to this simple python & numpy test case:

#!/usr/bin/env python

import numpy

a = [[0.67115835, -0.74131401], [0.74131401, 0.67115835]]
b = [[-4.95494273, -1.77170756, ...], [1.87737564, 4.99951546, ...]]
c = numpy.matmul(a, b)
print(c)

On one host it was returning (correct result):

[[-4.71727605 -4.89530718 -4.71727605 -4.89530718 -4.71727605 -4.89530718
...

On different host it was returning (wrong result):

[[ 0.34761728  0.12429531  0.34761728  0.12429531 -0.13170853 -0.35074431
...

We have been googling a bit and found some tips:

• Same Python code, same data, different results on different machines: There is a suggestion to use export MKL_CBWR=AUTO - this did not helped
• Different results on different computers when using Scipy stack compiled with MKL: Some more tips are here: export MKL_CBWR=AVX; export OMP_NUM_THREADS=1 or with export MKL_CBWR=SSE4_2 - this did not helped again (anyway, our Numpy does not seem to be build with this Intel Math Kernel Library - see ldd output below)
• How can I make sure svd having same result on different machine: Here this is mentioned: export NUMEXPR_NUM_THREADS=1; export OPENBLAS_NUM_THREADS=1 - no luck, but finally: export OPENBLAS_CORETYPE=prescott - bingo!

According to OpenBLAS ussage instructions (OpenBLAS is "an optimized BLAS (Basic Linear Algebra Subprograms) library" if you have same knowleadge about it as I do), OPENBLAS_CORETYPE is environment variable which control the kernel selection. Looking at Prescott CPU description, it was launched in 2000, so is probably a safe default. Some more details about our setup:

Numpy in our setup is linked with these libraries:

ldd $( rpm -ql python3-numpy | grep '\.so$' ) | grep -v '\.so:$' | sed 's/([0-9a-zx]\+)/(...)/' | sort -u
	/lib64/ld-linux-x86-64.so.2 (...)
	libc.so.6 => /lib64/libc.so.6 (...)
	libdl.so.2 => /lib64/libdl.so.2 (...)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (...)
	libgfortran.so.5 => /lib64/libgfortran.so.5 (...)
	libm.so.6 => /lib64/libm.so.6 (...)
	libopenblasp.so.0 => /lib64/libopenblasp.so.0 (...)
	libpthread.so.0 => /lib64/libpthread.so.0 (...)
	libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...)
	libquadmath.so.0 => /lib64/libquadmath.so.0 (...)
	libutil.so.1 => /lib64/libutil.so.1 (...)
	linux-vdso.so.1 (...)

The code is packaged in Singularity containers and is running on Metacentrum cloud. Two machines we have hit were - the one with correct result:

Singularity> tail -n 28 /proc/cpuinfo

processor       : 15
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel Xeon E3-12xx v2 (Ivy Bridge)
stepping        : 9
microcode       : 0x1
cpu MHz         : 2199.998
cache size      : 16384 KB
physical id     : 15
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 15
initial apicid  : 15
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt
+tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm kaiser fsgsbase smep erms xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 4399.99
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Singularity> uname -a
Linux [hostname] 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21) x86_64 x86_64 x86_64 GNU/Linux

The other host - the one with wrong results:

Singularity> tail -n 28 /proc/cpuinfo

processor       : 63
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
stepping        : 4
microcode       : 0x200004d
cpu MHz         : 2399.392
cache size      : 22528 KB
physical id     : 1
siblings        : 32
core id         : 15
cpu cores       : 16
apicid          : 63
initial apicid  : 63
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology
+nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault
+epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt
+clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips        : 4201.71
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

Singularity> uname -a
Linux [hostname] 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 x86_64 x86_64 GNU/Linux

Packages in the container are:

• python3-3.7.4-1.fc30.x86_64
• python3-numpy-1.16.4-2.fc30.x86_64

If you want to try, full test case is here:

import numpy

a = [[0.67115835,-0.74131401],[0.74131401,0.67115835]]
b = [[-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936],[1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674]]
c = numpy.matmul(a,b)
print(c)