Tekton notes

Some time ago I was tasked to create a pipeline in Tekton and here comes some my notes I would like to know few days back :-)

1.  It is not that hard. It is just a fancy way how split your shell automation script :-)
2. Tasks are not that useful on it's own (I think), you have to stack them into a Pipeline, but Tekton Getting started with Tasks is nice start. Once you need more details, see Tasks.
3. Pipelines are the core thing and starting with Getting Started with Pipelines helped me a lot. Later I was looking into Pipelines as well.
4. Blog post Building in Kubernetes Using Tekton was also very helpful. Also used my company's CI/CD guide here ad there.
5. Tekton Hub is full of tasks (and more) and I was able to easilly see documentation for them and more importantly the actual YAML behind them - having a practical examples of how the tasks could look like behind simple hello-world tasks was very helpful. E.g. see kubernetes-actions and git-clone or git-cli.
   
6. To test things, I have used Kind as "Getting started" guide suggested and Tekton installed there really easily.
7. Creating user on Kind to be able to follow some Tekton how-tos out there that are building container using Tekton was beyond mine possibilities. I did not needed to build images, so I'm good.
8. To be able to talk to the app running in Kind cluster, I used Ingress NGINX and it's rewrite rule annotation as mine app did not liked extra data in URI. Mine specific example: perfcale-demo-app-ingress.yaml.
   
9. Results are quite simple concept. You just configure them in the task and in the script you redirect the value (their size is quite limited) to filename stored in some variable.
10. When something does not make sense, you can always add a step to your task with sleep 1000 and kubectl exec -ti pod/... -- bash.
11. Every pipeline run name have to be unique. It would be boring to create new ones with kubectl apply -f ... on each of the attempts I have done without some script, but having generateName in pipeline run metadata and using kubectl create -f ... saved my day.
    

At the end mine pipeline worked like this:

1. Clones required repos:
1. Demo application: perfscale-demo-app
2. YAMLs and misc: my-tekton-perfscale-experiment
3. Results repo: my-tekton-perfscale-experiment-results
2. Deploys the demo application (no need to build images as it is done by quay.io)
   
1. It is a simple bank-like application exposing REST API
2. There is a Locust framework based perf test included with the application that stresses the API and measures RPS
3. Application consist of one pod with PostgreSQL and another one with application itself and Gunicorn application server
3. Populates test data into the application (code for it is built in into the demo application for ease of use)
   
4. Runs the Locust framework based perf test from demo application's repository, but wrapped in thin OPL helper that stores the test results in nice JSON
5. Runs a script that loads historical results for the same test with same parameters and determines if new result is PASS or FAIL
6. Adds a new result into results repository and pushes it to GitHub
7. Deletes a demo app deployment

 The commands I have used most when working on the pipeline were:

• kubectl apply --filename pipeline.yaml - to apply changes I have done to the pipeline
  
• kubectl create --filename pipeline-run.yaml - to create new pipeline run with random suffix
  
• tkn pipelinerun logs --follow --last --all --prefix - to follow logs of the current pipeline run
• tkn pipelinerun delete --all --force - to remove all previous pipeline runs
  

Use Google Chat webhook API to send message to channel

Sending message to the Google Chat (chat.google.com, recently integrated to mail.google.com/chat/) is surprisingly simple with their webhook API. Just took me some time to figure out a data structure to send (although it it very simple as I found on Incoming webhook with Python page):

curl -X POST -H "Content-Type: application/json; charset=UTF-8" --data '{"text": "Hello @jhutar, how are you?"}' "https://chat.googleapis.com/v1/spaces/.../messages?key=...&token=..."
{
  "name": "spaces/.../messages/...",
  "sender": {
    "name": "users/...",
    "displayName": "Jenkins incomming webhook",
    "avatarUrl": "",
    "email": "",
    "domainId": "",
    "type": "BOT",
    "isAnonymous": false
  },
  "text": "Hello @jhutar",
  "cards": [],
  "previewText": "",
  "annotations": [],
  "thread": {
    "name": "spaces/.../threads/..."
  },
  "space": {
    "name": "spaces/...",
    "type": "ROOM",
    "singleUserBotDm": false,
    "threaded": true,
    "displayName": "Name of the channel"
  },
  "fallbackText": "",
  "argumentText": "Hello @jhutar, how are you?",
  "attachment": [],
  "createTime": "2021-10-11T22:07:39.490063Z",
  "lastUpdateTime": "2021-10-11T22:07:39.490063Z"
}

Using redirect() on https:// site handled by Flask -> Gunicorn -> Nginx redirects me to http

And this might be hard to notice as we usually configure Nginx to also redirect all http requests to https, so at the end you end up on correct link, but going through http is not nice and it can also break CORS as I was told.

There are two parts of the problem.

First, Nginx need to set certain headers when proxying application running in Gunicorn (e.g. see them in Deploying Gunicornbehind Nginx):

proxy_set_header    Host                $host;
proxy_set_header    X-Real-IP           $remote_addr;
proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
proxy_set_header    X-Forwarded-Proto   $scheme;
proxy_set_header    X-Forwarded-Host    $http_host;
proxy_pass  http://my_app;

Second, Flask app needs to know to use content of these headers to overwrite normal request metadata (it is called Proxy Fix and brought to us by Werkzung which is a Flask's dependency):

from flask import Flask
from werkzeug.middleware.proxy_fix import ProxyFix

app = Flask(__name__, instance_relative_config=True)
app.wsgi_app = ProxyFix(app.wsgi_app, x_for=1, x_proto=1, x_host=1)

Obligatory note: see the docs linked above as these numbers are actually important from security point of view.

Accessing Red Hat OpenShift Streams for Apache Kafka from Python

Recently Red Hat launched a way how to get managed Kafka instance and you can get one for 2 days for free. There is a limit for 1 MB per second. So far I was only using Kafka without any auth and without any encription, so here is what I had to do to make it work - typing here so I do not need to reinvent once I forgot it :-) I'm using python-kafka.

I have created a cluster and under it's "Connection" menu item I got bootstrap server jhutar--c-jc--gksg-rukm-fu-a.bf2.kafka-stage.rhcloud.com:443. It also advised me to create a service account, so I created one and it generated "Client ID" like srvc-acct-00000000-0000-0000-0000-000000000000 and "Client secret" like 00000000-0000-0000-0000-000000000000. Although "SASL/OAUTHBEARER" authentication method is recommended, as of now it is too complicated for my poor head, so I used "SASL/PLAIN" where you just use "Client ID" as username and "Client secret" as password. To create a topic, there is UI as well

To create producer and consumer:

producer = KafkaProducer(
    bootstrap_servers='jhutar--c-jc--gksg-rukm-fu-a.bf2.kafka-stage.rhcloud.com:443',
    sasl_plain_username='srvc-acct-00000000-0000-0000-0000-000000000000',
    sasl_plain_password='00000000-0000-0000-0000-000000000000',
    security_protocol='SASL_SSL',
    sasl_mechanism='PLAIN',
)

And consumer needs same parameters:

consumer = KafkaConsumer(
    '<topic>',
    bootstrap_servers='jhutar--c-jc--gksg-rukm-fu-a.bf2.kafka-stage.rhcloud.com:443',
    sasl_plain_username='srvc-acct-00000000-0000-0000-0000-000000000000',
    sasl_plain_password='00000000-0000-0000-0000-000000000000',
    security_protocol='SASL_SSL',
    sasl_mechanism='PLAIN',
)

How to access oldish Dell DRAC console? Install old Firefox and Java in Docker container

Sometimes I need to access DRAC console (i.e. "remote screen") of some older Dell system - in this case it is PowerEdge R610 and "About" says "Integrated Dell Remote Access Controller 6 - Enterprise, Version 1.98, © 2008-2011 Dell Inc.". I have tried bunch of browsers on my not super recent Fedora 30 and it failed. One solution I have found is to access the console with Firefox and IcedTea plugin from Fedora 27. VM feels too heavy for this usecase, so just allow connections to my X server:

# xhost local:root

start Fedora 27 container with some extra vars and mounts:

# docker run \
        --network host \
        -e DISPLAY=:0.0 \
        -v /tmp/.X11-unix:/tmp/.X11-unix \
        -v /root/.Xauthority:/root/.Xauthority:rw \
        -ti fedora:27 /bin/bash

and then in the container install all needed and run the browser:

# dnf -y install firefox xorg-x11-xauth icedtea-web
# firefox

Now I'm able to open the console, hooray!

Dumping my notes on Jenkins shared library use in declarative pipeline

Recently I have tasked myself to work on how we organize our Jenkins jobs code. We are using Jenkins declarative pipeline (officially that is "simplified and opinionated syntax on top of the Pipeline sub-systems", but basically it is "nicely structured, but they forbid you to use mostly anything fancy in you Groovy code").

In our case we have lots (and it is slowly growing) of jobs which are running tests from different directories in a same way, but with different parameters. Also we have another set of jobs that are checking something and in some case they trigger the test jobs. This all loudly calls for sharing the code, so I wanted to take a look at how to do it.

Here is list of tabs I'm closing now once I'm done :)

• Shared libraries - Create another git repo with vars/ directory and share the code there
• Defining Declarative Pipelines - You can define function that implements whole declarative pipeline (you can also define custom step, but you can not define shared stage)
• Groovy Maps - If you want to pass lots of parameters to the function, passing a Map is very handy and easy to read - in Python equivalent would be passing a dict I think
• Cannot define variable in pipeline stage - In declarative pipeline you can not define variable directly, use this workaround
• sh: Shell Script - Jenkins help for sh step
• Catching both output and error code - spoiler: try { ... } catch ( ... ) { ... } is also forbidden in declarative pipeline, use script { ... } again
• when - this is way how to skip a stage in declarative pipeline based on something
• Writing file in groovy - This would probably work, but Jenkins was giving me lots of security related warnings
• Library variable in jenkins shared library - how to get variables from vars/someVariables.groovy in vars/someFunction.groovy

Different Numpy results on different systems

Recently mine wife in her compute intensive project noticed strange issue when same input date and same code produced different output on different hosts in the cloud she is using. She tracked it down to this simple python & numpy test case:

#!/usr/bin/env python

import numpy

a = [[0.67115835, -0.74131401], [0.74131401, 0.67115835]]
b = [[-4.95494273, -1.77170756, ...], [1.87737564, 4.99951546, ...]]
c = numpy.matmul(a, b)
print(c)

On one host it was returning (correct result):

[[-4.71727605 -4.89530718 -4.71727605 -4.89530718 -4.71727605 -4.89530718
...

On different host it was returning (wrong result):

[[ 0.34761728  0.12429531  0.34761728  0.12429531 -0.13170853 -0.35074431
...

We have been googling a bit and found some tips:

• Same Python code, same data, different results on different machines: There is a suggestion to use export MKL_CBWR=AUTO - this did not helped
• Different results on different computers when using Scipy stack compiled with MKL: Some more tips are here: export MKL_CBWR=AVX; export OMP_NUM_THREADS=1 or with export MKL_CBWR=SSE4_2 - this did not helped again (anyway, our Numpy does not seem to be build with this Intel Math Kernel Library - see ldd output below)
• How can I make sure svd having same result on different machine: Here this is mentioned: export NUMEXPR_NUM_THREADS=1; export OPENBLAS_NUM_THREADS=1 - no luck, but finally: export OPENBLAS_CORETYPE=prescott - bingo!

According to OpenBLAS ussage instructions (OpenBLAS is "an optimized BLAS (Basic Linear Algebra Subprograms) library" if you have same knowleadge about it as I do), OPENBLAS_CORETYPE is environment variable which control the kernel selection. Looking at Prescott CPU description, it was launched in 2000, so is probably a safe default. Some more details about our setup:

Numpy in our setup is linked with these libraries:

ldd $( rpm -ql python3-numpy | grep '\.so$' ) | grep -v '\.so:$' | sed 's/([0-9a-zx]\+)/(...)/' | sort -u
	/lib64/ld-linux-x86-64.so.2 (...)
	libc.so.6 => /lib64/libc.so.6 (...)
	libdl.so.2 => /lib64/libdl.so.2 (...)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (...)
	libgfortran.so.5 => /lib64/libgfortran.so.5 (...)
	libm.so.6 => /lib64/libm.so.6 (...)
	libopenblasp.so.0 => /lib64/libopenblasp.so.0 (...)
	libpthread.so.0 => /lib64/libpthread.so.0 (...)
	libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...)
	libquadmath.so.0 => /lib64/libquadmath.so.0 (...)
	libutil.so.1 => /lib64/libutil.so.1 (...)
	linux-vdso.so.1 (...)

The code is packaged in Singularity containers and is running on Metacentrum cloud. Two machines we have hit were - the one with correct result:

Singularity> tail -n 28 /proc/cpuinfo

processor       : 15
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel Xeon E3-12xx v2 (Ivy Bridge)
stepping        : 9
microcode       : 0x1
cpu MHz         : 2199.998
cache size      : 16384 KB
physical id     : 15
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 15
initial apicid  : 15
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt
+tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm kaiser fsgsbase smep erms xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 4399.99
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Singularity> uname -a
Linux [hostname] 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21) x86_64 x86_64 x86_64 GNU/Linux

The other host - the one with wrong results:

Singularity> tail -n 28 /proc/cpuinfo

processor       : 63
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
stepping        : 4
microcode       : 0x200004d
cpu MHz         : 2399.392
cache size      : 22528 KB
physical id     : 1
siblings        : 32
core id         : 15
cpu cores       : 16
apicid          : 63
initial apicid  : 63
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology
+nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault
+epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt
+clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips        : 4201.71
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

Singularity> uname -a
Linux [hostname] 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 x86_64 x86_64 GNU/Linux

Packages in the container are:

• python3-3.7.4-1.fc30.x86_64
• python3-numpy-1.16.4-2.fc30.x86_64

If you want to try, full test case is here:

import numpy

a = [[0.67115835,-0.74131401],[0.74131401,0.67115835]]
b = [[-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,-4.95494273,-1.77170756,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,0.64695557,-3.83073022,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,-1.91809893,2.14768601,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.99713467,-0.2208969,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936,3.96850733,4.57202936],[1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,1.87737564,4.99951546,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,3.8703295,-0.52141675,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,-2.14706037,1.84069058,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,0.15277681,-3.98429871,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674,-2.93121093,-5.91274674]]
c = numpy.matmul(a,b)
print(c)