Kinda SQL "join" in Prometheus

I'm using Prometheus query language, PromQL, quite a bit these days. But all I do are very simple queries like sum(...) or rate(...[5m]) on a OpenShift cluster I work with.

For few weeks now, mine inner me was bothered with one slightly more complex stuff. To filter one metric by label from different metric - something like JOIN in SQL world. Specifically, I wanted to see number of pods running on each cluster node with "worker" role.

We have (I'm on OpenShift 4.13) kube_node_role{role="worker"} (AFAICT this is what we call "vector" in PromQL) that have these labels:

Name            container             endpoint    job                 namespace             node                     prometheus                role    service             Value
kube_node_role  kube-rbac-proxy-main  https-main  kube-state-metrics  openshift-monitoring  ip-1-2-3-4.ec2.internal  openshift-monitoring/k8s  worker  kube-state-metrics  1
kube_node_role  kube-rbac-proxy-main  https-main  kube-state-metrics  openshift-monitoring  ip-1-2-3-5.ec2.internal  openshift-monitoring/k8s  worker  kube-state-metrics  1
[...]

And we have kube_pod_info with these labels:

Name           container             created_by_kind  created_by_name  endpoint    host_ip        host_network  job                 namespace       node                     pod                                               pod_ip       priority_class           prometheus                service             uid                                   Value
kube_pod_info  kube-rbac-proxy-main  <none>           <none>           https-main  10.201.24.232  false         kube-state-metrics  openshift-etcd  ip-1-2-3-6.ec2.internal  etcd-guard-ip-10-201-24-232.ec2.internal          10.128.2.14  system-cluster-critical  openshift-monitoring/k8s  kube-state-metrics  a2eec7b0-9f29-42b4-853d-6919d963ffa1  1
kube_pod_info  kube-rbac-proxy-main  <none>           <none>           https-main  10.201.24.232  false         kube-state-metrics  openshift-etcd  ip-1-2-3-6.ec2.internal  revision-pruner-13-ip-10-201-24-232.ec2.internal  10.128.2.4   system-node-critical     openshift-monitoring/k8s  kube-state-metrics  df5cdd67-b0f5-4896-b0b0-85095a9f3122  1

We will use on(...) and group_left(...) PromQL operators. I had some issues understanding what these do, so here is mine interpretation:

* because values are always 1 in these vectors, it is safe to multiply these.

on(...) allows me to define common label(s) that should be used to match two different vectors.

group_left(...) ... thinking, thinking, nah. I forgot mine mental model here :-/

And this is the final query I used:

sum(
    kube_pod_info{} * on(node) group_left(role) kube_node_role{role="worker"}
) by(node)

These links helped me a lot:

• How to have labels for machine roles
• Prometheus Cheat Sheet - How to Join Multiple Metrics (Vector Matching)
• PromQL operators

My Prometheus@OpenShift cheat-sheet

Prometheus is monitoring solution in OpenShift and I'm reading some basic out of it after some performance tests. Here are the queries I'm using:

Get CPU consumption by pods xyz...:

sum(pod_name:container_cpu_usage:sum{pod_name=~'xyz.*',namespace='qa'})

Now for memory usage (these "POD" and "''" container names seems to be doubling the value):

sum(container_memory_usage_bytes{namespace='qa', pod_name=~'xyz.*', container_name!='POD', container_name!=''})

Also see these nice examples on how to construct query.

To Querying Prometheus via API, I have used range query and this Python code:

assert start is not None and end is not None, \
    "We need timerange to approach Prometheus"

# Get data from Prometheus
token = 'your `oc whoami -t`'
url = 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query_range'   # I'm running this inside the cluster, so I can use internal hostname
headers = {
    'Authorization': f'Bearer {token}',
    'Content-Type': 'application/json',
}
params = {
    'query': monitoring_query,   # this will be some query from above
    'step': monitoring_step,   # using 60 seconds here
    'start': start.strftime('%s'),
    'end': end.strftime('%s'),
}
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)   # security is hard ;)
response = requests.get(url, headers=headers, params=params, verify=False)

# Check that what we got back seems OK
response.raise_for_status()
json_response = response.json()
assert json_response['status'] == 'success'
assert 'data' in json_response
assert 'result' in json_response['data']
assert len(json_response['data']['result']) == 1
assert 'values' in json_response['data']['result'][0]

data = [float(i[1]) for i in json_response['data']['result'][0]['values']]

Running wsgi application in OpenShift v.3 for the first time

For some time I'm running publicly available web application Brno People team uses to determine technical interests of employee candidates. The app was running on the OpenShift v.2, but that was discontinued and I had to port it to OpenShift v.3. I was postponing the task for multiple moths and got to the state when v.2 instances were finally discontinued. It turned out that porting is not that hard. This is what I have done.

Note that I'm using Red Hat's Employee account, so some paths might be different when OpenShift is being used "normally" (you will see something like ....starter-us-east-1.openshift.com instead of mine ....rh-us-east-1.openshift.com).

Because I need to put some private data into the image, I want the image to be accessible only from mine OpenShift Online account. Anyway, I have created Dockerfile based on Fedora Dockerfile template for Python (is it official?) like this:

FROM fedora
MAINTAINER Jan Hutar <jhutar@redhat.com>
RUN dnf -y update && dnf clean all
RUN dnf -y install subversion python mod_wsgi && dnf clean all
RUN ...
VOLUME ["/xyz/data/"]
WORKDIR /xyz
EXPOSE 8080
USER root
CMD ["python", "/xyz/application"]

TODO for the container is: move to Python 3 (so I do not need to install python2 and dependencies, figure out how to have the private data available to the container but not being part of it (it is quite big directory structure), go through these nice General Container Image Guidelines and explore this Image Metadata thingy.

Once I had that, I needed to login to the OpenShift's registry, build my image locally, test it and push it:

sudo docker build --tag selftest .
sudo docker run -ti --publish 8080:8080 --volume $( pwd )/data/:/xyz/data/ xyz   # now I can check if all looks sane with `firefox http://localhost:8080`
oc whoami -t   # this shows token I can use below
sudo docker login -u <username> -p <token> registry.rh-us-east-1.openshift.com
sudo docker tag xyz registry.rh-us-east-1.openshift.com/xyz/xyz
sudo docker push registry.rh-us-east-1.openshift.com/xyz/xyz

Now I have used Console on https://console.rh-us-east-1.openshift.com/ to create new application, then added a deployment to that application with Add to Project -> Deploy Image and selected (well, I could use cli tool oc for that):

• surprisingly you do not choose "Image Name" here
• but you choose "Image Stream Tag" with:
  • Namespace: selftest
  • Image Stream: selftest
  • Tag: latest

Next step looks logical, but I got stuck on it for some time, but OpenShift folks helped me (thanks Jiří!). I just need to be aware of OpenShift Online Restrictions.

So, because I wanted persistent storage and because my account uses Amazon EC2, I can not use "Shared Access (RWX)" storage type (useful when new pod is starting while old pod is still running), I had to change process of new pods start to first stop old and the start new: Applications -> Deployments -> my deployment -> Actions -> Edit -> Strategy Type: Recreate. I have created a storage with "RWO (Read-Write-Once)" access mode, added it to the deployment (... -> Actions -> Add Storage) and made sure that that storage is the only one attached to the deployment (... -> Actions -> Edit YAML and check that keys spec.template.spec.containers.env.volumeMounts and spec.template.spec.volumes only contain one volume you have just attached). In my case, there is this in the YAML definition:

[...]
    spec:
      containers:
        - env:
          [...]
          volumeMounts:
            - mountPath: /xyz/data
              name: volume-jt8t6
      [...]
      volumes:
        - name: volume-jt8t6
          persistentVolumeClaim:
            claimName: xyz-storage-claim
[...]

When working with this, I have also used ... -> Actions -> Pause Rollouts. It is also possible to configure environment variable for a deployment in ... -> Actions -> Edit -> Environment Variables which is useful to pass passwords and stuff into your app (so I do not need to store them in the image). In the app I use something like import os; SMTP_SERVER_PASSWORD = os.getenv('XYZ_SMTP_SERVER_PASSWORD', default='') to read that.

To make the app available from outside world, I have created a route in Applications -> Routes -> Create Route. It created domain like http://xyz-route-xyz.6923.rh-us-east-1.openshiftapps.com for me.

Now, looks like everything works for me and I'm kinda surprised how easy it was. I plan to get nicer domain and configure its CNAME DNS record and to explore monitoring possibilities OpenShift have. I'll see how it goes.