My Prometheus@OpenShift cheat-sheet

Prometheus is monitoring solution in OpenShift and I'm reading some basic out of it after some performance tests. Here are the queries I'm using:

Get CPU consumption by pods xyz...:

sum(pod_name:container_cpu_usage:sum{pod_name=~'xyz.*',namespace='qa'})

Now for memory usage (these "POD" and "''" container names seems to be doubling the value):

sum(container_memory_usage_bytes{namespace='qa', pod_name=~'xyz.*', container_name!='POD', container_name!=''})

Also see these nice examples on how to construct query.

To Querying Prometheus via API, I have used range query and this Python code:

assert start is not None and end is not None, \
    "We need timerange to approach Prometheus"

# Get data from Prometheus
token = 'your `oc whoami -t`'
url = 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query_range'   # I'm running this inside the cluster, so I can use internal hostname
headers = {
    'Authorization': f'Bearer {token}',
    'Content-Type': 'application/json',
}
params = {
    'query': monitoring_query,   # this will be some query from above
    'step': monitoring_step,   # using 60 seconds here
    'start': start.strftime('%s'),
    'end': end.strftime('%s'),
}
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)   # security is hard ;)
response = requests.get(url, headers=headers, params=params, verify=False)

# Check that what we got back seems OK
response.raise_for_status()
json_response = response.json()
assert json_response['status'] == 'success'
assert 'data' in json_response
assert 'result' in json_response['data']
assert len(json_response['data']['result']) == 1
assert 'values' in json_response['data']['result'][0]

data = [float(i[1]) for i in json_response['data']['result'][0]['values']]