Prometheus

The Prometheus Check connects to the Prometheus host, performs the desired query, and displays the results.

prometheus.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: prometheus
spec:
  schedule: "@every 5m"
  prometheus:
    - url: https://prometheus.demo.aws.flanksource.com/
      name: prometheus-check
      query: kubernetes_build_info{job!~"kube-dns|coredns"}
      display:
        expr: results[0].git_version

Field	Description	Scheme
name*	Name of the check, must be unique within the canary	string
query*	PromQL query to execute
url*	Prometheus instance, defaults to --prometheus command line argument
bearer	Bearer token to use for authentication	EnvVar
oauth
password	Password for basic auth	EnvVar
username	Username for basic auth	EnvVar
description	Description for the check	string
display	Expression to change the formatting of the display
icon	Icon for overwriting default icon on the dashboard	Icon
labels	Labels for check
metrics	Metrics to export from
test	Evaluate whether a check is healthy
transform	Transform data from a check into multiple individual checks

OAuth

Field	Description	Scheme
clientID	Client ID	EnvVar
clientSecret	Client Secret	EnvVar
params	OAuth parameters when requesting a token
scope	Scopes to request
tokenURL	Token URL	string

Result Variables

Name	Description	Scheme
value		float
firstValue	Number of rows returned	int
results	A list of results	[]map[string]interface

Examples

Create a check per prometheus job using a transform

jobs.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: prometheus-jobs
spec:
  schedule: "@every 5m"
  prometheus:
    - name: Jobs
      query: up{namespace!~"kube-system|monitoring"}
      url: http://prometheus.monitoring.svc:9090
      transform:
        expr: |
          dyn(results).map(r, {
            'name': r.job,
            'namespace': 'namespace' in r ? r.namespace : '',
            'labels': r.omit(["value", "__name__"]),
            'pass': r["value"] > 0
          }).toJSON()

Create a check per failing job only

jobs.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: prometheus-failing-jobs
spec:
  schedule: "@every 5m"
  prometheus:
    - name: Jobs
      query: up{namespace!~"kube-system|monitoring"} == 0
      url: http://prometheus.monitoring.svc:9090
      transform:
        expr: |
          dyn(results).map(r, {
            'name': r.job,
            'namespace': 'namespace' in r ? r.namespace : '',
            'labels': r.omit(["value", "__name__"]),
            'pass': false
          }).toJSON()