Alertmanager

Checks Prometheus AlertManager for any firing alerts.

The following example transforms the list of alerts so that each alert becomes a single check result. Without the transform the health check fails if any alerts are firing.

alert-manager-check.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: alertmanager
spec:
  schedule: "@every 5m"
  alertmanager:
    - url: https://alertmanager.demo.aws.flanksource.com
      name: alertmanager-check
      alerts:
        - .*
      ignore:
        - KubeScheduler.*
      exclude_filters:
        namespace: elastic-system
      transform:
        expr: |
          results.alerts.map(r, {
            'name': r.name + r.fingerprint,
            'namespace': 'namespace' in r.labels ? r.labels.namespace : '',
            'labels': r.labels,
            'icon': 'alert',
            'message': r.message,
            'description': r.message,
          }).toJSON()
      relationships:
        components:
          - name:
              label: pod
            namespace:
              label: namespace
            type:
              value: KubernetesPod
        configs:
          - name:
              label: pod
            namespace:
              label: namespace
            type:
              value: Kubernetes::Pod

Field	Description	Scheme
name*	Name of the check, must be unique within the canary	string
url*	The URL of the Prometheus Alertmanager instance	string
alerts	A list of alert prefix names to include	[]string
filters	A map of label to value prefixes to find alerts on	map[string]string
ignore	A list of alert prefix names to exclude	[]string
description	Description for the check	string
display	Expression to change the formatting of the display	Expression
icon	Icon for overwriting default icon on the dashboard	Icon
labels	Labels for check	map[string]string
metrics	Metrics to export from	[]Metrics
test	Evaluate whether a check is healthy	Expression
transform	Transform data from a check into multiple individual checks	Expression

Inserting checks into different namespaces

You can specify different namespaces for checks using the namespace field. This is helpful when checks are dynamically generated via transformation and need to be assigned their respective namespace.

alert-manager-check.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: alertmanager
spec:
  schedule: "@every 5m"
  alertmanager:
    - url: https://alertmanager.demo.aws.flanksource.com
      name: alertmanager-check
      alerts:
        - .*
      ignore:
        - KubeScheduler.*
      exclude_filters:
        namespace: elastic-system
      transform:
        expr: |
          results.alerts.map(r, {
            'name': r.name + r.fingerprint,
            'namespace': 'namespace' in r.labels ? r.labels.namespace : '',
            'labels': r.labels,
            'icon': 'alert',
            'message': r.message,
            'description': r.message,
          }).toJSON()
      relationships:
        components:
          - name:
              label: pod
            namespace:
              label: namespace
            type:
              value: KubernetesPod
        configs:
          - name:
              label: pod
            namespace:
              label: namespace
            type:
              value: Kubernetes::Pod

The above alertmanager canary can reside on any namespace and the check it generates are assigned the correct namespace derived from the alerts.