Kubernetes

The Kubernetes check performs requests on Kubernetes resources such as Pods to get the desired information.

kubernetes.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: kube-system-checks
spec:
  schedule: "@every 5m"
  kubernetes:
    - namespace: kube-system
      name: kube-system
      kind: Pod
      # ready: true
      # resource:
      #   labelSelector: k8s-app=kube-dns
      namespaceSelector:
        name: default
      display:
        expr: |
          dyn(results).
          map(i, i.Object).
          filter(i, !k8s.isHealthy(i)).
          map(i, "%s/%s -> %s".format([i.metadata.namespace, i.metadata.name, k8s.getHealth(i).message])).join('\n')
      test:
        expr: dyn(results).all(x, k8s.isHealthy(x))

Field	Description	Scheme
kind*	Kubernetes object kind	string
name*	Name of the check, must be unique within the canary	string
healthy	Fail the check if any resources are unhealthy	bool
ignore	Ignore the specified resources from the fetched resources. Can be a glob pattern.	[]glob
namespace	Failing checks are placed in this namespace, useful if you have shared namespaces. NOTE: this does not change the namespace of the resources being queried
namespaceSelector	Filters namespaces by name or labels	Resource Selector
ready	Fail the check if any resources are not ready	bool
resource	Filters resources by name, namespace, or labels	Resource Selector
description	Description for the check	string
display	Expression to change the formatting of the display	Expression
icon	Icon for overwriting default icon on the dashboard	Icon
labels	Labels for check	map[string]string
metrics	Metrics to export from	[]Metrics
test	Evaluate whether a check is healthy	Expression
transform	Transform data from a check into multiple individual checks	Expression
kubeconfig	Path to a kubeconfig on disk, or a reference to an existing secret	EnvVar

Resource Selector

Field	Description	Scheme	Required
name	Name of the component/config	string	No
namespace	Select resources in this namespace only, if empty find resources in all namespaces	string	No
labelSelector	Kubernetes Style Label Selector	LabelSelector	No
fieldSelector	Kubernetes Style Field Selector Property fields of the component in kubernetes format (or database columns: owner, topology_id, parent_id)	FieldSelector	No
search	Search for resources via key value pairs using parsing expression grammar	Search	No

Search

The query syntax is field1=value1 field2>value2 field3=value3* field4=*value4. * is for prefix and suffix matching.

Supported operators:

Operator	Syntax	Types
=	field=value	string int json
!=	field!=value	string int json
*	field=*value or field=value*	string int
> <	field>value or field<value	datetime int

Healthy

Using healthy: true is functionally equivalent to:

  test:
        expr: dyn(results).all(x, k8s.isHealthy(x))

kubnetes-healthy.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: kube-system-checks
spec:
  interval: 30
  kubernetes:
    - namespace: kube-system
      name: kube-system
      kind: Pod
      healthy: true
      resource:
        labelSelector: k8s-app=kube-dns
      namespaceSelector:
        name: kube-system

See the CEL function k8s.isHealthy for more details

Ready

Similar to the healthy flag, there's also a ready flag which is functionally equivalent to having the following test expression

dyn(results).all(x, k8s.isReady(x))

---

Checking for certificate readiness

cert-manager.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: cert-manager
spec:
  schedule: "@every 15m"
  kubernetes:
    - name: cert-manager-check
      kind: Certificate
      test:
        expr: |
          dyn(results).
          map(i, i.Object).
          filter(i, i.status.conditions[0].status != "True").size() == 0
      display:
        expr: |
          dyn(results).
          map(i, i.Object).
          filter(i, i.status.conditions[0].status != "True").
          map(i, "%s/%s -> %s".format([i.metadata.namespace, i.metadata.name, i.status.conditions[0].message])).join('\n')

Remote clusters

A single canary-checker instance can connect to any number of remote clusters via custom kubeconfig. Either the kubeconfig itself or the path to the kubeconfig can be provided.

kubeconfig from kubernetes secret

remote-cluster.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: pod-access-check
spec:
  schedule: "@every 5m"
  kubernetes:
    - name: pod access on aws cluster
      namespace: default
      description: "deploy httpbin"
      kubeconfig:
        valueFrom:
          secretKeyRef:
            name: aws-kubeconfig
            key: kubeconfig
      kind: Pod
      ready: true
      namespaceSelector:
        name: default

Kubeconfig inline

remote-cluster.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: pod-access-check
spec:
  schedule: "@every 5m"
  kubernetes:
    - name: pod access on aws cluster
      namespace: default
      kubeconfig:
        value: |
          apiVersion: v1
          clusters:
              - cluster:
                  certificate-authority-data: xxxxx
                  server: https://xxxxx.sk1.eu-west-1.eks.amazonaws.com
                name: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
          contexts:
              - context:
                  cluster: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
                  namespace: mission-control
                  user: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
                name: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
          current-context: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
          kind: Config
          preferences: {}
          users:
              - name: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
                user:
                  exec:
                      ....
      kind: Pod
      ready: true
      namespaceSelector:
        name: default

Kubeconfig from local filesystem

remote-cluster.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: pod-access-check
spec:
  schedule: "@every 5m"
  kubernetes:
    - name: pod access on aws cluster
      namespace: default
      kubeconfig:
        value: /root/.kube/aws-kubeconfig
      kind: Pod
      ready: true
      namespaceSelector:
        name: default