Kubernetes Resource

The Kubernetes resource check creates kubernetes resources based on the provided manifests & perform checks on them. Some common use case of this check would be to see if a service is accessible via the ingress as shown in the example below.

ingress_test.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: ingress-test
  labels:
    "Expected-Fail": "false"
spec:
  schedule: "@every 5m"
  kubernetesResource:
    - name: ingress-accessibility-check
      namespace: default
      description: "deploy httpbin & check that it's accessible via ingress"
      waitFor:
        expr: "dyn(resources).all(r, k8s.isReady(r))"
        interval: 2s
        timeout: 5m
      staticResources:
        - apiVersion: networking.k8s.io/v1
          kind: Ingress
          metadata:
            name: httpbin
            namespace: default
          spec:
            rules:
              - host: "httpbin.127.0.0.1.nip.io"
                http:
                  paths:
                    - pathType: Prefix
                      path: /
                      backend:
                        service:
                          name: httpbin
                          port:
                            number: 80
      resources:
        - apiVersion: v1
          kind: Pod
          metadata:
            name: httpbin
            namespace: default
            labels:
              app: httpbin
          spec:
            containers:
              - name: httpbin
                image: "kennethreitz/httpbin:latest"
                ports:
                  - containerPort: 80
        - apiVersion: v1
          kind: Service
          metadata:
            name: httpbin
            namespace: default
          spec:
            selector:
              app: httpbin
            ports:
              - port: 80
                targetPort: 80
      checks:
        - http:
            - name: Call httpbin via ingress
              url: "http://ingress-nginx.ingress-nginx.svc"
              headers:
                - name: Host
                  value: "{{(index ((index .staticResources 0).Object.spec.rules) 0).host}}"
      checkRetries:
        delay: 3s
        interval: 2s
        timeout: 5m

Field	Description	Scheme
`name*`	Name of the check, must be unique within the canary	string
`resources*`	Manifests that should be applied	`[]KubernetesManifest`
`checkRetries`	Retry configuration for the checks	CheckRetries
`checks`	canary spec for the checks to be performed after the resources are created	CanarySpec
`clearResources`	When set to true, resources from previous checks are deleted before every run. Even though the resources are deleted at the end of a check, setting this to `true` guarantees that there are no leftover resources from a previous failed run.	bool
`staticResources`	Static resources are like `resources` but preserved between checks. They are only deleted when the canary is deleted.	`[]KubernetesManifest`
`waitFor`	Specify the desired state of the static/non-static resources before running the checks	WaitFor
`description`	Description for the check	string
`display`	Expression to change the formatting of the display	`Expression`
`icon`	Icon for overwriting default icon on the dashboard	Icon
`labels`	Labels for check	map[string]string
`metrics`	Metrics to export from	`[]Metrics`
`test`	Evaluate whether a check is healthy	`Expression`
`transform`	Transform data from a check into multiple individual checks	`Expression`
`kubeconfig`	Path to a kubeconfig on disk, or a reference to an existing secret	EnvVar

Check Retries

Field	Description	Scheme
`delay`	Initial delay before the checks are run	Duration
`interval`	Retry the checks, on failure, on this interval	Duration
`timeout`	Timeout for the check	Duration

Wait For

Field	Description	Scheme
`delete`	When set to true, the check waits for the resources to be deleted	bool
`disable`	Disable the default behavior of waiting for resources to be healthy.	Duration
`expr`	CEL expression that determines whether all the resources are in their desired state before running checks on them. It receives a `resources` array of the static and non-static resources. The default behavior is to wait until all the resources are ready `dyn(resources).all(r, k8s.isReady(r))`.	CEL
`interval`	Interval to check if all static & non-static resources are ready. (Default: 5s)	Duration
`timeout`	Timeout to wait for all static & non-static resources to satisfy the expression. (Default: 10m)	Duration

Remote clusters

A single canary-checker instance can connect to any number of remote clusters via custom kubeconfig. Either the kubeconfig itself or the path to the kubeconfig can be provided.

kubeconfig from kubernetes secret

remote-cluster.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: pod-creation-test
spec:
  schedule: "@every 5m"
  kubernetesResource:
    - name: pod creation on aws cluster
      namespace: default
      description: "deploy httpbin"
      kubeconfig:
        valueFrom:
          secretKeyRef:
            name: aws-kubeconfig
            key: kubeconfig
      resources:
        - apiVersion: v1
          kind: Pod
          metadata:
            name: httpbin
            namespace: default
            labels:
              app: httpbin
          spec:
            containers:
              - name: httpbin
                image: "kennethreitz/httpbin:latest"
                ports:
                  - containerPort: 80

Kubeconfig inline

remote-cluster.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: pod-creation-test
spec:
  schedule: "@every 5m"
  kubernetesResource:
    - name: pod creation on aws cluster
      namespace: default
      description: "deploy httpbin"
      kubeconfig:
        value: |
          apiVersion: v1
          clusters:
              - cluster:
                  certificate-authority-data: xxxxx
                  server: https://xxxxx.sk1.eu-west-1.eks.amazonaws.com
                name: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
          contexts:
              - context:
                  cluster: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
                  namespace: mission-control
                  user: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
                name: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
          current-context: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
          kind: Config
          preferences: {}
          users:
              - name: arn:aws:eks:eu-west-1:765618022540:cluster/aws-cluster
                user:
                  exec:
                      ....
      resources:
        - apiVersion: v1
          kind: Pod
          metadata:
            name: httpbin
            namespace: default
            labels:
              app: httpbin
          spec:
            containers:
              - name: httpbin
                image: "kennethreitz/httpbin:latest"
                ports:
                  - containerPort: 80

Kubeconfig from local filesystem

remote-cluster.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: pod-creation-test
spec:
  schedule: "@every 5m"
  kubernetesResource:
    - name: pod creation on aws cluster
      namespace: default
      description: "deploy httpbin"
      kubeconfig:
        value: /root/.kube/aws-kubeconfig
      resources:
        - apiVersion: v1
          kind: Pod
          metadata:
            name: httpbin
            namespace: default
            labels:
              app: httpbin
          spec:
            containers:
              - name: httpbin
                image: "kennethreitz/httpbin:latest"
                ports:
                  - containerPort: 80

Templating

The resources and staticResources fields can be templated using Go Templates. This is helpful in creating resources with random names. Example: you can setup a resource to create a pod with random name on each check run. This way you don't have to wait for the pod to be deleted on every check.

info

Templating the Group, Version, Kind & Namespace however isn't allowed.

Creating a pod with a unique name on every run

pod_exit_code_check.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: pod-exit-code-check
  labels:
    "Expected-Fail": "false"
spec:
  schedule: "@every 5m"
  kubernetesResource:
    - name: "pod exit code"
      description: "Create pod & check its exit code"
      namespace: default
      display:
        expr: |
          "Result of check 'exit-code-check': " + display["exit-code-check"]
      resources:
        - apiVersion: v1
          kind: Pod
          metadata:
            name: "hello-world-{{strings.ToLower (random.Alpha 10)}}"
            namespace: default
          spec:
            restartPolicy: Never
            containers:
              - name: hello-world
                image: hello-world
      waitFor:
        expr: "dyn(resources).all(r, k8s.isHealthy(r))"
        interval: "1s"
        timeout: "20s"
      checkRetries:
        delay: 2s
        timeout: 5m
      checks:
        - kubernetes:
            - name: exit-code-check
              kind: Pod
              namespaceSelector:
                name: default
              resource:
                name: "{{(index .resources 0).Object.metadata.name}}"
              test:
                expr: >
                  size(results) == 1 &&
                  results[0].Object.status.containerStatuses[0].state.terminated.exitCode == 0

Examples

Creating a namespace

namespace_creation.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: namespace-creation
  labels:
    "Expected-Fail": "false"
spec:
  schedule: "@every 5m"
  kubernetesResource:
    - name: "namespace creation"
      namespace: "default"
      description: "create a namespace and pod in it"
      waitFor:
        timeout: 3m
        delete: true
      staticResources:
        - apiVersion: v1
          kind: Namespace
          metadata:
            name: test
      resources:
        - apiVersion: v1
          kind: Pod
          metadata:
            name: httpbin
            namespace: test
            labels:
              app: httpbin
          spec:
            containers:
              - name: httpbin
                image: "kennethreitz/httpbin:latest"
                ports:
                  - containerPort: 80

warning

Since static resources are deleted when the canary is deleted, extra care must be taken when providing their manifests. When this canary is deleted, the test namespace is deleted and consequently all the other resources within it, even those not created by this check.

Crossplane Example

namespace_creation.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: crossplane-kubernetes-resource
spec:
  schedule: "@every 10m"
  kubernetesResource:
    - name: crossplane-kubernetes-resource
      namespace: canaries
      description: "Create an S3 bucket via crossplane and run s3 check on it"
      waitFor: 
        expr: 'dyn(resources).all(r, has(r.Object.status.atProvider) && has(r.Object.status.atProvider.arn))'
        interval: 30s
        timeout: 5m
      resources:
        - apiVersion: s3.aws.crossplane.io/v1beta1
          kind: Bucket
          metadata:
            name: check-bucket
          spec:
            forProvider:
              acl: private
              locationConstraint: us-east-1
            providerConfigRef:
              name: localstack

      checks:
        - s3:
          - name: s3-check
            bucketName: "{{ (index .resources 0).Object.metadata.name }}"
            objectPath: dummy
            region: "{{ (index .resources 0).Object.spec.forProvider.locationConstraint }}"
            url: http://localstack-localstack.localstack.svc.cluster.local:4566
            usePathStyle: true
            accessKey:
              value: test
            secretKey:
              value: test
      checkRetries:
        delay: 60s
        interval: 10s
        timeout: 5m

Testing Helm Deployments in a Virtual Cluster

This example demonstrates how to test a Helm deployment in a virtual cluster.

It creates a vcluster, installs the canary-checker helm chart, and then verifies the deployment is working using an http check.

vcluster-canary-checker.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: kubernetes-resource-helm-chart-install
spec:
  schedule: "@every 1h"
  kubernetesResource:
    - name: canary-checker-helm-in-vcluster
      description: Install a vcluster and deploy canary-checker using HelmChart
      staticResources:
        - apiVersion: v1
          kind: Namespace
          metadata:
            name: canary-checker-vcluster-test
        - apiVersion: source.toolkit.fluxcd.io/v1
          kind: HelmRepository
          metadata:
            name: loft-sh
            namespace: canary-checker-vcluster-test
          spec:
            url: https://charts.loft.sh
            interval: 1h
        - apiVersion: source.toolkit.fluxcd.io/v1
          kind: HelmRepository
          metadata:
            name: flanksource
            namespace: canary-checker-vcluster-test
          spec:
            url: https://flanksource.github.io/charts
            interval: 1h
        - apiVersion: helm.toolkit.fluxcd.io/v2
          kind: HelmRelease
          metadata:
            name: vcluster
            namespace: canary-checker-vcluster-test
          spec:
            chart:
              spec:
                chart: vcluster
                version: "0.16.4"
                sourceRef:
                  kind: HelmRepository
                  name: loft-sh
                  namespace: canary-checker-vcluster-test
            values:
              plugin:
                sync-host-secrets:
                  image: docker.io/flanksource/vcluster-sync-host-secrets:v0.1.6
                  env:
                  - name: DESTINATION_NAMESPACE
                    value: c-test
              sync:
                secrets:
                  all: true
              syncer:
                extraArgs:
                  - --tls-san=vcluster.canary-checker-vcluster-test.svc
                  - --out-kube-config-server=https://vcluster.canary-checker-vcluster-test.svc:443
            install:
              createNamespace: true
            interval: 5m
      resources:
        - apiVersion: helm.toolkit.fluxcd.io/v2
          kind: HelmRelease
          metadata:
            name: canary-checker
            namespace: canary-checker-vcluster-test
          spec:
            chart:
              spec:
                chart: canary-checker
                sourceRef:
                  kind: HelmRepository
                  name: flanksource
                  namespace: canary-checker-vcluster-test
            kubeConfig:
              secretRef:
                name: vc-vcluster
                key: config
            targetNamespace: default
            storageNamespace: default
            values:
              flanksource-ui:
                enabled: false
              serviceAccount:
                rbac:
                  clusterRole: false
            install:
              createNamespace: true
            interval: 5m
      waitFor:
        timeout: 5m
        expr: dyn(resources).all(r, k8s.isReady(r))
        interval: 2s
      checks:
        - http:
            - url: http://canary-checker-x-default-x-vcluster.canary-checker-vcluster-test.svc:8080/health
              name: canary-checker-health-check
              timeout: 5000
              thresholdMillis: 3000
      checkRetries:
        delay: 5s
        timeout: 1m
        interval: 5s

Check Retries​

Wait For​

Remote clusters​

kubeconfig from kubernetes secret​

Kubeconfig inline​

Kubeconfig from local filesystem​

Templating​

Examples​