Skip to main content

Folder

Checks the contents of a folder for size, age, and count. Folder based checks are useful in a number of scenarios:

  • Verifying that backups have been uploaded and are the appropriate size
  • Checking that logs or other temporary files are being cleaned up
  • For batch processes:
    • Checking if files are being processed (and/or produced)
    • Checking the size of queue processing backlog
    • Checking if any error (.err or .log) files have been produced.
folder-check.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
name: folder-check
spec:
schedule: "@every 5m"
folder:
- path: /etc/
name: folder-check-min
description: Checks if there are at least 10 files in the folder
minCount: 10
FieldDescriptionScheme
name*

Name of the check, must be unique within the canary

string

path*

A local folder path or a remote folder smb://, sftp://, s3:// or gcs://

awsConnection

AWS Access credentials

AWS

filter

Filter objects out before checking if they are valid

FolderFilter

gcpConnection

Connection details for GCP

GCP

maxAge

The oldest age a file can be, often used to check for unprocessed files or files that have not been cleaned up

Duration

maxCount

The maximum number of files inside the path, can be used in conjunction with filter.regex to detect error files

integer

maxSize

The maximum file size

Size

minAge

The youngest age a file can be

Duration

minCount

The minimum number of files inside the path

integer

minSize

The minimum file size, can be used to detect backups that did not upload successfully

Size

sftpConnection

Connection details

SFTP

smbConnection

SMB connection details

SMB

description

Description for the check

string

display

Expression to change the formatting of the display

Expression

icon

Icon for overwriting default icon on the dashboard

Icon

labels

Labels for check

map[string]string

metrics

Metrics to export from

[]Metrics

test

Evaluate whether a check is healthy

Expression

transform

Transform data from a check into multiple individual checks

Expression

FolderFilter

FieldDescriptionSchemeRequired
maxAgeMaxAge the latest object should be younger than defined ageDuration
maxSizeMaxSize of the files inside the searchPathSize
minAgeMinAge the latest object should be older than defined ageDuration
minSizeMinSize of the files inside the searchPathSize
regexFilter files based on regular expressionregex

e.g. to verify that database backups are being performed

postgres-backup-check.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
name: folder-check
spec:
schedule: "@every 5m"
folder:
- name: pg-backup checks
path: /data/backups
filter:
regex: "pg-backups-.*.zip"
maxAge: 1d # require a daily backup
minSize: 10mb # the backup should be at least 10mb

Result Variables

The following fields are available in test, display and transform expressions

FieldScheme
Oldestos.FileInfo
Newestos.FileInfo
MinSizeos.FileInfo
MaxSizeos.FileInfo
SupportsTotalSize (Only true for SMB folders)bool
SupportsAvailableSize (Only true for SMB folders)bool
TotalSizeint64
AvailableSizeint64
Files[]os.FileInfo

FolderFilter

FieldDescriptionSchemeRequired
minAgeDuration
maxAgeDuration
minSizeSize
maxSizeSize
regexstring

Connection Types

SFTP

FieldDescriptionScheme
host

SFTP host

string

port

SFTP port

integer

username

EnvVar

password

EnvVar

SFTP Check
exec_artifact.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
name: sftp-pass
spec:
schedule: "@every 5m"
folder:
- path: /tmp
name: sample sftp check
sftpConnection:
host: 192.168.1.5
username:
value: <username>
password:
value: <password>
maxCount: 10

S3

FieldDescriptionScheme
accessKey

EnvVar

secretKey

EnvVar

region

The AWS region

string

endpoint

Custom AWS Endpoint to use

skipTLSVerify

Skip TLS verify when connecting to AWS

boolean

S3 Bucket
exec_artifact.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
name: s3-bucket-pass
namespace: canaries
spec:
schedule: "@every 5m"
folder:
# Check for any backup not older than 7 days and min size 25 bytes
- path: s3://tests-e2e-1
name: mysql backup check
awsConnection:
accessKey:
valueFrom:
secretKeyRef:
name: aws-credentials
key: AWS_ACCESS_KEY_ID
secretKey:
valueFrom:
secretKeyRef:
name: aws-credentials
key: AWS_SECRET_ACCESS_KEY
region: "minio"
endpoint: "http://minio.minio:9000"
usePathStyle: true
skipTLSVerify: true
filter:
regex: "(.*)backup.zip$"
maxAge: 7d
minSize: 25b
# Check for any mysql backup not older than 7 days and min size 25 bytes
- path: s3://tests-e2e-1
name: mysql backup check (7 days and 25 bytes size)
awsConnection:
accessKey:
valueFrom:
secretKeyRef:
name: aws-credentials
key: AWS_ACCESS_KEY_ID
secretKey:
valueFrom:
secretKeyRef:
name: aws-credentials
key: AWS_SECRET_ACCESS_KEY
region: "minio"
endpoint: "http://minio.minio:9000"
usePathStyle: true
skipTLSVerify: true
filter:
regex: "mysql\\/backups\\/(.*)\\/mysql.zip$"
maxAge: 7d
minSize: 25b
# Check for any pg backup not older than 7 days and min size 50 bytes
- path: s3://tests-e2e-1
name: pg backup check (7 days and 50 bytes size)
awsConnection:
accessKey:
valueFrom:
secretKeyRef:
name: aws-credentials
key: AWS_ACCESS_KEY_ID
secretKey:
valueFrom:
secretKeyRef:
name: aws-credentials
key: AWS_SECRET_ACCESS_KEY
region: "minio"
endpoint: "http://minio.minio:9000"
usePathStyle: true
skipTLSVerify: true
filter:
regex: "pg\\/backups\\/(.*)\\/backup.zip$"
maxAge: 7d
minSize: 25b
- name: recursive folders
namespace: default
path: s3://recursive-test/developers/**/*.txt
minCount: 4
display:
expr: results.?files.orValue([]).map(i, i.name).join(", ")
awsConnection:
region: "minio"
endpoint: "http://minio.minio:9000"
usePathStyle: true
skipTLSVerify: true
accessKey:
valueFrom:
secretKeyRef:
name: aws-credentials
key: AWS_ACCESS_KEY_ID
secretKey:
valueFrom:
secretKeyRef:
name: aws-credentials
key: AWS_SECRET_ACCESS_KEY

SMB

FieldDescriptionScheme
host

The SMB host

string

port

integer

domain

Windows Domain

username

EnvVar

password

EnvVar

SMB / CIFS
smb.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
name: smb-pass
spec:
schedule: "@every 5m"
folder:
# Check for any backup not older than 7 days and min size 25 bytes
- path: \\windows-server\sharename\folder
smbConnection:
username:
valueFrom:
secretKeyRef:
name: smb-credentials
key: USERNAME
password:
valueFrom:
secretKeyRef:
name: ssmb-credentials
key: PASSWORD
filter:
regex: "(.*)backup.zip$"
maxAge: 7d
minSize: 25b

GCS

FieldDescriptionScheme
credentials

The credentials to use for authentication

EnvVar

GCS Check
gcs.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
name: recursive-folder-check
spec:
schedule: "@every 5m"
folder:
- path: gcs://folder-check-test/recursive-test
name: recursive folders
namespace: default
minCount: 3
recursive: true
display:
expr: results.?files.orValue([]).map(i, i.name).join(", ")
gcpConnection:
connection: connection://gcs/flanksource-prod