Continuous Benchmark Tool

What is the Continuous Benchmark Tool?

Continuous Benchmark Tool allows you to get benchmark of Vald cluster in 24/7.

Assumed use case is:

• Verification with workload close to the production environment
• Verification before service installation when Vald version up

Architecture

Continuous Benchmark Tool has following 2 components:

• Benchmark Operator: Manages benchmark jobs
• Benchmark Job: Executes CRUDs request to the target Vald cluster

Benchmark component and its feature

Benchmark Operator

• Manages benchmark jobs according to applied manifest.
• Apply method:
  • Scenario method: one manifest with multiple benchmark jobs
  • Job method: one manifest with one benchmark job

Benchmark Job

• Executes CRUD request to the target Vald cluster based on defined config.
• Execute steps are:
  1. Load dataset (valid only for HDF5 format )
  2. Execute request with load dataset

Benchmark CRD

Benchmark workload can be set by applying the Kubernetes Custom Resources(CRDs), ValdBenchmarkScenarioResource or ValdBenchmarkJobResource. Benchmark Operator manages benchmark job according to the applied manifest.

ValdBenchmarkJob

ValdBenchmarkJob is used for executing single benchmark job.

And, Benchmark Operator also applies it to the Kubernetes cluster based on ValdBenchmarkScenarioResource.

main properties

ValdBenchmarkScenario is used for executing single or multiple benchmark job.

Benchmark Operator decomposes manifest and creates benchmark resources one by one. The target and dataset property are the global config for scenario, they can be overwritten when each job has own config.

main properties

Deploy Benchmark Operator

Continuous benchmark operator can be applied with Helm same as Vald cluster.

It requires ValdBenchmarkOperatorRelease for deploying vald-benchmark-operator.

It is not must to apply, so please edit and apply as necessary.

# @schema {"name": "name", "type": "string"}
# name -- name of the deployment
name: vald-benchmark-operator
# @schema {"name": "time_zone", "type": "string"}
# time_zone -- time_zone
time_zone: ""
# @schema {"name": "image", "type": "object"}
image:
  # @schema {"name": "image.repository", "type": "string"}
  # image.repository -- image repository
  repository: vdaas/vald-benchmark-operator
  # @schema {"name": "image.tag", "type": "string"}
  # image.tag -- image tag
  tag: v1.7.5
  # @schema {"name": "image.pullPolicy", "type": "string", "enum": ["Always", "Never", "IfNotPresent"]}
  # image.pullPolicy -- image pull policy
  pullPolicy: Always
# @schema {"name": "job_image", "type": "object"}
job_image:
  # @schema {"name": "job_image.repository", "type": "string"}
  # image.repository -- job image repository
  repository: vdaas/vald-benchmark-job
  # @schema {"name": "job_image.tag", "type": "string"}
  # image.tag -- image tag for job docker image
  tag: v1.7.5
  # @schema {"name": "job_image.pullPolicy", "type": "string", "enum": ["Always", "Never", "IfNotPresent"]}
  # image.pullPolicy -- image pull policy
  pullPolicy: Always
# @schema {"name": "resources", "type": "object"}
# resources -- kubernetes resources of pod
resources:
  # @schema {"name": "resources.limits", "type": "object"}
  limits:
    cpu: 300m
    memory: 300Mi
  # @schema {"name": "resources.requests", "type": "object"}
  requests:
    cpu: 200m
    memory: 200Mi
# @schema {"name": "logging", "type": "object"}
logging:
  # @schema {"name": "logging.logger", "type": "string", "enum": ["glg", "zap"]}
  # logging.logger -- logger name.
  logger: glg
  # @schema {"name": "logging.level", "type": "string", "enum": ["debug", "info", "warn", "error", "fatal"]}
  # logging.level -- logging level.
  level: debug
  # @schema {"name": "logging.format", "type": "string", "enum": ["raw", "json"]}
  # logging.format -- logging format.
  format: raw

For more details of the configuration of vald-benchmark-operator-release, please refer to here

1. Add Vald repo into the helm repo

   helm repo add vald https://vdaas.vald.org
   

2. Deploy vald-benchmark-operator-release

   helm install vald-benchmark-operator-release vald/vald-benchmark-operator
   

3. Apply vbor.yaml (optional)

   kubectl apply -f vbor.yaml
   

Running Continuous Benchmarks

After deploy the benchmark operator, you can execute continuous benchmark by applying ValdBenchmarkScenarioRelease or ValdBenchmarkJobRelease.

Please configure designed benchmark and apply by kubectl command.

The sample manifests are here.

See also