Deploy Vald Agent on Docker
Vald is designed and implemented based on Cloud-Native architecture. However, there may be cases that want to use only Vald Agent without Kubernetes.
This article will show you how to deploy and run the Vald Agent on Docker. Fashion-mnist is used as an example dataset, same as Get Started.
Requirements
- docker: v19.0 ~
- go: v1.15 ~
- libhdf5 (only required for this tutorial.)
Hdf5 is required for this tutorial. If hdf5 is not installed, please install hdf5.
[Optional] Install hdf5
# yum
yum install -y hdf5-devel
# apt
apt-get install libhdf5-serial-dev
# homebrew
brew install hdf5
Deploy
This chapter will show you how to deploy Vald Agent on docker.
This chapter will use NGT for the core engine of Vald Agent.
Clone the vdaas/vald repository
git clone https://github.com/vdaas/vald.git
Create directory for setting deploy vald-agent-ngt
cd vald mkdir -p tutorial && cd tutorial
Create
config.yaml
The configuration of Vald agent for docker is set using
config.yaml
You can also check the sample configuration.cat << EOF > config.yaml --- version: v0.0.0 time_zone: JST logging: logger: glg level: debug format: raw server_config: servers: - name: agent-grpc host: 0.0.0.0 port: 8081 mode: GRPC probe_wait_time: "3s" http: shutdown_duration: "5s" handler_timeout: "" idle_timeout: "" read_header_timeout: "" read_timeout: "" write_timeout: "" startup_strategy: - agent-grpc shutdown_strategy: - agent-grpc full_shutdown_duration: 600s tls: enabled: false # cert: /path/to/cert # key: /path/to/key # ca: /path/to/ca ngt: # path to index data index_path: "/etc/server/backup" # vector dimension dimension: 784 # bulk insert chunk size bulk_insert_chunk_size: 10 # distance_type, which should be "l1", "l2" "angle", "hamming", "cosine", "normalizedangle", "normalizedcosine" or "jaccard" distance_type: l2 # object_type, which should be "float" or "uint8" object_type: float # creation edge size creation_edge_size: 20 # search edge size search_edge_size: 10 # The limit duration of automatic indexing # auto_index_duration_limit should be 30m-6h for production use. Below setting is a just example auto_index_duration_limit: 1m # Check duration of automatic indexing. # auto_index_check_duration be 10m-1h for production use. Below setting is a just example auto_index_check_duration: 10s # The number of cache to trigger automatic indexing auto_index_length: 100 # The limit duration of auto saving indexing # auto_save_index_duration should be 30m-60m for production use. The below setting is a just example. auto_save_index_duration: 90s # The maximum limit duration for an initial delay # initial_delay_max_duration should be 3m-5m for production use. The below setting is a just example. initial_delay_max_duration: 60s EOF
Create backup directory
To avoid removing the indexing data due to any trouble after finishing indexing, we should prepare the path for auto backup.
mkdir -p backup
Deploy Vald Agent on Docker
To deploy Vald agent on docker with
config.yaml
, you can run below command. Note:- Please check whether there are
config.yaml
andbackup
directory in your current directory.
docker run -v $(pwd)/:/etc/server -p 8081:8081 --rm -it vdaas/vald-agent-ngt
- Please check whether there are
Verify
If the deployment success, you can confirm the output will be similar to below.
2020-07-01 03:02:41 [INFO]: maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined 2020-07-01 03:02:41 [INFO]: service agent ngt v0.0.0 starting... 2020-07-01 03:02:41 [INFO]: executing daemon pre-start function 2020-07-01 03:02:41 [INFO]: executing daemon start function 2020-07-01 03:02:41 [INFO]: server agent-grpc executing preStartFunc 2020-07-01 12:02:41 [INFO]: gRPC server agent-grpc starting on 0.0.0.0:8081
Run using example code
Download dataset
In this tutorial. we use fashion-mnist as a dataset for indexing and search query.
# move to working directory cd example/client/agent # download fashion-mnist testing dataset wget http://ann-benchmarks.com/fashion-mnist-784-euclidean.hdf5
Running example
Vald provides multiple language client libraries such as Go, Java, Node.js, Python, and so on.
In this example, the fashion-mnist dataset will insert into the Vald and search using vald-client-go.We use
example/client/agent/main.go
to run the example. The example code is the same as running an example only Vald agent on Kubernetes. If you want to learn the detail of running an example, please refer to the tutorial of standalone Vald Agent on kubernetes.# run example go run main.go
Note:
- We recommend you to run
CreateIndex()
afterInsert()
without waiting auto indexing. - When finish indexing completely, the backup files (metadata.json and ngt-meta.kvsdb) can be confirmed in your mount directory.
- We recommend you to run
Clean Up
Stop the Vald Agent docker container via
Ctrl+C
.