Vald Upsert APIs

Overview

Upsert Service is responsible for updating existing vectors in the vald-agent or inserting new vectors into the vald-agent if the vector does not exist.

service Upsert {

  rpc Upsert(payload.v1.Upsert.Request)
      returns (payload.v1.Object.Location) {}

  rpc StreamUpsert(stream payload.v1.Upsert.Request)
      returns (stream payload.v1.Object.StreamLocation) {}

  rpc MultiUpsert(payload.v1.Upsert.MultiRequest)
      returns (payload.v1.Object.Locations) {}
}

Upsert RPC

Upsert RPC is the method to update the inserted vector to a new single vector or add a new single vector if not inserted before.

Input

  • the scheme of payload.v1.Upsert.Request

    message Upsert {
      message Request {
        Object.Vector vector = 1 [ (validate.rules).repeated .min_items = 2 ];
        Config config = 2;
      }
    
      message Config {
        bool skip_strict_exist_check = 1;
        Filter.Config filters = 2;
        int64 timestamp = 3;
      }
    }
    
    message Object {
        message Vector {
            string id = 1 [ (validate.rules).string.min_len = 1 ];
            repeated float vector = 2 [ (validate.rules).repeated .min_items = 2 ];
        }
    }
    
    • Upsert.Request

      fieldtypelabelrequireddescription
      vectorObject.Vector*The information of vector.
      configConfig*The configuration of the update request.
    • Upsert.Config

      fieldtypelabelrequireddescription
      skip_strict_exist_checkboolCheck whether the same vector is already inserted or not.
      The ID should be unique if the value is true.
      timestampint64The timestamp of the vector inserted.
      If it is N/A, the current time will be used.
      filtersFilter.ConfigConfiguration for filter.
      disable_balanced_updateboolA flag to disable balanced update (split remove -> insert operation) during update operation.
    • Object.Vector

      fieldtypelabelrequireddescription
      idstring*The ID of a vector. ID should consist of 1 or more characters.
      vectorfloatrepeated(Array[float])*The vector data. Its dimension is between 2 and 65,536.

Output

  • the scheme of payload.v1.Object.Location

    message Object {
        message Location {
          string name = 1;
          string uuid = 2;
          repeated string ips = 3;
        }
    }
    
    • Object.Location
      fieldtypelabeldescription
      namestringThe name of vald agent pod where the request vector is updated/inserted.
      uuidstringThe ID of the updated/inserted vector. It is the same as an Object.Vector.
      ipsstringrepeated(Array[string])The IP list of vald-agent pods where the request vector is updated/inserted.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
6ALREADY_EXISTS
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Dimension of the request vector is NOT the same as Vald Agent’s config, the requested vector’s ID is empty, or some request payload is invalid.Check Agent config, request payload, and fix request payload or Agent config.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
ALREADY_EXISTSRequested pair of ID and vector is already insertedChange request payload or nothing to do if update is unnecessary.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

StreamUpsert RPC

StreamUpsert RPC is the method to update multiple existing vectors or add new multiple vectors using the bidirectional streaming RPC.
Using the bidirectional streaming RPC, the upsert request can be communicated in any order between the client and server. Each Upsert request and response are independent. It’s the recommended method to upsert a large number of vectors.

Input

  • the scheme of payload.v1.Upsert.Request stream

    message Upsert {
        message Request {
            Object.Vector vector = 1 [ (validate.rules).repeated .min_items = 2 ];
            Config config = 2;
        }
        message Config {
            bool skip_strict_exist_check = 1;
            Filter.Config filters = 2;
            int64 timestamp = 3;
        }
    }
    
    message Object {
        message Vector {
            string id = 1 [ (validate.rules).string.min_len = 1 ];
            repeated float vector = 2 [ (validate.rules).repeated .min_items = 2 ];
        }
    }
    
    • Upsert.Request

      fieldtypelabelrequireddescription
      vectorObject.Vector*The information of vector.
      configConfig*The configuration of the update request.
    • Upsert.Config

      fieldtypelabelrequireddescription
      skip_strict_exist_checkboolCheck whether the same vector is already inserted or not.
      The ID should be unique if the value is true.
      timestampint64The timestamp of the vector inserted.
      If it is N/A, the current time will be used.
      filtersFilter.ConfigConfiguration for filter.
      disable_balanced_updateboolA flag to disable balanced update (split remove -> insert operation) during update operation.
    • Object.Vector

      fieldtypelabelrequireddescription
      idstring*The ID of a vector. ID should consist of 1 or more characters.
      vectorfloatrepeated(Array[float])*The vector data. Its dimension is between 2 and 65,536.

Output

  • the scheme of payload.v1.Object.StreamLocation

    message Object {
        message StreamLocation {
          oneof payload {
              Location location = 1;
              google.rpc.Status status = 2;
          }
        }
    
        message Location {
          string name = 1;
          string uuid = 2;
          repeated string ips = 3;
        }
    }
    
    • Object.StreamLocation

      fieldtypelabeldescription
      locationObject.LocationThe information of Object.Location data.
      statusgoogle.rpc.StatusThe status of Google RPC.
    • Object.Location

      fieldtypelabeldescription
      namestringThe name of vald agent pod where the request vector is updated/inserted.
      uuidstringThe ID of the updated/inserted vector. It is the same as an Object.Vector.
      ipsstringrepeated(Array[string])The IP list of vald-agent pods where the request vector is updated/inserted.
    • google.rpc.Status

      fieldtypelabeldescription
      codeint32Status code (code list is next section)
      messagestringError message
      detailsgoogle.protobuf.Anyrepeated(Array[any])The details error message list

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
6ALREADY_EXISTS
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Dimension of the request vector is NOT the same as Vald Agent’s config, the requested vector’s ID is empty, or some request payload is invalid.Check Agent config, request payload, and fix request payload or Agent config.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
ALREADY_EXISTSRequested pair of ID and vector is already insertedChange request payload or nothing to do if update is unnecessary.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

MultiUpsert RPC

MultiUpsert is the method to update existing multiple vectors and add new multiple vectors in 1 request.

gRPC has a message size limitation.
Please be careful that the size of the request exceeds the limit.

Input

  • the scheme of payload.v1.Upsert.MultiRequest

    message Upsert {
        message MultiRequest { repeated Request requests = 1; }
    
        message Request {
            Object.Vector vector = 1 [ (validate.rules).repeated .min_items = 2 ];
            Config config = 2;
        }
    
        message Config {
            bool skip_strict_exist_check = 1;
            Filter.Config filters = 2;
            int64 timestamp = 3;
        }
    }
    
    message Object {
        message Vector {
            string id = 1 [ (validate.rules).string.min_len = 1 ];
            repeated float vector = 2 [ (validate.rules).repeated .min_items = 2 ];
        }
    }
    
    • Upsert.MultiRequest

      fieldtypelabelrequireddescription
      requestsUpsert.Requestrepeated(Array[Insert.Request])*The request list.
    • Upsert.Request

      fieldtypelabelrequireddescription
      vectorObject.Vector*The information of vector.
      configConfig*The configuration of the update request.
    • Upsert.Config

      fieldtypelabelrequireddescription
      skip_strict_exist_checkboolCheck whether the same vector is already inserted or not.
      The ID should be unique if the value is true.
      timestampint64The timestamp of the vector inserted.
      If it is N/A, the current time will be used.
      filtersFilter.ConfigConfiguration for filter.
      disable_balanced_updateboolA flag to disable balanced update (split remove -> insert operation) during update operation.
    • Object.Vector

      fieldtypelabelrequireddescription
      idstring*The ID of a vector. ID should consist of 1 or more characters.
      vectorfloatrepeated(Array[float])*The vector data. Its dimension is between 2 and 65,536.

Output

  • the scheme of payload.v1.Object.Locations.

    message Object {
        message Locations { repeated Location locations = 1; }
    
        message Location {
          string name = 1;
          string uuid = 2;
          repeated string ips = 3;
        }
    }
    
    • Object.Locations

      fieldtypelabeldescription
      locationObject.Locationrepeated(Array[Object.Location])The list of Object.Location.
    • Object.Location

      fieldtypelabeldescription
      namestringThe name of vald agent pod where the request vector is updated/inserted.
      uuidstringThe ID of the updated/inserted vector. It is the same as an Object.Vector.
      ipsstringrepeated(Array[string])The IP list of vald-agent pods where the request vector is updated/inserted.

Status Code

codename
0OK
1CANCELLED
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
6ALREADY_EXISTS
10ABORTED
13INTERNAL

Please refer to Response Status Code for more details.

Troubleshooting

The request process may not be completed when the response code is NOT 0 (OK).

Here are some common reasons and how to resolve each error.

namecommon reasonhow to resolve
CANCELLEDExecuted cancel() of rpc from client/server-side or network problems between client and server.Check the code, especially around timeout and connection management, and fix if needed.
INVALID_ARGUMENTThe Dimension of the request vector is NOT the same as Vald Agent’s config, the requested vector’s ID is empty, or some request payload is invalid.Check Agent config, request payload, and fix request payload or Agent config.
DEADLINE_EXCEEDEDThe RPC timeout setting is too short on the client/server side.Check the gRPC timeout setting on both the client and server sides and fix it if needed.
ALREADY_EXISTSRequested pair of ID and vector is already insertedChange request payload or nothing to do if update is unnecessary.
INTERNALTarget Vald cluster or network route has some critical error.Check target Vald cluster first and check network route including ingress as second.

See also