🎁 Get the FREE AI Skills Starter Guide β€” Subscribe β†’
BytesAgainBytesAgain
BytesAgain β€Ί Skills β€Ί volcengine-tos-vectors-skills
πŸ¦€ ClawHub

volcengine-tos-vectors-skills

by @jneless

Manage vector storage and similarity search using TOS Vectors service. Use when working with embeddings, semantic search, RAG systems, recommendation engines, or when the user mentions vector databases, similarity search, or TOS Vectors operations.

Versionv1.0.2
Downloads2,393
Stars⭐ 3
#legal#productivity
TERMINAL
clawhub install volcengine-tos-vectors-skills

πŸ“– About This Skill

name: tos-vectors description: Manage vector storage and similarity search using TOS Vectors service. Use when working with embeddings, semantic search, RAG systems, recommendation engines, or when the user mentions vector databases, similarity search, or TOS Vectors operations.

TOS Vectors Skill

Comprehensive skill for managing vector storage, indexing, and similarity search using the TOS Vectors service - a cloud-based vector database optimized for AI applications.

Quick Start

Initialize Client

import os
import tos
Get credentials from environment

ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
account_id = os.getenv('TOS_ACCOUNT_ID')
Configure endpoint and region

endpoint = 'https://tosvectors-cn-beijing.volces.com'
region = 'cn-beijing'
Create client

client = tos.VectorClient(ak, sk, endpoint, region)

Basic Workflow

# 1. Create vector bucket (like a database)
client.create_vector_bucket('my-vectors')
2. Create vector index (like a table)

client.create_index(
    account_id=account_id,
    vector_bucket_name='my-vectors',
    index_name='embeddings-768d',
    data_type=tos.DataType.DataTypeFloat32,
    dimension=768,
    distance_metric=tos.DistanceMetricType.DistanceMetricCosine
)
3. Insert vectors

vectors = [
    tos.models2.Vector(
        key='doc-1',
        data=tos.models2.VectorData(float32=[0.1] * 768),
        metadata={'title': 'Document 1', 'category': 'tech'}
    )
]
client.put_vectors(
    vector_bucket_name='my-vectors',
    account_id=account_id,
    index_name='embeddings-768d',
    vectors=vectors
)
4. Search similar vectors

query_vector = tos.models2.VectorData(float32=[0.1] * 768)
results = client.query_vectors(
    vector_bucket_name='my-vectors',
    account_id=account_id,
    index_name='embeddings-768d',
    query_vector=query_vector,
    top_k=5,
    return_distance=True,
    return_metadata=True
)

Core Operations

Vector Bucket Management

Create Bucket 

client.create_vector_bucket(bucket_name)

List Buckets 

result = client.list_vector_buckets(max_results=100)
for bucket in result.vector_buckets:
    print(bucket.vector_bucket_name)

Delete Bucket (must be empty) 

client.delete_vector_bucket(bucket_name, account_id)

Vector Index Management

Create Index 

client.create_index(
    account_id=account_id,
    vector_bucket_name=bucket_name,
    index_name='my-index',
    data_type=tos.DataType.DataTypeFloat32,
    dimension=128,
    distance_metric=tos.DistanceMetricType.DistanceMetricCosine
)

List Indexes 

result = client.list_indexes(bucket_name, account_id)
for index in result.indexes:
    print(f"{index.index_name}: {index.dimension}d")

Vector Data Operations

Insert Vectors (batch up to 500) 

vectors = []
for i in range(100):
    vector = tos.models2.Vector(
        key=f'vec-{i}',
        data=tos.models2.VectorData(float32=[...]),
        metadata={'category': 'example'}
    )
    vectors.append(vector)
client.put_vectors(
    vector_bucket_name=bucket_name,
    account_id=account_id,
    index_name=index_name,
    vectors=vectors
)

Query Similar Vectors (KNN search) 

results = client.query_vectors(
    vector_bucket_name=bucket_name,
    account_id=account_id,
    index_name=index_name,
    query_vector=query_vector,
    top_k=10,
    filter={"$and": [{"category": "tech"}]},  # Optional metadata filter
    return_distance=True,
    return_metadata=True
)
for vec in results.vectors:
    print(f"Key: {vec.key}, Distance: {vec.distance}")

Get Vectors by Keys 

result = client.get_vectors(
    vector_bucket_name=bucket_name,
    account_id=account_id,
    index_name=index_name,
    keys=['vec-1', 'vec-2'],
    return_data=True,
    return_metadata=True
)

Delete Vectors 

client.delete_vectors(
    vector_bucket_name=bucket_name,
    account_id=account_id,
    index_name=index_name,
    keys=['vec-1', 'vec-2']
)

Common Use Cases

1. Semantic Search

Build a semantic search system for documents:

# Index documents
for doc in documents:
    embedding = get_embedding(doc.text)  # Your embedding model
    vector = tos.models2.Vector(
        key=doc.id,
        data=tos.models2.VectorData(float32=embedding),
        metadata={'title': doc.title, 'content': doc.text[:500]}
    )
    vectors.append(vector)
client.put_vectors(
    vector_bucket_name=bucket_name,
    account_id=account_id,
    index_name=index_name,
    vectors=vectors
)
Search

query_embedding = get_embedding(user_query)
results = client.query_vectors(
    vector_bucket_name=bucket_name,
    account_id=account_id,
    index_name=index_name,
    query_vector=tos.models2.VectorData(float32=query_embedding),
    top_k=5,
    return_metadata=True
)

2. RAG (Retrieval Augmented Generation)

Retrieve relevant context for LLM prompts:

# Retrieve relevant documents
question_embedding = get_embedding(user_question)
search_results = client.query_vectors(
    vector_bucket_name=bucket_name,
    account_id=account_id,
    index_name='knowledge-base',
    query_vector=tos.models2.VectorData(float32=question_embedding),
    top_k=3,
    return_metadata=True
)
Build context

context = "\n\n".join([
    v.metadata.get('content', '') for v in search_results.vectors
])
Generate answer with LLM

prompt = f"Context:\n{context}\n\nQuestion: {user_question}"

3. Recommendation System

Find similar items based on user preferences:

# Query with metadata filtering
results = client.query_vectors(
    vector_bucket_name=bucket_name,
    account_id=account_id,
    index_name='products',
    query_vector=user_preference_vector,
    top_k=10,
    filter={"$and": [{"category": "electronics"}, {"price_range": "mid"}]},
    return_metadata=True
)

Best Practices

Naming Conventions

  • Bucket names: 3-32 chars, lowercase letters, numbers, hyphens only
  • Index names: 3-63 chars
  • Vector keys: 1-1024 chars, use meaningful identifiers

    • Batch Operations

  • Insert up to 500 vectors per call
  • Delete up to 100 vectors per call
  • Use pagination for listing operations

    • Error Handling

    try:
    result = client.create_vector_bucket(bucket_name)
except tos.exceptions.TosClientError as e:
    print(f'Client error: {e.message}')
except tos.exceptions.TosServerError as e:
    print(f'Server error: {e.code}, Request ID: {e.request_id}')

    Performance Tips

  • Choose appropriate vector dimensions (balance accuracy vs performance)
  • Use metadata filtering to reduce search space
  • Use cosine similarity for normalized vectors
  • Use Euclidean distance for absolute distances

    • Important Limits

  • Vector buckets: Max 100 per account
  • Vector dimensions: 1-4096
  • Batch insert: 1-500 vectors per call
  • Batch get/delete: 1-100 vectors per call
  • Query TopK: 1-30 results

    • Additional Resources

    For detailed API reference, see REFERENCE.md For complete workflows, see WORKFLOWS.md For example scripts, see the scripts/ directory

    πŸ’‘ Examples

    Initialize Client

    import os
import tos
    Get credentials from environment
    
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
account_id = os.getenv('TOS_ACCOUNT_ID')
    Configure endpoint and region
    
endpoint = 'https://tosvectors-cn-beijing.volces.com'
region = 'cn-beijing'
    Create client
    
client = tos.VectorClient(ak, sk, endpoint, region)

    Basic Workflow

    # 1. Create vector bucket (like a database)
client.create_vector_bucket('my-vectors')
    2. Create vector index (like a table)
    
client.create_index(
    account_id=account_id,
    vector_bucket_name='my-vectors',
    index_name='embeddings-768d',
    data_type=tos.DataType.DataTypeFloat32,
    dimension=768,
    distance_metric=tos.DistanceMetricType.DistanceMetricCosine
)
    3. Insert vectors
    
vectors = [
    tos.models2.Vector(
        key='doc-1',
        data=tos.models2.VectorData(float32=[0.1] * 768),
        metadata={'title': 'Document 1', 'category': 'tech'}
    )
]
client.put_vectors(
    vector_bucket_name='my-vectors',
    account_id=account_id,
    index_name='embeddings-768d',
    vectors=vectors
)
    4. Search similar vectors
    
query_vector = tos.models2.VectorData(float32=[0.1] * 768)
results = client.query_vectors(
    vector_bucket_name='my-vectors',
    account_id=account_id,
    index_name='embeddings-768d',
    query_vector=query_vector,
    top_k=5,
    return_distance=True,
    return_metadata=True
)

    πŸ“‹ Tips & Best Practices

    Naming Conventions

  • Bucket names: 3-32 chars, lowercase letters, numbers, hyphens only
  • Index names: 3-63 chars
  • Vector keys: 1-1024 chars, use meaningful identifiers

    • Batch Operations

  • Insert up to 500 vectors per call
  • Delete up to 100 vectors per call
  • Use pagination for listing operations

    • Error Handling

    try:
    result = client.create_vector_bucket(bucket_name)
except tos.exceptions.TosClientError as e:
    print(f'Client error: {e.message}')
except tos.exceptions.TosServerError as e:
    print(f'Server error: {e.code}, Request ID: {e.request_id}')

    Performance Tips

  • Choose appropriate vector dimensions (balance accuracy vs performance)
  • Use metadata filtering to reduce search space
  • Use cosine similarity for normalized vectors
  • Use Euclidean distance for absolute distances