Vector Stores

LangChain ArangoDB provides powerful vector store implementations that allow you to store, index, and retrieve embeddings using ArangoDB’s native vector search capabilities.

Overview

The ArangoVector class is the main vector store implementation that integrates with LangChain’s embedding interfaces and provides:

Efficient vector similarity search with cosine and Euclidean distance metrics
Approximate and exact nearest neighbor search
Maximal marginal relevance (MMR) search for diverse results
Hybrid search combining vector and keyword-based retrieval (RRF)
Batch operations for adding and managing documents
Configurable vector indexing with customizable parameters

Quick Start

from arango import ArangoClient
from langchain_openai import OpenAIEmbeddings
from langchain_arangodb.vectorstores import ArangoVector

# Connect to ArangoDB
client = ArangoClient("http://localhost:8529")
db = client.db("langchain", username="root", password="openSesame")

# Initialize embeddings
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Create vector store from texts
texts = [
    "ArangoDB is a multi-model database",
    "LangChain provides tools for AI applications",
    "Vector search enables semantic similarity"
]

vectorstore = ArangoVector.from_texts(
    texts=texts,
    embedding=embeddings,
    database=db,
    collection_name="my_documents"
)

# Search for similar documents
results = vectorstore.similarity_search("database technology", k=2)
for doc in results:
    print(doc.page_content)

# Stream results for large queries (memory efficient)
cursor = vectorstore.similarity_search("query", k=10000, stream=True)
for doc in cursor:
    print(doc.page_content)

Creating from Existing Collections

You can create a vector store from an existing ArangoDB collection by embedding specific text properties:

from langchain_arangodb.vectorstores import ArangoVector

# Create vector store from existing collection
vectorstore = ArangoVector.from_existing_collection(
    collection_name="existing_docs",
    text_properties_to_embed=["title", "description", "content"],
    embedding=embeddings,
    database=db,
    embedding_field="text_embedding",
    text_field="combined_text",
    batch_size=1000,
    insert_text=True,  # Store concatenated text for hybrid search
    skip_existing_embeddings=False,  # Re-embed all documents
    search_type=SearchType.HYBRID
)

Key Parameters:

text_properties_to_embed: List of document properties to concatenate and embed
batch_size: Number of documents to process at once (default: 1000)
insert_text: Whether to store concatenated text (required for hybrid search)
skip_existing_embeddings: Skip documents that already have embeddings
aql_return_text_query: Custom AQL query for text extraction (optional)

Configuration

Constructor Parameters

class ArangoVector(embedding, embedding_dimension, database, **kwargs)

Parameters:

embedding – Any embedding function implementing LangChain’s Embeddings interface
embedding_dimension – The dimension of the embedding vectors (must match your embedding model)
database – ArangoDB database instance from python-arango
collection_name – Name of the collection to store documents (default: “documents”)
search_type – Type of search - supports “vector” and “hybrid” search modes
embedding_field – Field name for storing embedding vectors (default: “embedding”)
text_field – Field name for storing text content (default: “text”)
vector_index_name – Name of the vector index (default: “vector_index”)
distance_strategy – Distance metric for similarity calculation (default: “COSINE”)
num_centroids – Number of centroids for vector index clustering (default: 1)
relevance_score_fn – Custom function to normalize relevance scores (optional)
keyword_index_name – Name of the keyword search index (default: “keyword_index”)
keyword_analyzer – Text analyzer for keyword search (default: “text_en”)
rrf_constant – Constant for Reciprocal Rank Fusion in hybrid search (default: 60)
rrf_search_limit – Maximum results for RRF scoring (default: 100)

Distance Strategies

The vector store supports multiple distance metrics:

COSINE: Cosine similarity (default) - good for normalized vectors
EUCLIDEAN_DISTANCE: L2 distance - good for absolute distance measurements
MAX_INNER_PRODUCT: Maximum inner product similarity
DOT_PRODUCT: Dot product similarity
JACCARD: Jaccard similarity coefficient

Note: Currently only COSINE and EUCLIDEAN_DISTANCE are fully supported in the vector search implementation.

from langchain_arangodb.vectorstores.utils import DistanceStrategy

# Using Euclidean distance
vectorstore = ArangoVector(
    embedding=embeddings,
    embedding_dimension=1536,
    database=db,
    distance_strategy=DistanceStrategy.EUCLIDEAN_DISTANCE
)

Keyword Analyzers

For hybrid search, ArangoDB supports multiple text analyzers for different languages:

Supported Analyzers:

text_en: English (default)
text_de: German
text_es: Spanish
text_fi: Finnish
text_fr: French
text_it: Italian
text_nl: Dutch
text_no: Norwegian
text_pt: Portuguese
text_ru: Russian
text_sv: Swedish
text_zh: Chinese

# Using German analyzer for hybrid search
vectorstore = ArangoVector(
    embedding=embeddings,
    embedding_dimension=1536,
    database=db,
    search_type=SearchType.HYBRID,
    keyword_analyzer="text_de"
)

Search Methods

Basic Similarity Search

# Simple similarity search
results = vectorstore.similarity_search("your query", k=5)

# Search with additional fields returned
results = vectorstore.similarity_search(
    "your query",
    k=5,
    return_fields={"metadata_field", "custom_field"}
)

# Search with custom embedding
custom_embedding = embeddings.embed_query("your query")
results = vectorstore.similarity_search_by_vector(custom_embedding, k=5)

Search with Scores

# Get similarity scores with results
docs_and_scores = vectorstore.similarity_search_with_score("your query", k=5)

for doc, score in docs_and_scores:
    print(f"Score: {score:.3f} - Content: {doc.page_content[:100]}...")

Maximal Marginal Relevance (MMR)

MMR search helps ensure diverse results by balancing relevance and diversity:

# MMR search for diverse results
diverse_results = vectorstore.max_marginal_relevance_search(
    query="your query",
    k=5,                    # Number of final results
    fetch_k=20,            # Number of initial candidates to fetch
    lambda_mult=0.5        # Balance between relevance (0) and diversity (1)
)

Approximate vs Exact Search

# Approximate search (faster, requires ArangoDB >= 3.12.4)
results = vectorstore.similarity_search("query", use_approx=True)

# Exact search (slower but precise)
results = vectorstore.similarity_search("query", use_approx=False)

Hybrid Search

Hybrid search combines vector similarity with traditional keyword search using Reciprocal Rank Fusion (RRF), providing more comprehensive and accurate results:

from langchain_arangodb.vectorstores import ArangoVector, SearchType

# Create vector store with hybrid search enabled
vectorstore = ArangoVector.from_texts(
    texts=["Machine learning algorithms", "AI-powered applications"],
    embedding=embeddings,
    database=db,
    search_type=SearchType.HYBRID,
    insert_text=True,  # Required for hybrid search
)

# Create both vector and keyword indexes
vectorstore.create_vector_index()
vectorstore.create_keyword_index()

# Perform hybrid search
results = vectorstore.similarity_search_with_score(
    query="AI technology",
    k=3,
    search_type=SearchType.HYBRID,
    vector_weight=1.0,      # Weight for vector similarity
    keyword_weight=1.0,     # Weight for keyword matching
)

Hybrid Search Parameters:

search_type=SearchType.HYBRID: Enables hybrid search mode
vector_weight: Weight for vector similarity scores (default: 1.0)
keyword_weight: Weight for keyword search scores (default: 1.0)
rrf_search_limit: Number of top results to consider for RRF fusion (default: 50)
keyword_search_clause: Custom AQL search clause for keyword matching (optional)

Custom Keyword Search:

# Custom keyword search with metadata filtering
custom_keyword_clause = f"""
    SEARCH ANALYZER(
        doc.{vectorstore.text_field} IN TOKENS(@query, @analyzer),
        @analyzer
    ) AND doc.category == "technology"
"""

results = vectorstore.similarity_search_with_score(
    query="machine learning",
    k=5,
    search_type=SearchType.HYBRID,
    keyword_search_clause=custom_keyword_clause,
)

Keyword Index Management:

# Create keyword index for hybrid search
vectorstore.create_keyword_index()

# Check if keyword index exists
keyword_index = vectorstore.retrieve_keyword_index()
if keyword_index:
    print(f"Keyword index: {keyword_index['name']}")

# Delete keyword index
vectorstore.delete_keyword_index()

Weight Balancing Examples:

# Favor vector similarity (semantic search)
semantic_results = vectorstore.similarity_search_with_score(
    query="artificial intelligence",
    k=3,
    search_type=SearchType.HYBRID,
    vector_weight=10.0,
    keyword_weight=1.0,
)

# Favor keyword matching (traditional search)
keyword_results = vectorstore.similarity_search_with_score(
    query="machine learning algorithms",
    k=3,
    search_type=SearchType.HYBRID,
    vector_weight=1.0,
    keyword_weight=10.0,
)

# Balanced hybrid approach
balanced_results = vectorstore.similarity_search_with_score(
    query="AI applications",
    k=3,
    search_type=SearchType.HYBRID,
    vector_weight=1.0,
    keyword_weight=1.0,
)

Entity Resolution

The ArangoDB vectorstore provides entity resolution capabilities through the find_entity_clusters method. It compares documents within the collection to each other and returns entities grouped with their most similar documents. Each entity is returned with a list of the top k most similar entities based on the chosen similarity function.

# Default entity clustering
clusters = vectorstore.find_entity_clusters(
    threshold=0.8,      # Minimum similarity score
    k=4,               # Number of similar documents per entity
    use_approx=True   # Use approximate search for faster results
)

Entity ResolutionParameters:

threshold: Minimum similarity score for documents to be considered similar (default: 0.8)
k: Number of similar documents to return for each entity (default: 4)
distance_strategy: Distance metric to use for clustering (default: DistanceStrategy.COSINE)
use_approx: Whether to use approximate nearest neighbor or exact search (default: True)
use_subset_relations: Whether to analyze subset relationships between clusters (default: False) (optional)
merge_similar_entities: Whether to merge entities based on subset relationships (default: False) (optional)

Subset Relationship Analysis

Enable use_subset_relations = True highlights the subset and superset relationships between entities:

# Analyze subset relationships
result = vectorstore.find_entity_clusters(
    threshold=0.7,
    k=5,
    use_subset_relations=True
)

Entity Merging

Enable merge_similar_entities = True to merge overlapping entity clusters to create consolidated, non-overlapping groups:

Note: In-order to merge entities, you need to enable use_subset_relations = True, by default use_subset_relations = False.

# Merge similar entities based on subset relationships
merged_result = vectorstore.find_entity_clusters(
    threshold=0.75,
    k=6,
    use_subset_relations=True,
    merge_similar_entities=True
)

Distance Strategy Support

Entity clustering works with different distance metrics and automatically selects the appropriate AQL functions:

from langchain_arangodb.vectorstores.utils import DistanceStrategy

# Cosine similarity (default) - higher scores are better, sorted DESC
vectorstore_cosine = ArangoVector(
    embedding=embeddings,
    embedding_dimension=embedding_dimension,
    database=db,
    distance_strategy=DistanceStrategy.COSINE
)

# Euclidean distance - lower distances are better, sorted ASC
vectorstore_euclidean = ArangoVector(
    embedding=embeddings,
    embedding_dimension=embedding_dimension,
    database=db,
    distance_strategy=DistanceStrategy.EUCLIDEAN_DISTANCE
)

# Jaccard distance - lower distances are better, sorted DESC
vectorstore_jaccard = ArangoVector(
    embedding=embeddings,
    embedding_dimension=embedding_dimension,
    database=db,
    distance_strategy=DistanceStrategy.JACCARD
)

NOTE: for JACCARD, use_approx is automatically set to False

Approximate Search (requires ArangoDB >= 3.12.4)

Automatically uses APPROX_NEAR_COSINE or APPROX_NEAR_L2 functions and creates vector index:

# Uses APPROX_NEAR_COSINE function, automatically creates vector index
cosine_clusters = vectorstore_cosine.find_entity_clusters(
    threshold=0.8,
    use_approx=True  # Fast approximate search
)

# Uses APPROX_NEAR_L2 function, automatically creates vector index
euclidean_clusters = vectorstore_euclidean.find_entity_clusters(
    threshold=0.5,
    use_approx=True  # Fast approximate search
)

Exact Search (works with all ArangoDB versions)

Uses COSINE_SIMILARITY , L2_DISTANCE , JACCARD functions:

# Uses COSINE_SIMILARITY function, no vector index required
cosine_exact = vectorstore_cosine.find_entity_clusters(
    threshold=0.8,
    use_approx=False  # Precise
)

# Uses L2_DISTANCE function, no vector index required
euclidean_exact = vectorstore_euclidean.find_entity_clusters(
    threshold=0.5,
    use_approx=False  # Precise
)

# Uses JACCARD function, no vector index required
jaccard_exact = vectorstore_jaccard.find_entity_clusters(
    threshold=0.3,
    use_approx=False
)

Threshold Selection

Choose thresholds based on your distance strategy:

Cosine similarity: Higher values (0.8-0.95)
Euclidean distance: Lower values (0.3-0.5)
Jaccard distance: Lower values (0.3-0.5)

Return Formats

The method returns different formats based on the parameters used:

Basic clustering (use_subset_relations=False):

# Function call
clusters = vectorstore.find_entity_clusters(
    threshold=0.8,
    k=4,
    use_subset_relations=False
)

# Output
[
    {'entity': 'jon_snow', 'similar': ['jon', 'j_snow','jonsnow']},
    {'entity': 'daenerys_targaryen', 'similar': ['daenerys_t', 'daene']},
    {'entity': 'tyrion_lannister', 'similar': ['tyrion','tyrion_L']}
]

With subset analysis (use_subset_relations=True, merge_similar_entities=False):

# Function call
result = vectorstore.find_entity_clusters(
    threshold=0.8,
    k=4,
    use_subset_relations=True,
    merge_similar_entities=False
)

# Output - shows which groups are subsets of others
{
    'similar_entities': [
        {'entity': 'jon_snow', 'similar': ['jon', 'j_snow','jonsnow']},
        {'entity': 'jon_s', 'similar': ['jon']},
        {'entity': 'daenerys_targaryen', 'similar': ['daenerys_t', 'daene']},
        {'entity': 'daentargaryen', 'similar': ['daene']},
        {'entity': 'tyrion_lannister', 'similar': ['tyrion','tyrion_L']}
    ],
    'subset_relationships': [
        {'subsetGroup': 'jon_s', 'supersetGroup': 'jon_snow'},
        {'subsetGroup': 'daentargaryen', 'supersetGroup': 'daenerys_targaryen'}
    ]
}

With merging (use_subset_relations=True, merge_similar_entities=True):

# Function call
merged_result = vectorstore.find_entity_clusters(
    threshold=0.8,
    k=4,
    use_subset_relations=True,
    merge_similar_entities=True
)

# Output - consolidates overlapping groups into final clusters
{
    'similar_entities': [
        {'entity': 'jon_snow', 'similar': ['jon', 'j_snow','jonsnow']},
        {'entity': 'jon_s', 'similar': ['jon']},
        {'entity': 'daenerys_targaryen', 'similar': ['daenerys_t', 'daene']},
        {'entity': 'daentargaryen', 'similar': ['daene']},
        {'entity': 'tyrion_lannister', 'similar': ['tyrion','tyrion_L']}
    ],
    'subset_relationships': [
        {'subsetGroup': 'jon_s', 'supersetGroup': 'jon_snow'},
        {'subsetGroup': 'daentargaryen', 'supersetGroup': 'daenerys_targaryen'}
    ]
    'merged_entities': [
        {'entity': 'jon_snow', 'merged_entities': ['jon', 'j_snow', 'jonsnow','jon_s',]},
        {'entity': 'daenerys_targaryen', 'merged_entities': ['daene', 'daenerys_t', 'daentargaryen']}
    ]
}

Document Management

Adding Documents

# Add texts with metadata
texts = ["Document 1", "Document 2"]
metadatas = [{"category": "tech"}, {"category": "science"}]
ids = vectorstore.add_texts(texts, metadatas=metadatas)

# Add pre-computed embeddings
embeddings_list = [embedding.embed_query(text) for text in texts]
ids = vectorstore.add_embeddings(
    texts=texts,
    embeddings=embeddings_list,
    metadatas=metadatas,
    ids=["custom_id_1", "custom_id_2"],  # Optional custom IDs
    batch_size=1000,                     # Custom batch size
    use_async_db=True                    # Use async operations
)

# IDs are automatically generated using farmhash if not provided
# This ensures consistent, deterministic IDs based on content
auto_ids = vectorstore.add_texts(["New document"])
print(f"Auto-generated ID: {auto_ids[0]}")

Retrieving Documents

# Get documents by IDs
documents = vectorstore.get_by_ids(["doc_id_1", "doc_id_2"])

Deleting Documents

# Delete specific documents
vectorstore.delete(ids=["doc_id_1", "doc_id_2"])

# Delete all documents in collection
vectorstore.delete()

Advanced Configuration

Vector Index Management

# Check if vector index exists
index_info = vectorstore.retrieve_vector_index()
if index_info:
    print(f"Index exists: {index_info['name']}")

# Create vector index manually
vectorstore.create_vector_index()

# Delete vector index
vectorstore.delete_vector_index()

Batch Operations

# Large batch insertion with custom batch size
large_texts = ["text"] * 10000
vectorstore.add_texts(
    texts=large_texts,
    batch_size=1000,      # Process in batches of 1000
    use_async_db=True     # Use async operations for better performance
)

Custom Collection Setup

# Initialize with specific collection settings
vectorstore = ArangoVector(
    embedding=embeddings,
    embedding_dimension=1536,
    database=db,
    collection_name="custom_vectors",
    vector_index_name="my_vector_index",
    num_centroids=10,     # More centroids for larger datasets
    search_type=SearchType.VECTOR,  # or SearchType.HYBRID
)

# Create from texts with index management
vectorstore = ArangoVector.from_texts(
    texts=["Document content"],
    embedding=embeddings,
    database=db,
    collection_name="my_collection",
    overwrite_index=True,    # Recreate existing indexes
    embedding_field="custom_embedding",
    text_field="custom_text",
    ids=["custom_id_1"],     # Custom document IDs
    insert_text=True,        # Store text content (required for hybrid search)
)

Custom Relevance Scoring

You can provide custom relevance score normalization functions:

def custom_relevance_function(score: float) -> float:
    """Custom normalization that inverts and scales scores."""
    return 1.0 / (1.0 + score)

vectorstore = ArangoVector(
    embedding=embeddings,
    embedding_dimension=1536,
    database=db,
    relevance_score_fn=custom_relevance_function
)

# Get relevance scores with custom normalization
docs_with_scores = vectorstore.similarity_search_with_score("query", k=3)
for doc, score in docs_with_scores:
    print(f"Custom score: {score}")

Performance Tips

Choose the right distance strategy: Use COSINE for normalized embeddings, EUCLIDEAN for raw distances
Use approximate search: Enable use_approx=True for large datasets (requires ArangoDB >= 3.12.4)
Optimize batch size: Use larger batch sizes (500-1000) for bulk operations
Configure centroids: Increase num_centroids for larger collections (rule of thumb: sqrt(num_documents))
Use async operations: Enable use_async_db=True for non-blocking operations
Hybrid search optimization: For hybrid search, ensure both vector and keyword indexes are created
Custom field names: Use descriptive field names for embedding and text fields to avoid conflicts
Memory management: Use import_bulk operations with appropriate batch sizes for large datasets

Example: Complete Workflow

from arango import ArangoClient
from langchain_openai import OpenAIEmbeddings
from langchain_arangodb.vectorstores import ArangoVector
from langchain_arangodb.vectorstores.utils import DistanceStrategy

# Setup
client = ArangoClient("http://localhost:8529")
db = client.db("vectorstore_demo", username="root", password="openSesame")
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Create vector store with hybrid search support
vectorstore = ArangoVector(
    embedding=embeddings,
    embedding_dimension=1536,
    database=db,
    collection_name="knowledge_base",
    search_type=SearchType.HYBRID,
    distance_strategy=DistanceStrategy.COSINE,
    num_centroids=5,
    insert_text=True  # Required for hybrid search
)

# Add documents with metadata
documents = [
    "Python is a programming language",
    "Machine learning uses algorithms",
    "Databases store structured data",
    "APIs enable system integration"
]

metadatas = [
    {"topic": "programming", "difficulty": "beginner"},
    {"topic": "ai", "difficulty": "intermediate"},
    {"topic": "database", "difficulty": "beginner"},
    {"topic": "integration", "difficulty": "intermediate"}
]

# Add to vector store
doc_ids = vectorstore.add_texts(documents, metadatas=metadatas)
print(f"Added {len(doc_ids)} documents")

# Create indexes for hybrid search
vectorstore.create_vector_index()
vectorstore.create_keyword_index()

# Perform searches
print("\n--- Vector Search ---")
vector_results = vectorstore.similarity_search(
    "programming languages",
    k=2,
    search_type=SearchType.VECTOR
)
for doc in vector_results:
    print(f"- {doc.page_content}")

print("\n--- Hybrid Search ---")
hybrid_results = vectorstore.similarity_search_with_score(
    "data storage algorithms",
    k=2,
    search_type=SearchType.HYBRID,
    vector_weight=1.0,
    keyword_weight=1.0
)
for doc, score in hybrid_results:
    print(f"Score: {score:.3f} - {doc.page_content}")

print("\n--- MMR Search for Diversity ---")
diverse_results = vectorstore.max_marginal_relevance_search(
    "technology concepts",
    k=3,
    lambda_mult=0.7
)
for doc in diverse_results:
    print(f"- {doc.page_content}")

Future Enhancements

Additional Distance Strategies

Support for additional distance strategies is planned:

MAX_INNER_PRODUCT: Maximum inner product similarity
DOT_PRODUCT: Dot product similarity
JACCARD: Jaccard similarity coefficient

Graph-Enhanced Search

Integration with ArangoDB’s graph capabilities for enhanced semantic search:

# Future graph-enhanced search API (planned)
# results = vectorstore.graph_enhanced_search(
#     query="your query",
#     k=5,
#     graph_traversal_depth=2,
#     include_connected_nodes=True
# )

Vector and Keyword Search

For direct control over hybrid search with both vector and keyword components:

# Hybrid search with pre-computed embedding
query_embedding = embeddings.embed_query("machine learning")

# Search combining vector similarity and keyword matching
results = vectorstore.similarity_search_by_vector_and_keyword(
    query="artificial intelligence",
    embedding=query_embedding,
    k=5,
    vector_weight=1.0,
    keyword_weight=1.0,
    keyword_search_clause="SEARCH doc.text IN TOKENS(@query, 'text_en')"
)

# With scores
docs_and_scores = vectorstore.similarity_search_by_vector_and_keyword_with_score(
    query="deep learning",
    embedding=query_embedding,
    k=3,
    vector_weight=2.0,  # Favor vector similarity
    keyword_weight=1.0
)

Important Notes:

insert_text requirement: When using hybrid search (SearchType.HYBRID), the insert_text parameter must be set to True in add_texts, add_embeddings, and from_texts methods. This ensures text content is stored for keyword search.
Hybrid search prerequisites: Both vector and keyword indexes must be created before performing hybrid search
RRF scoring: Hybrid search uses Reciprocal Rank Fusion to combine vector and keyword search results

# Correct usage for hybrid search
vectorstore.add_texts(
    texts=["New document"],
    insert_text=True  # Required for hybrid search
)