Post

Vector Databases: Weaviate

Posted Updated
By Maryam Khani
2 min read
Vector Databases: Weaviate

Guide

Table of Contents

Overview

This post provides a practical guide to integrating vector databases—specifically Weaviate—into AI/ML pipelines for tasks such as Retrieval-Augmented Generation (RAG), recommendation systems, and semantic search. It draws on real-world orchestration and implementation patterns from Airflow-based pipelines.

Understanding Vector Databases

What Are Vector Databases?

Vector databases are designed to store and search high-dimensional vectors (embeddings) efficiently. They are essential for modern AI/ML applications that rely on semantic similarity, such as document retrieval, recommendation, and question answering. Unlike traditional databases, vector databases enable fast nearest-neighbor search and hybrid queries combining structured and unstructured data.

Key Concepts:

  • Embeddings: Numeric representations of data (text, images, etc.) in high-dimensional space.
  • Similarity Search: Finding items most similar to a query vector (e.g., cosine similarity, Euclidean distance).
  • Hybrid Search: Combining vector search with filters on metadata (e.g., search for books by topic and similarity).

When to Use Vector Databases

  • When you need semantic search (beyond keyword matching)
  • For recommendation systems (find similar users/items)
  • For RAG (Retrieval-Augmented Generation) in LLM applications
  • For image/audio similarity and clustering

Weaviate: An Open-Source Vector Database

Weaviate is a popular open-source vector database that supports:

  • Native vector storage and search
  • Integration with embedding models
  • Hybrid queries (vector + keyword)
  • RESTful and Python client APIs
  • Easy deployment via Docker

Typical Workflow with Weaviate

  1. Define a Schema: Describe your data objects and their properties.
  2. Ingest Data: Insert objects with both metadata and vectors.
  3. Query: Use vector similarity, filters, or hybrid queries.
  4. Integrate: Connect with ML pipelines, LLMs, or search interfaces.

Best Practices

  • Normalize vectors before storage (if required by your model)
  • Use batch inserts for large datasets
  • Monitor index build and query latency
  • Secure your deployment (authentication, network)
  • Regularly update your embedding models for best results

Example: Book Recommendation Pipeline with Airflow, Embeddings, and Weaviate

This example demonstrates how to build a modern recommendation system using vector databases and orchestration tools. The pipeline covers the full lifecycle: generating embeddings from raw text, running a vector database (Weaviate) locally, storing vectors with metadata, and performing semantic search for recommendations. By following these steps, you learn how to connect machine learning models with scalable, production-ready infrastructure for real-world AI/ML applications. This approach is widely applicable to document retrieval, question answering, and any use case requiring similarity search over unstructured data.

1. Generate Embeddings: Use a model like BAAI/bge-small-en-v1.5 to convert text (e.g., book descriptions) into dense vectors:

1
2
3

from fastembed import TextEmbedding
model = TextEmbedding("BAAI/bge-small-en-v1.5")
embeddings = list(model.embed(["book description"]))

2. Run Weaviate Locally:

Deploy Weaviate using Docker:

1

docker run -d -p 8081:8080 semitechnologies/weaviate:latest

3. Store Vectors in Weaviate:

Connect with the Python client and insert objects with both metadata and vectors:

1
2
3
4
5
6
7
8
9
10
11

import weaviate
client = weaviate.Client("http://localhost:8081")
collection = client.collections.get("Books")
collection.data.insert({
    "vector": embedding,
    "properties": {
        "title": title,
        "author": author,
        "description": description
    }
})

4. Query by Vector Similarity:

Retrieve recommendations or similar items using vector search:

1
2
3
4

results = collection.query.near_vector(
    near_vector=query_embedding,
    limit=1
)

Resources

Data Engineering, Big Data Processing
structured-data unstructured-data semi-structured-data databases vector-database weaviate embeddings RAG python data-pipeline recommendation mlops big-data-processing
This post is licensed under CC BY 4.0 by the author.