Orchestration Tool: Weaviate and Airflow Integration (Section Two)

Guide

Table of Contents

• Overview
• Weaviate and Airflow Integration Guide
  • 1. Install and Run Weaviate
    • Option A: Run Weaviate with Docker Compose
    • Option B: Run Weaviate as a Standalone Docker Container
  • 2. Install the Weaviate Python Client in the Airflow Environment
  • 3. Connect Airflow Code to Weaviate
  • 4. Use Weaviate Inside Airflow Tasks
  • 5. Configure a Connection in Airflow (Optional but Recommended)
  • 6. Restart or Rebuild Airflow When Needed
  • 7. Verify that the Integration Works
  • 8. Summary
• Quick Notes

Overview

This page provides a practical guide for integrating Weaviate, a vector database, with Apache Airflow pipelines. It covers installation, Docker-based deployment, Python client setup, and

This document explains how to install Weaviate and connect it to an Airflow pipeline using Docker and Python. The goal is to make Weaviate available as a local vector database and let Airflow tasks insert data into it or query it during a workflow.

Weaviate and Airflow Integration Guide

1. Install and Run Weaviate

The easiest way to run Weaviate locally is with Docker. You can either add it to your existing docker-compose.yaml or run it as a standalone container.

Option A: Run Weaviate with Docker Compose

Add a weaviate service to your existing docker-compose.yaml file:

services:
  weaviate:
    image: semitechnologies/weaviate:latest
    ports:
      - "8081:8080"
    environment:
      - QUERY_DEFAULTS_LIMIT=20
      - AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true
      - PERSISTENCE_DATA_PATH=/var/lib/weaviate
      - DEFAULT_VECTORIZER_MODULE=text2vec-transformers
      - TRANSFORMERS_INFERENCE_API=http://localhost:8081
      - ENABLE_MODULES=text2vec-transformers

Then start only the Weaviate service with:

docker compose up -d weaviate

This command starts Weaviate in the background and maps it to localhost:8081 on your machine.

Option B: Run Weaviate as a Standalone Docker Container

If you do not want to modify docker-compose.yaml, you can run Weaviate directly with Docker:

docker run -d -p 8081:8080 semitechnologies/weaviate:latest

This also makes Weaviate available locally at:

http://localhost:8081

2. Install the Weaviate Python Client in the Airflow Environment

Your Airflow tasks need the Python client library to communicate with Weaviate. The best approach is to install it in the same environment or Docker image used by Airflow.

Add the package to your requirements.txt file:

weaviate-client

Or install it manually with:

pip install weaviate-client

If you are using Airflow with Docker, it is better to add this dependency to your custom Airflow image rather than installing it manually inside a running container. That makes the setup more stable and reproducible.

3. Connect Airflow Code to Weaviate

Once Weaviate is running and the Python client is installed, you can connect to it from your DAG code.

In your Airflow DAG Python files, use code like this:

import weaviate

client = weaviate.Client("http://localhost:8081")
# Use the client to create collections, insert data, or run queries

This client object is the entry point for interacting with the local Weaviate instance.

If Airflow and Weaviate are running in separate Docker containers, make sure the hostname is correct. In some setups, localhost inside the Airflow container does not point to the Weaviate container. In that case, you may need to use the service name, such as:

client = weaviate.Client("http://weaviate:8080")

The correct address depends on how your Docker network is configured.

4. Use Weaviate Inside Airflow Tasks

After the connection is working, you can create Airflow tasks that insert data into Weaviate or query it during a pipeline run.

For example:

from airflow.decorators import task

@task
def insert_to_weaviate():
    import weaviate

    client = weaviate.Client("http://localhost:8081")
    # Add logic here to create a collection, insert objects, or query data

This task can be part of a larger DAG. For example, one task may read and process book descriptions, another may generate embeddings, and then insert_to_weaviate() may store the embeddings and metadata in the vector database.

It is a good idea to keep these tasks focused and simple. For example:

• one task for reading data
• one task for creating embeddings
• one task for loading data into Weaviate
• one task for querying recommendations

This makes the pipeline easier to debug, retry, and maintain.

5. Configure a Connection in Airflow (Optional but Recommended)

Instead of hardcoding the Weaviate URL in every DAG file, you can define the connection in Airflow. This makes your code cleaner and easier to manage.

You can add a connection such as my_weaviate_conn either:

• in the .env file for your Airflow project, or
• manually in the Airflow UI under Admin > Connections

This connection stores the details needed for Airflow tasks to reach the Weaviate service. Using Airflow connections is a better long-term practice, especially when moving from local development to a shared or production environment.

6. Restart or Rebuild Airflow When Needed

If you add new Python dependencies, change your Docker configuration, or update your Airflow image, you may need to restart the services.

For a normal restart:

docker compose down
docker compose up -d

If you changed the Dockerfile or requirements.txt, you should rebuild the image before starting again:

docker compose build --no-cache
docker compose up -d

This ensures Airflow containers use the updated environment.

7. Verify that the Integration Works

After everything is running, test the setup before building your full pipeline.

You can verify that:

• Weaviate is running on the expected port
• The Python client imports successfully inside Airflow
• A simple Airflow task can connect to Weaviate without errors

For example, you can test package installation with:

docker compose run --rm airflow-webserver python -c "import weaviate; print('weaviate import OK')"

You can also create a simple DAG task that connects to the client and prints a success message.

8. Summary

To integrate Weaviate with Airflow, you first run Weaviate locally with Docker, then install the weaviate-client package in the Airflow environment, and finally connect to Weaviate from within your DAG tasks. Once the connection is established, Airflow can automate steps such as loading embeddings into Weaviate and querying the vector database as part of a RAG or recommendation pipeline.

This setup is especially useful for projects such as a book recommendation app, where Airflow can orchestrate the workflow and Weaviate can store and retrieve vectorized book descriptions efficiently.

Quick Notes:

Note 1: To accessing Data Stored in Weaviate Running Inside a Container

When Weaviate is running inside a Docker container, accessing the data depends on where your code is running (inside another container or on your local machine). This is an important concept when integrating with Apache Airflow.

Even though Weaviate runs inside a container, it exposes an API (usually HTTP). You interact with it through this API using the Weaviate Python client.

However, the correct URL/hostname depends on your setup.

Case 1: Access from Local Machine (Host)

If your Python script (or notebook) runs on your local machine, and you mapped the port like this:

ports:
  - "8081:8080"

Then you can access Weaviate using:

client = weaviate.Client("http://localhost:8081")

This works because Docker maps container port 8080 to your machine port 8081.

Case 2: Access from Another Docker Container (for example, Airflow)

If Airflow is also running inside Docker (which is common), then localhost will not work, because each container has its own network namespace.

Instead, use the service name defined in docker-compose.yaml:

client = weaviate.Client("http://weaviate:8080")

Here:

• weaviate is the service name in docker-compose.yaml
• 8080 is the internal container port

Example: Docker Compose Setup:

services:
  airflow:
    ...

  weaviate:
    image: semitechnologies/weaviate:latest
    ports:
      - "8081:8080"

In this setup:

• From your laptop: use localhost:8081
• From Airflow container: use weaviate:8080