What is Weaviate?
Weaviate is an open-source vector database, written in Go, that stores data objects together with their vectors so that vector search can be combined with structured filtering and keyword search.
Weaviate is developed by the company Weaviate and can be self-hosted or used as a managed cloud service. It is commonly used as the retrieval layer in RAG applications and AI agents, where it stores embeddings for documents and returns the most relevant ones for a query.
Core Architecture and Mechanics
Weaviate works through a few core behaviors:
- Stores objects alongside their embeddings. Each object has properties (like a JSON document) and one or more vectors. Vectors can be supplied by the application or generated by Weaviate’s integrations with embedding model providers.
- Executes similarity searches using approximate nearest neighbor indexes. HNSW is the default vector index, with flat and dynamic index types for smaller collections, and vector compression options to reduce memory use.
- Combines dense vector retrieval with BM25 keyword search in ‘Hybrid Search’. The two result lists are fused into one ranking, which helps with queries that mix concepts and exact terms such as product codes.
Weaviate scales horizontally through sharding and replication, and supports multi-tenancy for applications that keep each customer’s data separate. In a lakehouse architecture, it usually sits beside the Apache Iceberg tables: documents and their metadata live in the lakehouse, and a pipeline embeds them and loads the vectors into Weaviate for retrieval.
Why Weaviate Matters in the Modern Data Stack
Vector databases act as a memory layer for AI applications, letting Large Language Models search large proprietary document collections and answer questions grounded in them via RAG.
For data teams, the main design questions are how to keep Weaviate in sync with the source documents in the lakehouse, which embedding model to use, and how to filter results so users only retrieve documents they are allowed to see.
Key Benefits
- Hybrid retrieval: Vector, keyword, and filtered search in one query.
- Deployment choice: Open-source self-hosting or a managed cloud service.
- Ecosystem integrations: Client libraries for Python, TypeScript, Go, and Java, plus integrations with frameworks such as LangChain and LlamaIndex.
Frequently Asked Questions
How is a vector database different from a relational database?
Instead of searching for exact string matches, vector databases search for conceptual similarity based on proximity in high-dimensional space. Relational databases remain the better fit for exact lookups, joins, and transactions, so many architectures use both.
What is an embedding?
An embedding is a translation of text, images, or audio into a sequence of numbers that captures the underlying meaning of the content. Items with similar meanings get vectors that are close together.
How does Weaviate impact data governance and security?
Weaviate supports API key and OIDC authentication, and role-based access control (RBAC) has been generally available since version 1.29. Document-level permissions from source systems are not inherited automatically, so applications typically store access metadata on each object and filter on it at query time, or use separate tenants.
Learn More
To learn more about the Data Lakehouse, read The Lakehouse Built for Everyone by Alex Merced. You can find this and other books by Alex Merced at books.alexmerced.com.