What is a vector database and what should you consider when choosing one?

What Is a Vector Database? The new data infrastructure of the AI Era

If you search for "cell phone" on an e-commerce site and it also returns products described as "smartphone" or "mobile device," there is a good chance a vector database is working behind the scenes. Traditional databases match words exactly. Vector databases let you query by meaning, context and similarity.

The problem - data no longer fits in rows and columns

Relational databases were built for structured data: tables, schemas and exact matches. But much of today's data is unstructured: text, images, audio and video. Loading and preparing that data for AI workloads in a relational system is labor-intensive and keyword search only returns results containing the exact term you typed.

A vector database closes this gap. It converts data into arrays of numbers and measures how close they are to each other geometrically.

Core concepts - Vectors and Embeddings

A vector is an ordered list of numbers, where each number represents one feature (dimension) of the data. A day's low, average and high temperature, stored as three numbers, is a simple vector.

An embedding is what you get when an AI model converts text, an image, or audio into such a vector while preserving its meaning. In a good embedding space, "car," "automobile" and "vehicle" land close together even though they are spelled differently. Embeddings can have anywhere from a few hundred to tens of thousands of dimensions.

How a Vector Database works

The sources describe essentially the same workflow:

1. Vectorization: An embedding model converts the content you want to index into vectors.

2. Storage: Vectors are stored along with metadata (title, date, category) and a reference to the original content.

3. Indexing: Vectors are organized into specialized data structures so searches run fast.

4. Querying: The user's query is converted to a vector using the same model. The database finds the nearest vectors and returns the top K results.

5. Post-processing (optional): Results may be filtered or re-ranked.

Because embeddings are computed ahead of time and stored, nothing needs to be recomputed at query time, so results can come back in milliseconds.

How Is similarity measured?

Common measures include:

• Cosine similarity: Compares the direction of two vectors. It is often preferred for text embeddings.

• Euclidean distance: The straight-line distance between two points.

• Dot product: Takes both direction and magnitude into account.

The choice affects results, so it should match your use case.

Where does the speed come from? ANN and Indexes

Comparing a query against billions of vectors one by one is impractical. Vector databases use approximate nearest neighbor (ANN) search, trading a small amount of accuracy for a large gain in speed. Key techniques:

• HNSW: Builds a layered graph. Upper layers make long jumps to get near the target and the bottom layer uses dense local links to find the closest neighbors. It is the default choice in many systems.

• LSH: Uses hash functions to place similar vectors in the same bucket, then searches only within the relevant bucket.

• Product Quantization (PQ): Splits vectors into segments and represents each with a short code. It cuts memory use sharply at the cost of some accuracy.

• IVF / DiskANN: IVF is clustering-based and memory-efficient. DiskANN keeps most of the index on NVMe SSDs instead of RAM.

The cost of this choice is real. By Zilliz's example, 100 million vectors at 1024 dimensions need roughly 400 GB of RAM. At that scale, disk-based indexes cut cost significantly, in exchange for a few extra milliseconds of query latency.

A Vector Index ss not a Vector Database

Libraries like FAISS make similarity search very fast, but they are indexes, not databases. They lack much of what production systems need:

• Create, read, update, delete (CRUD) operations

• Metadata storage and filtering

• Replication, fault tolerance and backups

• Access control and multi-tenancy

• Real-time updates without rebuilding the whole index

A vector database combines vector search with the operational capabilities of a database.

Filtering and hybrid search

Real applications rarely want just "the most similar item." They want "the most similar documents written after a certain date, in this category." There are two approaches:

• Pre-filtering: Apply the metadata filter first, then run the vector search. This shrinks the search space but can miss relevant results.

• Post-filtering: Run the vector search first, then filter. This is more thorough but adds overhead.

Hybrid search is also becoming a standard expectation. It combines semantic (dense) vector search with keyword matching, metadata filters and re-ranking models. Since vector search can miss exact term matches, the two approaches complement each other.

Use sases

• RAG (Retrieval-Augmented Generation): LLMs are grounded in internal documents, policies, or support records, so answers rely on current sources rather than only training data. This reduces hallucinations and makes answers traceable to their sources.

• Semantic search: Enterprise knowledge bases, legal document search, customer support systems.

• Recommendation systems: User and item embeddings are compared to produce personalized suggestions.

• Visual search: Finding similar products, visual discovery, content moderation.

• Anomaly and fraud detection: Normal behavior clusters in vector space and points far from those clusters are flagged as suspicious.

Serverless Vector Databases

First-generation vector databases were fast, accurate and scalable, but expensive. The newer serverless approach targets three problems:

• Separating storage from compute: Compute runs only when needed, while the index sits in cheap storage.

• Multi-tenancy: Rarely queried datasets don't inflate costs.

• Freshness: Newly added data should be queryable within seconds. A "freshness layer" temporarily holds new vectors until they are merged into the main index.

Challenges and limits

A vector database is not a cure-all:

• New data pipelines are required. Raw data must be cleaned, embedded and kept in sync with its source.

• Security and governance need separate attention. Unauthorized users shouldn't be able to reach confidential data through an LLM.

• Not every query type fits. Tasks like topic summarization or broad thematic analysis need the model to read the full context rather than just nearest neighbors and non-vector structures can be more efficient there.

• Memory and cost management is a serious engineering topic at scale.

• The accuracy-speed tradeoff: Higher recall means slower queries. Production systems typically target somewhere between 80% and 99% recall depending on the use case.

How to choose

Options fall into several groups:

• Proprietary managed service (Pinecone): A good fit if you don't want to manage infrastructure.

• Open source, purpose-built (Milvus, Qdrant, Weaviate, Chroma): A good fit if you want flexibility and control.

• Library / index (FAISS): A good fit if you're building custom infrastructure.

• Extension to an existing database (PostgreSQL + pgvector): A good fit if you already use Postgres and don't want another system.

• Search engine (OpenSearch): A good fit if you already have search infrastructure.

• Lakehouse integration (Databricks, IBM watsonx.data): A good fit if your data platform lives in that ecosystem.

A few quick notes: Chroma is convenient for prototyping, Milvus is designed to scale to billions of vectors and Qdrant is written in Rust with a focus on filtering. With pgvector, HNSW is a good starting point in most cases, while IVFFlat is worth considering when you need memory efficiency and shorter index build times. Make sure your index fits in memory.

When deciding, ask: How much data do you have? How critical are hybrid search and metadata filtering? How well does it integrate with your existing data ecosystem? Do you want to run operations yourself or use a managed service?

Market and Trends

According to the estimate cited by Instaclustr, the global vector database market is about $2.58 billion today and is projected to reach $17.91 billion by 2034, a compound annual growth rate of 24%. North America holds roughly 41% of the market. IBM cites 2025 research showing vector database adoption grew 377% in a year, the fastest growth reported among LLM-related technologies.

On the technical side, notable trends include the spread of RAG and agentic AI workflows, hybrid search becoming a baseline expectation, improvements in multi-vector search and work on reducing embedding size and dimensionality.