Do I need a vector database for my AI agents?

What a Vector Database Does

A vector database stores content as numerical representations — embeddings — that capture semantic meaning rather than exact text. When an agent needs information, it searches by similarity: finding content that means something close to the query, even when the exact words don't match. That's what powers RAG pipelines. The agent gets relevant documents surfaced into its context window at the moment it needs them.

For retrieval at scale across unstructured content — documentation, support articles, product specs, contracts — vector databases are genuinely useful infrastructure. Semantic search outperforms keyword search for most natural-language queries, and the tooling has matured considerably. If retrieval is the problem, a vector database is a reasonable part of the answer.

The limitation is also straightforward: a vector database stores and retrieves whatever you put in with equal efficiency. It is neutral on quality. Whether what comes back is actually useful to the agent depends entirely on what went in and whether it was curated before ingestion.

The Curation Problem Vector Databases Don't Solve

Two organisations can have identical vector database infrastructure and radically different agent behaviour. The difference is almost always in what was put in and how carefully.

Automatic ingestion — pulling in documents because they exist — produces retrieval results that accurately reflect the corpus. If the corpus contains outdated policies alongside current ones, the agent retrieves both. If the reasoning behind a decision was never written down, the agent has no access to it. If a constraint is understood by everyone on the team but doesn't appear in any document, it simply isn't there.

This is a curation problem, not a retrieval problem. A better vector database doesn't fix it. More documents don't fix it. The knowledge that's missing was never in the corpus to begin with.

What a Vector Database Doesn't Tell Your Agent

There's a second gap that retrieval infrastructure doesn't address, even with good curation.

Some of the knowledge agents need to operate well isn't document-shaped. The legal requirement that every outbound recommendation remain human-reviewable. The pricing exception agreed with a group of enterprise customers. The support escalation path that changed after an incident in March. This knowledge exists — but it exists in people's reasoning, in meeting outcomes, in the shared understanding of teams that never needed to write it down because everyone already knew.

No retrieval system reaches it. You can't index what was never written. Vector database or not, an agent that lacks this context will produce outputs that are right by the documents, wrong for the situation.

Two Different Questions

'Do I need a vector database?' is a question about retrieval infrastructure. 'What should my agent know about my organisation?' is a question about authored context. Both matter, and confusing them leads to solving the first and assuming the second is covered.

A well-implemented vector database answers the retrieval question. Agents can find relevant documents. That is genuinely useful. It doesn't answer the second question — what the organisation has decided, what its agents are permitted to do, what constraints they're operating under — because that knowledge has to be captured deliberately before it can be surfaced at all.