Do not start with a vector database
Many internal AI assistants start with embeddings by default. Test classic search against real questions first, and add vectors only where an eval set shows they are needed.
When a team builds an internal assistant that answers questions from company documents, the architecture diagram often arrives with a vector database already drawn in. It is the default of the tutorials. It is rarely the result of a measurement. The outcome is one more system to run before anyone has shown that the search you already have is not enough.
Look at the questions people ask
Collect real questions from the people who will use the assistant, not invented ones. Sort them. A large share is usually lookup: a contract number, a customer name, a date range, a product code, a person. Vector search matches meaning approximately, which is exactly what makes it weak at exact matches. Keyword search and a SQL filter are the opposite: precise on identifiers, poor when the question uses different words from the document. Both are in your stack already.
Build the baseline first
Write down what a correct answer looks like for a set of fifty or so questions. Run them against full-text search and structured filters, and record how many come back right. That result is the baseline, and it is what every later addition has to beat. Without it, adding vectors cannot be called an improvement. It can only be called a change.
Add vectors where the baseline fails
Embeddings earn their place when the failure is vocabulary: the user asks about ending a rental early, and the document says early termination of the lease. When the eval set shows that class of failure, hybrid retrieval is justified: keyword plus vector, results merged. You can then measure the gain it brings, and decide whether it is worth what it costs.
The cost of one more store
A vector index is a copy of your documents. It has to be kept in sync with the source, rebuilt when the embedding model changes, and it has to enforce the same access rights as the system it copies. A user who cannot open a document in the original application must not be able to retrieve it through the assistant. Permission mistakes in a copy are quiet and hard to find, and when personal data is involved they are also a compliance problem.
None of this argues against vector search. It argues for an order of work. A team with a scored baseline can say what embeddings added. A team without one can only say the assistant now feels better. When the next model or index version arrives, the first team re-runs the question set and knows whether to switch.
Start with the boring retrieval, measure it, and let the failures tell you what to add.
Which questions in your assistant are lookups in disguise?