Kota AI

A customer uploads docs, URLs, or Q&A pairs, and the ingestion step turns that into something a model can actually search. Text gets extracted, chunked, and embedded through OpenRouter, then stored in a vector database, with one collection per chatbot so one customer's knowledge base never leaks into another's answers.

That pipeline used to run inline on the request path, which meant a large upload could time out before it finished. It now runs on its own queue backed background worker, so an upload starts, gets acknowledged immediately, and finishes asynchronously instead of holding a connection open until it either completes or times out.

Vector search alone misses exact terms a customer actually typed. Kota runs vector search and keyword search in parallel and fuses the two with Reciprocal Rank Fusion, so a literal match and a semantic match both get a fair shot at the top of the list.

A cross encoder reranking pass sits after that fusion step, but only fires when the fused results are genuinely ambiguous. When there's already a clear winner, reranking is skipped entirely, saving a model call on the queries that didn't need one. That conditional is a cost decision as much as a latency one, and it only works because the system knows how to tell a clear winner from a close call.

Answers are grounded in whatever the retrieval step actually returned, not generated freely. When confidence is low or nothing relevant comes back, the conversation doesn't get a made up answer. It gets handed to a human, live, inside the same inbox the bot was just working in, so the customer never has to repeat themselves to a person.

Replies stream end to end, from the retrieval engine through a relay layer to an embeddable widget running on the customer's own site, so a response appears token by token rather than arriving as one block after a pause. The inbox itself runs on real time socket delivery, so a human agent watching a handoff comes in sees the conversation update live, not on a refresh.

A CI gated eval harness runs a golden dataset through the live engine on every change, scoring retrieval hit rate, recall at k, mean reciprocal rank, handoff correctness, and answer quality through keyword coverage and an optional LLM as judge pass. It exits non zero on a regression, which means a retrieval change that quietly makes answers worse gets caught in CI, not by a customer.

The vector store itself moved once already, from a managed vector search product to a self hosted instance, for cost and control reasons, after the first choice stopped holding up under real usage. And every model call, chat, embeddings, reranking, was consolidated onto OpenRouter, replacing what had been a fragmented setup with hardcoded credentials scattered across providers.