What Building an Enterprise Knowledge Assistant Taught Us About Advanced RAG

---

Everyone is talking about AI assistants.

Upload some documents, connect an LLM, ask questions, and magically get answers.

At least that's the promise.

We developed and launched an enterprise knowledge assistant called Alto to 20,000 Certis staff in Mar across both frontliners and office-based staff. The idea is straightforward: teams can create knowledge channels around a particular topic, project, function, or community, upload or link relevant content, and then chat with that knowledge in natural language.

The content inside a channel can be almost anything — meeting minutes, policies, PowerPoint decks, spreadsheets, reports, operating procedures, technical documents, and sometimes thousands of files accumulated over years.

On the surface, it sounds simple.

Upload documents. Ask questions. Get answers.

In reality, that last step turns out to be surprisingly difficult.

The Problem Isn't Generating Answers. It's Finding the Right Information.

When most people see modern LLMs, they are impressed by how well they generate responses.

What we discovered is that generation is rarely the hardest problem.

Finding the right information is, that's what the first R of RAG stands for, the Retrieval.

Consider a seemingly simple question:

"Can I make a claim?"

To a human, the first reaction is usually:

"What kind of claim are you talking about?"

But to an AI system, that question immediately creates a retrieval challenge.

Does "claim" refer to:

• A business expense claim?

• A transport allowance claim?

• A work injury claim?

• A medical reimbursement claim?

All of these documents may exist within the same knowledge repository in Certis.

All contain the word "claim".

All describe completely different processes.

If we retrieve the wrong document, even the smartest LLM will confidently generate the wrong answer.

That was one of the first lessons we learned:

In enterprise AI, the quality of the answer is often determined long before the LLM starts generating it.

And that is exactly why Advanced RAG exists: dealing complex media (picture, charts, color-coded status bar, diagram, etc.), unclear user instructions, respository with thousands of files, temporal-linked documents (e.g. minutes for the same committee but over a period of time), etc.

Why Basic RAG Starts Breaking Down

One can easily create "Custom GPT" in their chatgpt account, and upload documents to the knowledge repository, and then release to whoever interested to query. The design works when both the knowledge base and user base are simple and small. It soon run into problems in enteprise setting.

Employees don't always use the same language as policy writers.

A user may ask about "reimbursement".

The document may use "expense recovery".

Many enterprise policies are dozens or even hundreds of pages long.

Some questions require information from multiple sources.

For example:

"Who approves this request and what supporting documents are required?"

The approval hierarchy may exist in one document.

The submission process may exist in another.

Neither document contains the complete answer.

Policies get updated.

Procedures change.

Meeting minutes capture decisions before formal documentation is revised.

What happens when different documents say different things and same things at the same time?

A good enterprise assistant should recognise the conflict, not confidently choose one version and pretend the other doesn't exist.

Thinking About RAG Differently

One of the best mental models I've come across is to stop thinking about RAG as search plus generation.

Instead, think of it as a multi-stage pipeline.

Each stage solves a different problem.

Stage 1: Understanding the Question

Query Rewriting

A vague question is transformed into a clearer search query.

For example:

"How much can I claim for lunch?"

might become below after taking additional information from users' profile and channel configuration:

"What is limit set for lunch claim in meal re-imbursement policy"

The retrieval system now has something meaningful to work with.

Query Expansion

Sometimes the user's words don't match the document vocabulary.

Instead of searching for only one version of the question, the system generates multiple related versions.

A query about parental leave may expand into:

• maternity leave

• paternity leave

• family leave

• new parent entitlement

The net becomes wider.

Recall improves significantly.

Query Decomposition

Complex questions are often split into smaller questions.

For example:

"Can contractors claim travel expenses and who approves them?"

can become:

• Can contractors claim travel expenses?

• Who approves travel expense claims?

The results are then merged together.

Stage 2: Finding the Right Documents

Once the question is understood, retrieval begins.

This is where many enterprise systems hit their first major limitation.

Hybrid Search

Traditional keyword search and semantic search each have strengths and weaknesses.

Keyword search is excellent when users know exact terms. It's fast and will not get lost in context overload.

Semantic search is excellent when users describe concepts.

Enterprise knowledge requires both.

A query about "expense claims" may need exact policy numbers while simultaneously understanding semantic meaning.

Hybrid retrieval combines keyword and vector search, consistently outperforming either approach alone.

If I could only recommend one retrieval improvement, this would probably be it.

Stage 3: Separating Signal from Noise

Retrieving documents is one thing.

Retrieving the right documents is another.

A search may return twenty seemingly relevant chunks.

Only three actually answer the question.

This is where reranking becomes valuable.

Reranking

Instead of asking:

"Which documents look similar?"

A reranker asks:

"Which document best answers this question?"

This may sound like a subtle distinction.

In practice, it is often one of the biggest quality improvements available.

For enterprise repositories where multiple policies contain similar terminology, reranking can make the difference between retrieving the correct policy and retrieving something merely related.

Stage 4: Understanding Large Knowledge Repositories

As repositories grow, another problem emerges.

Searching thousands of individual chunks becomes increasingly noisy.

Hierarchical Retrieval

Imagine trying to find information in a library.

You don't start by searching every page.

You first identify the correct section.

Then the correct book.

Then the correct chapter.

Then the specific page.

Hierarchical retrieval applies the same principle.

Instead of searching only detailed chunks, the system first searches summaries, then drills deeper into the relevant areas.

This becomes particularly useful for large policy repositories, operating manuals, and technical documentation.

GraphRAG

Some questions aren't really about documents.

They're about relationships.

For example:

"Which projects are related to this initiative?"

"Who owns this process?"

"Which systems are affected by this policy?"

Traditional retrieval finds documents.

GraphRAG finds connections. This technique is widely used in social networking apps to recommend "people you might know" for example.

It introduces relationships between entities and allows reasoning across the broader knowledge landscape.

While not necessary for every use case, it becomes increasingly valuable as enterprise knowledge becomes more interconnected. We so far has yet introduced this capability in our channel configurations, but we might add in later for some special use cases.

Stage 5: Verifying Before Answering

One lesson surprised me more than any other.

Many AI failures aren't generation failures.

They're confidence failures.

The system should always ask itself:

• Is the answer supported by evidence?

• Are there conflicting sources?

• Do I have enough information?

If the answer is uncertain, it should say so.

In enterprise environments, saying:

"I don't have enough information to answer confidently"

is often the correct answer.

Abstaining is not failure.

Hallucinating is.

In order to create more confidence in user, we have also added the quoted content link to every answer so that user can choose to read and validate against the source document.

Beyond Text: The Next Challenge

One thing that quickly became obvious is that enterprise knowledge isn't stored only in text.

Critical enterprise knowledge often resides in formats that traditional search struggles to understand:

• PowerPoint presentations

• Excel spreadsheets

• Tables

• Diagrams

• Images

• Charts and visualizations

In many organizations, the most important insight or decision-driving metric may exist only within a table buried on slide 37 of a presentation, rather than in searchable text.

To be truly effective, enterprise knowledge assistants must retrieve and reason over meaning—not just keywords or text.

This is where multimodal retrieval becomes essential. Our approach transforms rich-media content into machine-understandable knowledge by leveraging advanced reasoning models to generate detailed descriptions and semantic interpretations of tables, charts, diagrams, images, and other visual elements.

These enriched descriptions are then indexed and embedded into a vector database alongside the original content. As a result, when users ask questions, the retrieval engine can surface knowledge contained not only in text documents, but also in visual and structured content that would otherwise remain inaccessible.

This enables the assistant to understand and retrieve insights hidden within presentations, spreadsheets, diagrams, and other rich-media assets, significantly expanding the organization's searchable knowledge base.

Is RAG Dead?

Every few months, somebody declares that RAG is dead.

Usually for one of two reasons.

The first argument is that context windows are becoming enormous.

If models can process millions of tokens, why bother retrieving anything?

The answer is surprisingly simple.

Cost.

Latency.

And information overload.

Just because a model can read an entire repository doesn't mean it should.

Searching for one sentence inside millions of tokens is still inefficient.

The second argument is that AI agents will replace RAG.

I find this one particularly amusing.

When an agent needs information, what does it do?

It searches.

It retrieves relevant content.

It grounds its reasoning in retrieved knowledge.

In other words, it performs RAG.

Agents didn't replace RAG.

They simply gave RAG a bigger role to play.