From RAG to Agents: Building a Grounded Assistant on Amazon Bedrock

There is an assistant in the corner of this page. It answers questions about our work, declines everything else, and will book a meeting with us on a real calendar if you ask it to. We built it partly because prospects kept asking us questions at 11pm, and partly because we wanted the pattern we recommend to clients running in public where anyone can prod it.

This is how it works, and more usefully, why it is built this way rather than some other way.

The problem with a bare language model on a company website

Point a capable model at your website with a friendly system prompt and you get something that reads well and cannot be trusted. It will describe services you do not offer, because your industry generally offers them. It will quote pricing that sounds plausible. Asked about a technology you have never touched, it will produce a paragraph about your extensive experience with it.

None of this is the model malfunctioning. It is doing exactly what it was trained to do — produce likely text — and “likely” is not the same as “true of this company”. For a consultancy, where the entire proposition is that we know what we are talking about, a confidently wrong assistant is worse than no assistant.

So the design constraint was set before any code: the assistant may only say things we have actually published.

Retrieval, and the floor beneath it

The standard answer is retrieval-augmented generation. Fetch relevant content, put it in the context window, tell the model to answer from it. That is necessary but not sufficient, and the gap is where most implementations fall down.

Here is the failure mode. Someone asks about something you have never written about. Retrieval dutifully returns the four least irrelevant chunks it can find, because a similarity search always returns its top matches — there is no natural “nothing here” result. The model receives some marginally-related text and an instruction to be helpful, and produces an answer that is technically grounded in something and substantively about nothing.

The fix is a relevance floor. Every retrieved chunk carries a similarity score, and anything below a threshold is discarded. If nothing survives, the assistant is told explicitly that no relevant company knowledge was found, and instructed to decline and point the visitor at a person.

That single mechanism does more for trustworthiness than any amount of prompt refinement. It also makes the assistant’s behaviour legible: when it says “I don’t have anything on that”, it is reporting a real state of the system, not performing modesty.

Try it. Ask ours about something we have obviously never done. It will refuse, and it will offer you the contact page instead. That is the design working.

Why the knowledge base is generated, not curated

The obvious way to build the knowledge base is to write one: assemble a folder of documents describing the company, embed them, done. We tried that first and abandoned it within a fortnight.

The problem is drift. The website changes — a new case study, revised service copy, a product repositioned. The hand-written knowledge base does not, because updating it is a separate task that nobody owns. Within a month the assistant is confidently describing a version of the company that no longer exists, and the failure is invisible until a prospect notices.

So the knowledge base is generated from the website’s own content as part of the deploy. Our site content lives as structured markdown; a build step flattens each entry into readable prose and publishes the result to a shared bucket. The assistant polls that bucket, notices new content, and re-embeds.

The consequences are worth spelling out, because this is the part I would most want to steal if I were reading someone else’s write-up:

• The website is the single source of truth. There is no second copy to maintain, so there is no drift.
• Publishing is deploying. A new case study becomes answerable within minutes of going live, with no separate release for the assistant.
• The failure mode is safe. If content generation breaks, the assistant has less to say and says less. It does not start improvising.

This last point is a general principle for agent design: when something breaks, the agent should get quieter, not more creative.

Bedrock, and why the hosting choice is usually the deciding factor

The assistant runs on Claude via Amazon Bedrock. The model choice is mostly about quality of instruction-following — we need something that reliably honours a refusal instruction under pressure, and it does. The hosting choice matters more, and it is the thing clients ask about first.

Through Bedrock, the model call happens inside our own AWS account, in a region we choose, authenticated by an instance role. There is no API key sitting in an environment variable to leak, no third-party account in the data path, and prompts are not used to train anyone’s model. When a client’s data policy says “our data does not leave our tenancy”, this is the answer that gets past their security review — the same pattern works on Azure OpenAI for organisations standardised there.

I would go further: for most enterprise agent projects, where the model runs is a bigger constraint than which model it is. Model quality is converging and swappable. Data residency is contractual.

The one tool that matters

The assistant has one real capability beyond answering: it can book a meeting. It checks genuine availability and writes to a real calendar, entirely server-side.

We deliberately resisted giving it more. There is a strong pull, once you have an agent with tool access, to keep adding tools — it could look up your order, update your preferences, send a summary email. Every one of those sounds useful and every one makes the agent less reliable, because now it has to choose correctly between more options on every turn, and each new tool is a new surface to secure.

One tool, doing the thing that actually converts an interested visitor into a conversation, has been worth more than a menu would have been.

The unglamorous half

Things that are not interesting to write about and are most of why this works in production:

A hardened scope. The system prompt defines what the assistant is for and, explicitly, what it must refuse: general knowledge, coding help, other companies, writing tasks, and any attempt to talk it out of its own instructions. Prompt injection is not hypothetical on a public endpoint; it arrives within days.

Rate limiting. Per IP and per session, sitting behind edge-level protection. A public LLM endpoint is a cost-amplification target — one script can turn your inference budget into someone’s afternoon entertainment.

Short answers by construction. The assistant is instructed to answer in one to three sentences, and the token limit enforces it. Long answers on a website chat widget go unread, and every extra sentence is another chance to say something you did not intend.

Session handling with a sliding expiry. Enough history for the conversation to make sense, bounded so that memory and token costs stay predictable.

Never revealing internals. The assistant declines questions about its own construction. This article exists precisely so that curiosity has somewhere to go that we control.

What we would tell you before you build one

If you are putting a grounded assistant on your own site:

1. Decide the remit first, in writing. What it is for and what it must refuse. That list is your guardrail specification, not documentation you write afterwards.
2. Build the relevance floor early. It is the highest-leverage component and it is easy to bolt on late and get wrong.
3. Generate the knowledge base from content you already maintain. Anything requiring a human to remember to update it will drift.
4. Assume the endpoint is hostile. Rate limits, injection resistance, and scoped tool credentials from day one.
5. Give it one action worth taking. Then resist adding a second until the first is demonstrably reliable.

The pattern generalises well beyond a website widget. A support agent grounded in your help centre and ticket history, an internal agent grounded in your runbooks, a procurement agent grounded in your supplier data — same shape: retrieve, refuse when you have nothing, act narrowly, log everything.

The case study on this assistant covers the outcomes, and our agent development page covers how we do this for clients. Or ask the assistant itself — it is right there, and it will tell you when it doesn’t know.