RAG on AWS Bedrock: Architecture Guide for Enterprise Teams

AWS Bedrock Knowledge Bases makes a compelling promise: point it at an S3 bucket, and you get a managed RAG pipeline — chunking, embedding, a vector store, and a RetrieveAndGenerate API — without standing up the plumbing yourself. For getting a working RAG prototype on AWS, it's genuinely excellent.

But "managed RAG pipeline" and "production enterprise RAG architecture" are not the same thing, and the gap between them is where teams get surprised. Bedrock handles the middle of the pipeline well; the parts that determine whether your system is secure, multi-tenant, governable, and affordable at scale live around it — and those are decisions the console won't make for you. This guide covers the full picture for AWS Bedrock RAG: what Bedrock owns, what you still own, and the four architecture calls that separate a demo from production.

What Bedrock Knowledge Bases handles — and what you still own

Bedrock Knowledge Bases manages a real chunk of the pipeline:

• Ingestion from S3 (and a growing set of connectors), with managed sync.
• Chunking — fixed-size, semantic, or hierarchical options.
• Embedding — via Bedrock models (Amazon Titan or Cohere).
• Vector storage — in a vector store you nominate (more on that below).
• Retrieval + generation — the Retrieve and RetrieveAndGenerate APIs, with optional Bedrock Guardrails for content filtering.

That's the middle three layers of the 6-layer RAG architecture done for you. Here's what Bedrock does not solve, and what your team still has to architect:

• Getting data into S3 in the first place. Your knowledge lives in SharePoint, SQL, SAP, Confluence, Salesforce, and ticketing systems — not S3. Building secure, incremental, permission-preserving connectors into the bucket is on you.
• Permission-aware, multi-tenant access control. Bedrock supports metadata filtering, but enforcing "this user can only retrieve from sources they're allowed to see" — across tenants or departments — is an architecture you design, not a checkbox.
• Evaluation. Measuring retrieval precision/recall and faithfulness, building a golden dataset, and red-teaming are yours.
• Observability, audit, and cost governance. Logging every query, mirroring to a SIEM, tracking spend per team, and alerting on drift sit outside the managed layer.
• Memory and orchestration. Persistent cross-session memory and any agentic/adaptive routing are application-level concerns.

The mental model: Bedrock is a strong managed layer, not the whole architecture. Treat it as the engine, and design the chassis — connectors, access control, evaluation, governance — around it.

Decision 1: which vector store behind Bedrock?

Bedrock lets you choose the vector store, and the choice is an operating-model decision, not a performance one:

• OpenSearch Serverless — the default, fully managed, scales automatically, and gives you strong hybrid (BM25 + vector) retrieval. The trade-off is cost: serverless OpenSearch has a non-trivial minimum spend, which can sting for smaller workloads.
• Pinecone — fully managed and simple, integrates with Bedrock, but adds a second vendor and a SaaS data boundary outside your AWS account.
• Aurora PostgreSQL (pgvector) — keeps vectors in your own relational database inside your AWS account, with the data control and cost profile that implies. The best fit when residency and "nothing leaves our account" lead the requirements.
• Redis / MongoDB Atlas / Neptune Analytics — viable when you already operate them.

For most enterprises the real choice is OpenSearch Serverless (managed, hybrid, AWS-native) versus Aurora pgvector (maximum control, lower floor cost). Pick based on your residency and cost posture. (See the full vector database comparison for the cross-cloud view.)

Decision 2: IAM architecture for multi-tenant RAG

This is the decision teams underestimate most, and the one most likely to cause a breach. A single shared Knowledge Base with no tenant isolation will, sooner or later, return one tenant's (or department's) content to another. Production multi-tenant RAG on Bedrock needs a deliberate isolation strategy:

• IAM at the boundary — scope which identities can invoke which Knowledge Bases and models, with least-privilege roles per service and per tenant.
• Metadata filtering at retrieval — tag every chunk with tenant/owner/access metadata and enforce the filter on every Retrieve call so a query can only ever match permitted content.
• Application-layer enforcement — never trust the client to send the right filter; resolve the caller's permitted scope server-side and apply it.
• Isolation model — decide between a shared KB with strict metadata filtering, a KB per tenant, or per-tenant data sources, based on your tenancy and blast-radius requirements.

The principle is the same one that governs all enterprise RAG: a user must never be able to retrieve content they couldn't open directly. On Bedrock, that's an IAM-plus-metadata architecture you own.

Decision 3: model selection (Claude vs. Titan, and beyond)

Bedrock's strength is the model menu. The practical guidance:

• Generation — Anthropic's Claude models (Haiku/Sonnet/Opus tiers) are a strong default for grounded, well-reasoned answers; route by complexity (a cheap, fast tier for simple lookups, a stronger tier for hard or high-stakes questions) rather than using one model for everything.
• Embedding — Amazon Titan embeddings keep vectorization inside AWS and are cost-effective; Cohere on Bedrock is a strong multilingual alternative.
• Don't over-fit to one model. Models improve constantly. Architect so you can swap generation models behind the same retrieval and governance layer — the LLM is the most replaceable part of the stack.

Sphere's AI Foundry stack is deliberately multi-model and multi-cloud — AWS Bedrock, Azure OpenAI, Claude, OpenAI, Llama/Mistral — precisely so model choice stays a tunable parameter, not a lock-in.

Decision 4: cost optimization at scale

Bedrock RAG cost comes from three meters, and all three are controllable:

• Vector store — OpenSearch Serverless capacity is often the biggest surprise; right-size it, and consider Aurora pgvector if your floor cost matters more than fully-managed scaling.
• Embeddings — a one-time cost per chunk plus a per-query cost; re-embedding the whole corpus on every model change is the expensive mistake.
• Generation tokens — the recurring cost. Model routing (Haiku for the easy 80%, Sonnet/Opus for the hard 20%), tight retrieval (fewer, better chunks in the context window), and caching common answers can cut generation spend dramatically.

Wrap it in cost governance — spend tracking per team and budget alerts — so the bill is visible before it's a quarterly surprise.

Putting it together

A production Bedrock RAG architecture looks like this: secure connectors land enterprise data in S3 with permissions preserved → Bedrock Knowledge Bases chunks, embeds, and stores it in your chosen vector store → retrieval enforces IAM and metadata filters so users only see permitted content → Claude (routed by complexity) generates grounded, cited answers → and around all of it you run evaluation, audit logging, observability, and cost governance, with deployment matched to your risk profile (SaaS, VPC, private cloud, or on-prem). Bedrock does the heavy lifting in the middle; the architecture that makes it enterprise-grade is the part you design.

Building RAG on AWS? Get a Bedrock RAG architecture review and we'll pressure-test your vector store, IAM/multi-tenancy, model, and cost design.

Related: the 6-layer RAG architecture framework, the enterprise vector database comparison, and enterprise RAG security.