Beyond the chatbot: Building a shared knowledge layer for AI agents

On June 2, 2026, Microsoft announced general availability for core Foundry IQ knowledge bases, including a connection through which compatible agents can query them. On June 17, AWS announced Amazon Bedrock Managed Knowledge Base, bringing ingestion, storage, and retrieval together as a managed service. [1] (opens in new tab) [2] (opens in new tab)

The architectural opportunity is more interesting than another way to build a chatbot. When several applications need overlapping information, which parts should belong to each application—and which should the team maintain once?

Our recommendation is to separate the work of finding evidence from the agent's particular job. A support assistant and an onboarding agent can use the same retrieval service without sharing instructions, conversation histories, or responsibilities. That separation also creates a useful place to decide how much investigation a question deserves.

Give evidence its own service

Retrieval-augmented generation, or RAG, supplies a model with relevant source material to help it produce an answer. Finding that material and writing the answer are distinct operations. Microsoft's retrieval documentation makes this separation concrete: a knowledge base can return material for a calling model to use, rather than generate the final response itself. [3] (opens in new tab)

In the design we recommend, the knowledge service owns the route from source material to usable evidence. It connects to selected sources, maintains their searchable representations, retrieves permitted content, and preserves references to where that content came from. The application owns the user's task: understanding the request, deciding what to do with the evidence, and producing the appropriate output.

Suppose a support assistant and an onboarding agent both need a product's installation and compatibility guides. The support assistant explains an upgrade error. The onboarding agent prepares a setup plan. Their instructions differ, but both need a dependable way to find the same documentation.

Building separate pipelines would leave this hypothetical team maintaining the same source connections twice. A correction to document processing might reach one application before the other. With a shared service, the team has one place to improve that common retrieval behavior, then check its effect on both consumers.

Sharing evidence does not require identical results. The support assistant may need an explanation of an error, while onboarding needs prerequisites in the right order. Each request should carry enough task context to select useful material. Reuse should remove duplicated infrastructure, not erase the differences between applications.

Nor does this require one enormous company-wide database. We would keep separate collections where ownership, sensitivity, or workload requires them. The common element is the way agents request and receive evidence. Several specialized knowledge bases can sit behind that interface; forcing unrelated material into one index is not the goal.

Connect to evidence, not another chatbot

Microsoft offers one implementation through a Model Context Protocol, or MCP, endpoint that compatible agents can call. Its retrieval API can return extracted content, references, and retrieval activity. The MCP response uses a different envelope and does not return those separate activity and references arrays. Teams should inspect the actual interface rather than assume every connection exposes the same information. [3] (opens in new tab)

For a reusable service, we would make the evidence output explicit. A request should identify the information needed, relevant task context, and permitted source scope, with the caller's identity established through authentication. It should also express practical limits, such as how long the application can wait and how much material it can accept.

The response should carry selected content with source identifiers and available location or version information. It should make incomplete retrieval distinguishable from a successful search, rather than leaving the calling agent to infer success from a plausible-looking paragraph. These are proposed interface requirements, not a claim that every vendor returns every field automatically.

An optional summary can help, but retain access to its supporting material. Otherwise, the application has outsourced evidence selection and interpretation together, with fewer opportunities to inspect either. For our hypothetical support assistant, a cited compatibility passage is more reusable than another chatbot's customer-facing explanation: the onboarding agent can use that passage to construct a different output.

Keep authorization inside the boundary too. A request for a source must not grant access to it. Sharing the service should mean reusing the access mechanism while preserving each caller's permissions.

There is an important availability distinction. Microsoft's stable 2026-04-01 retrieval API supports minimal, extractive retrieval. Model-led query planning and answer synthesis remain preview capabilities in the documentation checked for this article. A generally available knowledge service is not a blanket endorsement of every advanced feature attached to it. [4] (opens in new tab)

Let harder questions earn more search

A shared service should not force every request through the same amount of work. For the onboarding agent, finding an installation command might require one lookup. Establishing a supported upgrade path could require the current installation guide, release notes, and a compatibility document. An initial search might reveal a dependency that changes what to look for next.

AWS's July 23 technical walkthrough describes a service-level approach to this distinction. Its agentic retrieval API plans searches, retrieves evidence, evaluates whether more is needed, and can stop before a configured iteration ceiling. AWS distinguishes that path from its simpler Retrieve API for focused lookups. These are documented mechanisms, not independent proof of better results on a particular team's data. [5] (opens in new tab)

We would translate that pattern into a straightforward default and a bounded investigation path. Begin with ordinary retrieval where the question is well scoped. Allow further searches when an important part remains unresolved: a missing prerequisite, an unexplained exception, or a document reference that needs following.

Make those gaps visible. In the upgrade example, finding a general procedure does not settle whether it supports the customer's starting version. The next search should address that unresolved condition. Repeating similar searches without identifying what is missing gives the team little basis for judging the extra work.

Set limits on search iterations, elapsed time, and model usage. An iteration limit alone is not a complete budget: individual calls can differ in duration and size. When a limit is reached, return the evidence gathered with an appropriate indication of what remains unresolved. The calling application can then provide a qualified response or seek more context.

Decide where that investigation lives. Our preference is for the application to own the overall task and the knowledge service to own its bounded evidence search. Avoid having both independently launch unrestricted follow-up loops for the same missing information. The service should report what it attempted; the application should decide whether the user's task warrants another request.

The benefit is a clearer place to improve retrieval behavior for every consumer. If the team develops a better way to find upgrade prerequisites, both applications can use it. Whether the extra search improves their final outputs still needs to be checked separately.

Build for a second consumer

For a single-purpose assistant, a straightforward retrieval pipeline may be sufficient. A shared service becomes more compelling when a second application needs many of the same sources. Start with that overlap rather than designing a universal knowledge platform before another consumer exists.

A practical first step is to separate the existing retrieval code from the first assistant's prompts and response formatting. Define the request and evidence response, then connect the second application through that boundary. Keep any genuinely application-specific selection rules visible instead of quietly making them defaults for everyone.

This also clarifies ownership. The knowledge-service owner maintains source connections and retrieval behavior. Application teams remain responsible for how their agents use the results. A managed provider can operate infrastructure, but the team still needs to decide which sources belong in scope and what a useful response must contain.

Centralization has a cost: a service change can affect several applications, and a demanding consumer can compete with interactive requests. We would give consumers separate budgets and version the interface when its behavior changes materially. Reuse earns its place when the team can improve the common capability without making every application move in lockstep.

For teams planning their 2027 architecture, that is a useful investment: preserve a stable route to evidence while allowing agents to evolve around it. The support assistant can change its model or conversational experience; onboarding can change its planning workflow. Neither should have to rebuild access to the same guides merely because its job has changed.

The next agent becomes easier to build when it inherits a useful capability, not the previous agent's assumptions.

Sources

Sources checked September 14, 2026. Named capabilities and release dates come from vendor publications, not independent effectiveness studies. The architecture recommendations and two-agent example are editorial analysis and illustration.

1. Microsoft — Foundry IQ: Build smarter agents faster with unified knowledge and serverless retrieval (opens in new tab). June 2, 2026.

2. AWS — Amazon Bedrock Managed Knowledge Base is now generally available (opens in new tab). June 17, 2026.

3. Microsoft Learn — Query a knowledge base via API or MCP (opens in new tab). Technical documentation, checked September 14, 2026.

4. Microsoft Learn — Migrate agentic retrieval code (opens in new tab). Stable-versus-preview guidance, checked September 14, 2026.

5. AWS — Agentic retrieval for Amazon Bedrock Managed Knowledge Base (opens in new tab). July 23, 2026. Used for the documented retrieval mechanism, not benchmark or pricing claims.