How to Evaluate Company Brain Vendors: 12 Questions Every Enterprise Should Ask

Most vendors in the AI-knowledge category can demo a chatbot. Few can prove retrieval quality on the buyer's own corpus, permission handling at the chunk level, freshness against live source systems, governance against real audit requirements, and operational ownership after go-live. That is the evaluation gap that produces most failed enterprise AI deployments. This guide is the 12-question buyer's scorecard Sphere uses with enterprises evaluating Company Brain platforms, grouped by the four areas where vendors most often fall short: retrieval architecture, security and governance, operational evaluation, and post-launch partnership.

What questions should buyers ask Company Brain vendors?

Four categories, three questions each. Twelve total. The right vendor produces specific, evidenced answers to all twelve; a vendor that produces only generic answers to most of them is selling a chatbot, not a Company Brain.

Retrieval architecture (questions 1–3)

1. Can the vendor describe the retrieval architecture in five layers, and demonstrate each one? Connectors, indexing, retrieval, response composition, governance. A vendor that cannot account for each layer is operating on a partial architecture, regardless of how impressive the demo looks. For the long-form description of the five-layer pattern see how a Company Brain works.

2. Is retrieval hybrid (semantic + lexical) with optional domain filtering, or vector-only? Vector-only retrieval has known failure modes on exact-term matches that matter in regulated, financial, and legal contexts. Hybrid retrieval with reciprocal-rank fusion and optional domain filters before re-ranking is the production pattern. Sphere's Enterprise RAG engagement at US Tax Services AG used domain-optimized chunking, hybrid vector-plus-keyword retrieval, and jurisdiction-aware metadata filtering. Result: a 66% retrieval-accuracy improvement over the firm's previous keyword search, and representative research down from six hours to under seven minutes.

3. What is the vendor's evidence on retrieval accuracy against domain-specific ground truth? Not a generic benchmark. The vendor's prior deployments calibrated against veteran-verified or regulator-anchored answer sets — and the recalibration cadence after launch. Without an evaluation harness against domain ground truth, accuracy claims are unfalsifiable.

Security and governance (questions 4–6)

4. How are permissions enforced — at retrieval time, or after the response is composed? This is the load-bearing security property. Permissions enforced at the retrieval layer mean the response model never sees content the asking user cannot access. Permissions applied after composition mean restricted content has already passed through the model, which is a categorically weaker posture and a non-starter in regulated industries.

5. What is in the audit log, per query? The vendor should produce: the query as asked, the retrieved candidate set with permission-check results, the composed answer, the cited sources with version identifiers, and the model and retrieval configuration in effect. Sphere's AI staffing optimization engagement for a multi-store retail client shipped an approvals inbox against which a single saved query returns every flag, every recommendation, every approval, every notification — signed and dated. That is the operational shape an enterprise audit trail should take.

6. What is the pre-production red-team protocol? Sphere runs a 50-query adversarial evaluation before any deployment goes live — hallucination, permission-boundary violation, prompt injection. Any failure blocks production launch. Vendors that do not run a structured red-team are taking the risk into production rather than out of it.

Operational evaluation (questions 7–9)

7. What freshness guarantees does the vendor offer, and how are they implemented? Connectors re-index on a schedule (typically hourly for high-velocity sources like Slack and Teams; daily for slower sources like SharePoint and NetSuite). When a source document is deleted, the corresponding index entries are invalidated, not orphaned. The system's notion of "current" is anchored to the source systems — not to the deployment snapshot.

8. What is the deployment timeline to a measurable success contract? Sphere's PDE™ (Precision-Driven Engineering) delivery pattern targets 45–90 days to production for a mid-market deployment, with a 20-day path for a single-system pilot. Vendors quoting nine-to-eighteen-month timelines are either rebuilding infrastructure the enterprise already has or running an undisciplined engagement.

9. What is the continuous-evaluation cadence after launch? The vendor should describe a recurring evaluation against the veteran-verified ground truth set, the cadence (typically monthly), and the action path when accuracy drifts. Without continuous evaluation, the system ages out within a quarter and the buyer discovers it the hard way.

Post-launch partnership (questions 10–12)

10. Who owns the deployment after go-live — the vendor, the buyer, or a defined joint operating model? The strongest pattern is functional ownership at the buyer (the VP whose operating outcomes the Company Brain most directly affects) with a governance partnership (CDO or CISO), and a defined operating team of two-to-four people responsible for source-system connector health, evaluation re-runs, and access-review cadence. The vendor's role becomes platform support and engagement expansion.

11. What is the engagement model for adding the next domain? A Company Brain is rarely one deployment. The vendor should describe the operating model for expanding into the second and third domain — and the cost trajectory should reflect that the connectors, the permission model, and the governance discipline are already in place from the first deployment.

12. What is the vendor's track record on incident response and operational continuity? Sphere's multinational NOC engagement — fully automated incident intake, classification, and assignment, with AI-driven routing to the right responder, network topology tracking, and an integrated knowledge base — cut time to resolution by the target 50% in the pilot NOC center. That is the operating shape of a vendor that has been on the other side of a high-stakes operational deployment. Sphere's Smart Building Operations engagement — an eleven-week engagement that held platform development on track after the client's CTO departed — is the equivalent on the continuity side. Vendors with no comparable operating evidence are demoing a system; vendors with it are shipping one.

How do you evaluate retrieval architecture?

The fastest test, and the one most buyers underweight, is to ask the vendor to run retrieval against the buyer's own corpus during the evaluation cycle. A focused proof of concept on a single source system can produce evidence inside a week — not a generic demo against a generic dataset, but quantified accuracy on a defined question set drawn from the buyer's actual operating reality.

The three sub-tests:

• Retrieval accuracy on a 20–50 question veteran-verified set drawn from the buyer's domain. Pre-deployment baseline measured by the current process; vendor's deployment measured against the same set.

• Cross-system reach. Does the retrieval span the source systems where the answer actually lives, or does it require a single repository?

• Permission boundary integrity. A query from a user with restricted access should not surface content from outside the access boundary, in any form, at any stage of the pipeline.

A vendor that resists this evaluation pattern is signaling its confidence level. Sphere offers it as part of the standard PDE™ scoping cycle.

What security and governance requirements matter?

Three layers, in priority order.

Document-level permission inheritance from source systems. The buyer's existing access boundaries — SharePoint permissions, Salesforce sharing rules, NetSuite roles, Confluence restrictions — carry through to the retrieval layer unchanged. The Company Brain inherits the model; it does not invent one.

Full audit log on every query. Tamper-evident, queryable by the buyer's internal audit team, and structured so external examiners can reconstruct the institutional reasoning the AI produced on a given day.

Pre-production red-teaming and continuous post-launch evaluation. The system is calibrated against domain ground truth before launch and re-calibrated continuously after. The vendor should describe both protocols specifically; vague answers here are a red flag.

How should ownership work after go-live?

The right ownership model is a defined joint operating arrangement that puts functional ownership inside the enterprise and platform partnership with the vendor. Three operating roles need to be staffed.

• Business sponsor. The VP whose operating outcomes the Company Brain most directly affects. Owns the success contract, the evaluation cadence at the business level, and the decision to expand to the next domain.

• Governance partner. Chief Data Officer or Chief Information Security Officer. Owns the access-review cadence, the audit-log review, and the regulatory posture.

• Operating team. Two-to-four people responsible for source-system connector health, evaluation re-runs against the veteran-verified ground-truth set, and the day-to-day incident response on the platform itself.

The vendor's role becomes platform support, expansion engagement, and continuous evaluation methodology. A vendor proposing to own the deployment indefinitely is selling a managed service that becomes harder to exit over time. A vendor proposing to partner the deployment with a clear handoff to the enterprise's operating team is selling a platform.

The scorecard as a decision instrument

The twelve questions are not a formality. Each is a place where vendors in this category differ measurably, and where a confidently wrong vendor selection produces a deployment that has to be replaced inside eighteen months. Sphere produces specific, evidenced answers to each — backed by SphereIQ KnowledgeAI™, Engram persistent memory, and PDE™ delivery — and the underlying operating record is in the case studies referenced throughout this guide.

Get Sphere's Company Brain Evaluation Scorecard. Read the Company Brain guide, revisit how to build a Company Brain, or reach a Sphere engineer at sphereinc.com/contact.