AI Architect
You own the technical shape of systems other people must trust with real decisions. The model is the least stable part of your architecture; design for that. This playbook is anonymised from building document intelligence and knowledge systems in regulated and engineering contexts.
The job in one sentence
Design AI systems whose answers can be checked, whose failures are visible, and whose models can be swapped without renegotiating trust.
Architecture principles
- Evidence first. Every answer carries its source. Citations and provenance are not features; they are the architecture's contract with its users.
- Abstain by default. When the corpus cannot answer, the system says so plainly. A confident wrong answer costs more trust than a hundred honest abstentions.
- Human gates are components, not process bolted on afterwards. Design the review queue, its interface and its capacity into the system the way you design the database.
- Pin to workflows, not models. Models change monthly; the user's workflow does not. The model sits behind an interface you can swap; the workflow contract is what you architect.
- Audit trail from day one. Who asked, what was retrieved, what was answered, who reviewed. In regulated contexts this is the difference between a pilot and a product.
The reference shape
- One gateway in front of all model access: routing, logging, cost control and access tiers in one place instead of scattered API keys.
- Retrieval with citations over approved sources, with source boundaries enforced at retrieval time, not in the prompt.
- An eval harness in the build pipeline: golden question sets, retrieval precision checks, regression runs on every model or prompt change.
- Structured outputs with schemas, confidence and provenance metadata, so downstream systems consume results without parsing prose.
- The review workbench: where human judgement is applied, captured and fed back as training signal for the next iteration. Staffing this gate is a delivery problem: see the delivery playbook.
Buy by default, build the differentiator
Buy or partner for anything the market does well. Build only where data sensitivity or genuine differentiation demands it, and know which of the two is your reason before writing code. The corollary is honest scoping: most organisations need governed access to good tools far sooner than they need a custom model. The executive version of this call is the three-question test in the CTO playbook.
The model is an implementation detail. The evidence chain is the product.
Evals are the contract
- Acceptance criteria live in the eval set, agreed with the workflow owner before build. "It feels better" is not a merge criterion.
- Every model swap runs the full set. That turns vendor churn from a risk conversation into a regression report.
- Track precision at coverage: what fraction of questions the system attempts, and how often attempted answers are right. Raising coverage while holding precision is the real progress curve.
Security posture, stated honestly
- Source boundaries enforced outside the prompt: retrieval scope, not instructions, decides what the model can see.
- Treat retrieved and user content as untrusted input; prompt injection is a data-flow problem, not a prompt-wording problem.
- Agents get least-privilege tool access with logged calls, and human gates in front of anything irreversible.
- Say what has not been reviewed. Nothing is "secure by default" without a security review, including this list.
The metrics that matter
| Leading (predicts trust) | Lagging (proves trust) | |
|---|---|---|
| Quality | Eval pass rate per change, retrieval precision on the golden set | Citation coverage sustained in production, incidents from wrong answers |
| Operations | Cost per query, review-queue depth, gateway error rates | Model swaps completed without user-visible change, audit completeness |
What not to claim
- No accuracy numbers without the eval set that produced them.
- No "production ready" before deployment validation, and no "secure by default" without a security review.
- Precision claims state their coverage. High precision on the 10% of questions attempted is a different claim from high precision overall.
Questions to ask before you start
- What decision will someone make with this answer, and what does a wrong answer cost?
- Which sources are approved, and who owns keeping that list true?
- Who staffs the review gate, and what happens when the queue grows?
- What is the exit path from every vendor in the critical chain?
The patterns are public: AskLondon (cited answers, honest abstention) and Engineering the Box (the appliance architecture).
Designing an AI system that has to be trusted? Start with the advisory services or email hello@krishnachodipilli.com.