Production AI Platform Map
A map of twelve responsibility layers in a production AI platform.
Problem
Without a map, every AI discussion collapses back to model choice. In production, that is too narrow: the real system lives between product scenario, gateway, inference, quality, cost, safety and ownership.
Symptoms
- Different teams solve gateway, evals, guardrails and observability differently.
- The same failure is rediscovered in multiple products.
- Leadership sees AI as a feature queue, not an operating surface.
- Cost, quality and incidents are discussed after launch, not before it.
Mental model
Product scenarios flow down into platform layers. Production feedback flows back up into product, model and ownership decisions.
If a layer is missing from the map, it does not disappear. A product team implements it locally: in service code, manual support or a late-night incident.
Architecture
Product use cases
Scenario intake, user value, risk profile, acceptance criteria.
AI Gateway
Unified API layer for auth, quotas, routing, policy, cost attribution.
Provider strategy
MaaS, OpenRouter-style research loops, self-hosted and hybrid decisions.
Execution-path routing
Choosing the lane: small model, large model, RAG, agent, human review or policy denial.
Inference runtime
LLM, STT, embeddings and reranker serving with latency and throughput budgets.
Caching
Prompt cache, prefix cache, KV-cache and stable request shape.
Model lifecycle
From research to shadow, canary, production, rollback and retire.
Evals and Quality Gate
Datasets, eval suites, regression checks, canary rollout and feedback loops.
Observability
Traces, tokens, TTFT, TPOT, fallback events, safety events and feedback.
Economics / FinOps
Scenario cost, cached tokens, retries, GPU utilization and cost per accepted outcome.
Guardrails / Security
Policies, PII, prompt injection, tool risks and audit trail.
Operations / Ownership
Ownership, SLOs, incidents, capacity planning, runbooks and platform DevEx.
Reading The Map Through A Scenario
The map becomes useful when it starts from a concrete AI scenario, not from a model:
AI scenario
-> data
-> SLA and latency budget
-> workload profile
-> model and execution boundary
-> evals and rollout
-> owner| Scenario | Data | SLA | Decision |
|---|---|---|---|
| AI chat | ordinary product data | real-time response | fast pool, MaaS or self-hosted |
| Call follow-up | may be sensitive | batch or async processing | self-hosted or deferred path |
| Fast experiment | no sensitive data | flexible | MaaS or OpenRouter |
| Bulk processing | internal boundary | not urgent | overnight batch |
| TTFT-critical flow | product-dependent | strict TTFT | dedicated pool |
Metrics
Each layer has its own metric: use-case adoption, gateway coverage, provider reliability, route quality, false_agentic_rate, false_direct_rate, cost_saved_by_router, inference utilization, cache hit rate by route, eval pass rate, trace completeness, cost per accepted outcome, safety event rate and ownership coverage.
Trade-offs
Maps simplify reality. The goal is not to freeze architecture but to make missing responsibilities visible before they become product incidents.
Anti-patterns
- Treating RAG, agents or model serving as isolated projects.
- Drawing a diagram without owners.
- Collapsing observability and economics into one "we will check later" layer.
- Ignoring cost and quality loops until after launch.
Checklist
- ✓Every layer has an owner or explicit gap.
- ✓Every product scenario enters through a documented contract.
- ✓Model and prompt lifecycle are visible on the map.
- ✓Cost, quality and safety loops are first-class layers.
- ✓STT, embeddings and rerankers are not lost behind the LLM layer.
- ✓Operations and incident response are not hidden in infra.
Example
An agent loop suddenly becomes expensive. The map prevents blaming only the model. Check route, retries, tool schema stability, cached tokens, eval pass rate, trace fields and budget policy. Often the root cause is not a bad model, but a broken request shape or fallback path.
Decision template
Before building a new scenario, mark which platform layers it touches, who owns release risk, which fields enter telemetry, where fallback lives and who accepts the cost/quality trade-off.