Building the unified control plane for LLM inference, routing, metering, safety, and governance at enterprise scale
Table of Contents
- Introduction
- AI Gateway Concepts
- Functional & Non-Functional Requirements
- Capacity Estimation & Back-of-Envelope Math
- Data Model & Storage Schema
- High-Level Architecture
- API Design & Protocol Layer
- Model Registry & Versioning
- Inference Routing & Load Balancing
- API Key Management & Authentication
- Usage Metering & Billing
- Constitutional AI Safety Layer
- Prompt Caching & Optimization
- Streaming Response Handling
- Multi-Model Orchestration
- Guardrails & Output Filtering
- Developer Portal & Documentation
- Enterprise Governance & Compliance
- Audit Logging & Observability
- Cost Optimization Strategies
- Monitoring, Alerting & SLOs
- Testing, Load Testing & Chaos Engineering
- Interview Q&A
1. Introduction
Large Language Models have transitioned from research curiosities to mission-critical infrastructure powering customer support, code generation, data analysis, creative production, and autonomous agents across every industry. As organizations adopt multiple LLM providers including Anthropic Claude, OpenAI GPT, Google Gemini, Mistral, Meta Llama, and open-source alternatives like Llama and Qwen they face an increasingly complex challenge: how do you manage, secure, route, meter, and govern all these model interactions through a single unified control plane?
Anthropic has built one of the most sophisticated AI gateway platforms in the industry. Their API handles billions of inference requests daily, enforcing safety policies, managing API keys for hundreds of thousands of developers, routing traffic across multiple model variants, and providing granular usage analytics. Building a similar platform requires deep expertise in distributed systems, API gateway design, security engineering, and real-time data processing.
In this comprehensive guide, we will design a complete Anthropic-style AI Gateway and Model Management Platform from the ground up. This is not a surface-level overview. We will dive deep into every component from the data model and API protocol to Constitutional AI safety layers, prompt caching, streaming response handling, multi-model orchestration, guardrails, enterprise governance, and cost optimization. Each section includes production-grade C# code, detailed architecture diagrams, capacity estimation formulas, and interview-ready answers.
Whether you are building an internal AI platform for your organization, creating a commercial API gateway product, or preparing for a senior staff-level systems design interview at a top AI company, this guide will give you the complete blueprint. The patterns and techniques described here are drawn from real-world production systems that serve millions of requests per second with five-nines uptime requirements.
Why Build an AI Gateway?
An AI gateway serves as the single entry point for all LLM interactions within an organization or across a developer platform. Without a gateway, teams scatter their API keys across codebases, cannot enforce consistent safety policies, have no unified billing or metering, and cannot optimize costs by routing to cheaper models when quality is comparable. The gateway pattern transforms chaos into controlled, observable, and governable infrastructure.
The business value is enormous. Companies that implement an AI gateway typically see thirty to fifty percent reduction in inference costs through intelligent routing and caching, complete elimination of safety incidents through guardrails, and full regulatory compliance through audit logging. The gateway also accelerates development velocity because teams can swap model providers without changing application code.
This guide covers every aspect of building such a platform, from initial requirements gathering through production deployment and ongoing operations. We will build a system capable of handling ten thousand requests per second in the baseline scenario, scaling to one hundred thousand requests per second at peak, with sub-two-hundred-millisecond median latency for non-streaming requests and first-token latency under one hundred milliseconds for streaming.
2. AI Gateway Concepts
An AI gateway is a specialized API gateway designed specifically for managing interactions with Large Language Model providers. Unlike traditional API gateways that handle generic REST traffic, an AI gateway must understand LLM-specific concepts such as token-level billing, streaming Server-Sent Events responses, prompt and completion pairs, model versioning, and content safety filtering.
Core Responsibilities
- Protocol Translation: Accept requests in a unified API format and translate them to provider-specific protocols. Anthropic uses a Messages API format, OpenAI uses a Chat Completions format, and open-source providers may use raw HTTP or gRPC. The gateway normalizes all of these.
- Authentication & Authorization: Validate API keys, manage key lifecycle, enforce rate limits per key, and map keys to organization-level quotas and budgets.
- Routing & Load Balancing: Select which model and provider to route each request to, based on model availability, latency requirements, cost constraints, and user preferences.
- Usage Metering: Track every request for input tokens, output tokens, total tokens, latency, and cost. This data feeds into billing, analytics, and cost optimization.
- Safety & Guardrails: Filter inputs and outputs for harmful content, enforce content policies, apply Constitutional AI principles, and block jailbreak attempts.
- Caching: Cache identical or semantically similar prompts to reduce redundant inference calls and cut costs.
- Rate Limiting: Enforce per-user, per-organization, and global rate limits to protect provider APIs and ensure fair resource allocation.
- Observability: Provide real-time dashboards, alerting, and detailed logs for every inference request.
How It Differs from Traditional API Gateways
Traditional API gateways like Kong, AWS API Gateway, or Envoy handle HTTP routing, authentication, and rate limiting. An AI gateway adds several critical capabilities that traditional gateways lack. First, it must handle long-running streaming responses that can last thirty seconds or more, requiring different connection management and timeout handling. Second, it must understand token economics where billing is based on the number of tokens processed rather than simple request counts. Third, it must perform content-aware filtering on both inputs and outputs, examining the actual text content for safety violations. Fourth, it must support model-specific features like tool use, system prompts, and temperature parameters that differ across providers.
Gateway Placement Patterns
There are three common patterns for deploying an AI gateway. In the centralized pattern, all traffic flows through a single gateway deployment managed by a platform team. This provides maximum control and consistency but can become a bottleneck. In the embedded pattern, each application team runs their own gateway instance with shared configuration, providing independence while maintaining standards. In the hybrid pattern, a central gateway handles authentication and routing while regional or application-specific sidecars handle caching and local optimizations. Most enterprise deployments settle on the hybrid pattern as it balances control with performance.
The Anthropic Model
Anthropic's approach to their AI gateway is notable for several design decisions. They treat the API as a first-class product with developer experience as a top priority. Their API design is clean and predictable, with consistent patterns across all endpoints. They invest heavily in safety, with Constitutional AI baked into the platform rather than bolted on as an afterthought. Their documentation and developer portal are among the best in the industry, providing interactive examples and detailed cookbooks. And their pricing model is transparent, with clear per-token costs for input and output across all model tiers.
3. Functional & Non-Functional Requirements
Functional Requirements
| Feature | Description | Priority |
|---|---|---|
| Unified API Proxy | Accept OpenAI-compatible and Anthropic-compatible request formats, translate to provider-specific APIs | P0 |
| Multi-Provider Support | Support Anthropic Claude, OpenAI, Google Gemini, Mistral, and self-hosted models | P0 |
| Streaming & Non-Streaming | Handle both SSE streaming and synchronous request-response patterns | P0 |
| API Key Management | Create, rotate, revoke keys with scoped permissions and expiration | P0 |
| Model Registry | Register, version, and configure available models with metadata | P0 |
| Usage Metering | Track token usage, latency, and cost per request, per key, per organization | P0 |
| Rate Limiting | Per-key, per-org, and global rate limits with configurable policies | P0 |
| Content Guardrails | Input and output content filtering for harmful, abusive, or policy-violating content | P1 |
| Prompt Caching | Cache identical prompts with configurable TTL and invalidation | P1 |
| Load Balancing | Intelligent routing across model replicas with health checking | P1 |
| Developer Portal | Self-service dashboard for key management, usage analytics, and documentation | P1 |
| Audit Logging | Complete audit trail of all API interactions for compliance | P1 |
| Billing Integration | Usage-based billing with invoice generation and payment tracking | P2 |
| Multi-Model Orchestration | Fallback chains, ensemble voting, and model cascading | P2 |
Non-Functional Requirements
| Requirement | Target | Rationale |
|---|---|---|
| Throughput | 10K rps baseline, 100K rps peak | Support large enterprise workloads and viral application spikes |
| Latency (p50) | < 200ms overhead (non-streaming) | Gateway overhead must not significantly impact end-to-end latency |
| First Token Latency | < 100ms for streaming | Users expect immediate response start for streaming interactions |
| Availability | 99.99% uptime | AI inference is becoming business-critical for many customers |
| Data Durability | 99.999999999% (11 nines) | Usage data drives billing; loss means revenue leakage |
| Security | SOC 2 Type II, HIPAA ready | Enterprise customers require compliance certifications |
| Scalability | Horizontal scaling with zero-downtime deploys | Must handle unpredictable traffic growth |
| Cost Efficiency | Gateway cost < 5% of inference spend | Infrastructure cost must not erode the value proposition |
4. Capacity Estimation & Back-of-Envelope Math
Let us establish the baseline numbers that drive our architecture decisions. These calculations assume a medium-scale deployment serving a mix of enterprise and developer customers.
Request Volume
| Metric | Value | Calculation |
|---|---|---|
| Baseline RPS | 10,000 | Starting target for production deployment |
| Peak RPS | 100,000 | 10x baseline for traffic spikes |
| Daily requests | 864 million | 10K * 86,400 seconds |
| Monthly requests | 25.9 billion | 864M * 30 days |
| % Streaming requests | 60% | Most LLM consumers prefer streaming |
| Avg input tokens | 800 | Mix of short prompts and longer context |
| Avg output tokens | 500 | Typical response length |
Storage & Bandwidth
| Metric | Value | Notes |
|---|---|---|
| Avg request payload | ~4 KB | JSON with messages array and parameters |
| Avg response payload | ~3 KB | Completion text with usage metadata |
| Ingress bandwidth | 40 MB/s baseline | 10K * 4KB |
| Egress bandwidth | 300 MB/s baseline | 10K * 3KB, higher due to streaming overhead |
| Daily metering data | ~260 GB | 864M records * ~300 bytes each |
| Monthly metering data | ~7.8 TB | Before compression and aggregation |
| Cache storage | ~500 GB | Cached prompt-response pairs with metadata |
Compute Requirements
For the gateway compute layer, each request requires authentication validation, rate limit checking, routing decision, payload transformation, and metering emission. Assuming 0.5ms of CPU time per request for these operations, 10,000 RPS requires 5 seconds of CPU time per second, meaning roughly 5 vCPUs dedicated to request processing. Adding overhead for connection handling, TLS termination, and garbage collection, we estimate 8 vCPUs for the baseline gateway tier. For the streaming proxy tier, where connections are held open for the duration of the response, we need additional capacity. Assuming 100 concurrent streaming connections per vCPU, 6,000 concurrent streaming requests (60% of 10K RPS with average 1-second hold time) requires approximately 60 vCPUs for the streaming tier.
Database Sizing
The primary transactional database (PostgreSQL) stores API keys, organizations, model configurations, and routing rules. With 100,000 organizations, 500,000 API keys, and 200 model configurations, the hot working set is approximately 2 GB. The metering time-series data flows into a separate analytics database. With 864 million events per day at 300 bytes each, and assuming thirty-day retention before aggregation, the raw storage requirement is approximately 7.8 TB, which compresses to roughly 2 TB in a columnar store like ClickHouse.
5. Data Model & Storage Schema
The data model is the foundation of the entire platform. A well-designed schema ensures data integrity, enables efficient queries, and supports the complex relationships between organizations, users, API keys, models, and usage records.
Core Entities
SQL
CREATE TABLE organizations (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL,
slug VARCHAR(100) UNIQUE NOT NULL,
plan VARCHAR(50) NOT NULL DEFAULT 'free',
monthly_budget_cents BIGINT DEFAULT 0,
spent_cents BIGINT DEFAULT 0,
rate_limit_rps INT DEFAULT 100,
safety_policy_id UUID REFERENCES safety_policies(id),
settings JSONB DEFAULT '{}',
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
deleted_at TIMESTAMPTZ
);
CREATE TABLE api_keys (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
org_id UUID NOT NULL REFERENCES organizations(id),
key_hash VARCHAR(255) NOT NULL UNIQUE,
key_prefix VARCHAR(12) NOT NULL,
name VARCHAR(255),
scopes TEXT[] DEFAULT ARRAY['inference:read', 'inference:write'],
model_whitelist TEXT[],
rate_limit_rps INT,
budget_cents BIGINT DEFAULT 0,
expires_at TIMESTAMPTZ,
last_used_at TIMESTAMPTZ,
is_revoked BOOLEAN DEFAULT FALSE,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE models (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
provider VARCHAR(50) NOT NULL,
model_id VARCHAR(100) NOT NULL,
display_name VARCHAR(255) NOT NULL,
version VARCHAR(50) NOT NULL,
input_price_per_1k_tokens DECIMAL(10,6) NOT NULL,
output_price_per_1k_tokens DECIMAL(10,6) NOT NULL,
max_context_tokens INT NOT NULL,
max_output_tokens INT NOT NULL,
supports_tools BOOLEAN DEFAULT FALSE,
supports_vision BOOLEAN DEFAULT FALSE,
supports_streaming BOOLEAN DEFAULT TRUE,
capabilities JSONB DEFAULT '{}',
is_active BOOLEAN DEFAULT TRUE,
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(provider, model_id, version)
);
CREATE TABLE routing_rules (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
org_id UUID REFERENCES organizations(id),
priority INT NOT NULL DEFAULT 100,
source_model VARCHAR(100),
target_model VARCHAR(100) NOT NULL,
target_provider VARCHAR(50) NOT NULL,
conditions JSONB DEFAULT '{}',
weight INT DEFAULT 100,
is_active BOOLEAN DEFAULT TRUE,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE usage_records (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
request_id VARCHAR(64) NOT NULL UNIQUE,
org_id UUID NOT NULL,
api_key_id UUID NOT NULL,
model_id VARCHAR(100) NOT NULL,
provider VARCHAR(50) NOT NULL,
input_tokens INT NOT NULL,
output_tokens INT NOT NULL,
total_tokens INT NOT NULL,
latency_ms INT NOT NULL,
time_to_first_token_ms INT,
is_streaming BOOLEAN DEFAULT FALSE,
is_cached BOOLEAN DEFAULT FALSE,
cost_cents INT NOT NULL,
status VARCHAR(20) NOT NULL,
error_code VARCHAR(50),
created_at TIMESTAMPTZ DEFAULT NOW()
) PARTITION BY RANGE (created_at);
CREATE TABLE safety_policies (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL,
rules JSONB NOT NULL DEFAULT '[]',
block_categories TEXT[] DEFAULT ARRAY['hate', 'violence', 'sexual', 'self-harm'],
max_toxicity_score DECIMAL(3,2) DEFAULT 0.80,
enable_prompt_injection_detection BOOLEAN DEFAULT TRUE,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE prompt_cache_entries (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
cache_key VARCHAR(64) NOT NULL UNIQUE,
prompt_hash VARCHAR(64) NOT NULL,
model_id VARCHAR(100) NOT NULL,
response TEXT NOT NULL,
input_tokens INT NOT NULL,
output_tokens INT NOT NULL,
hit_count INT DEFAULT 0,
expires_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE audit_logs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
org_id UUID NOT NULL,
actor_id UUID NOT NULL,
actor_type VARCHAR(20) NOT NULL,
action VARCHAR(100) NOT NULL,
resource_type VARCHAR(50) NOT NULL,
resource_id UUID,
details JSONB DEFAULT '{}',
ip_address INET,
user_agent TEXT,
created_at TIMESTAMPTZ DEFAULT NOW()
) PARTITION BY RANGE (created_at);Partitioning Strategy
Both usage_records and audit_logs are partitioned by monthly ranges. This enables efficient time-range queries for billing periods and audit investigations while keeping index sizes manageable. A partition maintenance job runs monthly to create future partitions and optionally archive or drop old ones based on retention policy.
SQL
CREATE TABLE usage_records_2026_07 PARTITION OF usage_records
FOR VALUES FROM ('2026-07-01') TO ('2026-08-01');
CREATE TABLE usage_records_2026_08 PARTITION OF usage_records
FOR VALUES FROM ('2026-08-01') TO ('2026-09-01');
CREATE INDEX idx_usage_org_created ON usage_records (org_id, created_at DESC);
CREATE INDEX idx_usage_model ON usage_records (model_id, created_at DESC);
CREATE INDEX idx_usage_api_key ON usage_records (api_key_id, created_at DESC);
CREATE INDEX idx_audit_org_created ON audit_logs (org_id, created_at DESC);
CREATE INDEX idx_audit_action ON audit_logs (action, created_at DESC);Redis Data Structures
Redis stores ephemeral data including rate limit counters, active sessions, and hot cache entries. Rate limits use sliding window counters stored as sorted sets with timestamps. Cache entries use a combination of hash maps for metadata and string values for cached responses. We also maintain a sorted set of model health scores for routing decisions.
C#
public class RateLimitStore
{
private readonly IConnectionMultiplexer _redis;
public async Task<bool> CheckRateLimitAsync(
string key, int maxRequests, TimeSpan window)
{
var db = _redis.GetDatabase();
var now = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
var windowStart = now - (long)window.TotalMilliseconds;
var luaScript = @"
redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, ARGV[1])
local count = redis.call('ZCARD', KEYS[1])
if count < tonumber(ARGV[2]) then
redis.call('ZADD', KEYS[1], ARGV[3], ARGV[3])
redis.call('EXPIRE', KEYS[1], ARGV[4])
return 1
end
return 0";
var result = await db.ScriptEvaluateAsync(
luaScript,
new RedisKey[] { $"ratelimit:{key}" },
new RedisValue[] { windowStart, maxRequests, now, (int)window.TotalSeconds + 1 });
return (long)result == 1;
}
}6. High-Level Architecture
The architecture follows a layered design with clear separation of concerns. Requests flow through several processing stages, each handled by a dedicated service component. This separation enables independent scaling, deployment, and evolution of each component.
Cloudflare / AWS ALB"] TLS["TLS Termination"] end subgraph "Gateway Core" AUTH["Auth Service
Key Validation"] RL["Rate Limiter
Sliding Window"] ROUTER["Request Router
Model Selection"] CACHE["Cache Layer
Prompt Matching"] SAFETY["Safety Filter
Input Guard"] end subgraph "Inference Layer" STREAM["Stream Proxy
SSE Handling"] ORCH["Model Orchestrator
Fallback & Retry"] POOL["Provider Pools
Connection Mgmt"] end subgraph "Provider Backends" ANTH["Anthropic API"] OPENAI["OpenAI API"] GEMINI["Gemini API"] OSS["Self-Hosted
vLLM / TGI"] end subgraph "Post-Processing" OUTGUARD["Output Guard
Content Filter"] METER["Usage Meter
Token Counter"] RESP["Response Transform
Format Normalize"] end subgraph "Data Layer" PG["PostgreSQL
Transactional"] REDIS["Redis Cluster
Cache & Rate Limits"] CLICK["ClickHouse
Analytics"] KAFKA["Kafka
Event Stream"] end SDK --> LB HTTP --> LB WEBHOOK --> LB LB --> TLS TLS --> AUTH AUTH --> RL RL --> CACHE CACHE --> ROUTER ROUTER --> SAFETY SAFETY --> STREAM STREAM --> ORCH ORCH --> POOL POOL --> ANTH POOL --> OPENAI POOL --> GEMINI POOL --> OSS ANTH --> OUTGUARD OPENAI --> OUTGUARD GEMINI --> OUTGUARD OSS --> OUTGUARD OUTGUARD --> METER METER --> RESP RESP --> SDK AUTH --> REDIS RL --> REDIS CACHE --> REDIS METER --> KAFKA KAFKA --> CLICK AUTH --> PG ROUTER --> PG CACHE --> PG
Component Responsibilities
| Component | Responsibility | Scaling Strategy |
|---|---|---|
| Auth Service | API key lookup, validation, scope checking | Stateless horizontal scaling, Redis-backed key cache |
| Rate Limiter | Sliding window rate limiting per key/org | Redis cluster with local fallback counters |
| Cache Layer | Prompt hash lookup, response caching | Redis cluster with L1 local in-memory cache |
| Request Router | Model selection based on rules and conditions | Stateless, scales with request volume |
| Safety Filter | Input content classification and blocking | Dedicated GPU nodes for classifier inference |
| Stream Proxy | Hold connections for SSE streaming responses | Connection-aware scaling, higher instance count |
| Model Orchestrator | Retry, fallback, circuit breaking | Stateless with provider connection pools |
| Output Guard | Post-generation content filtering | Scales with output volume |
| Usage Meter | Token counting, cost calculation, event emission | Async via Kafka, scales independently |
Request Lifecycle
When a client sends an inference request, the following sequence occurs within the gateway. The load balancer terminates TLS and forwards the request to an available gateway instance. The Auth Service extracts the API key from the Authorization header, looks up the key in Redis (with PostgreSQL as the source of truth), and validates it is not revoked and has not expired. The Rate Limiter checks whether the key has exceeded its rate limit using a sliding window algorithm backed by Redis sorted sets. If the key has budget remaining, the Cache Layer computes a hash of the prompt and model combination and checks for a cached response. On a cache hit, the cached response is returned immediately, bypassing the upstream provider entirely. On a cache miss, the Request Router evaluates routing rules to select the target model and provider, considering model whitelist constraints, org preferences, and provider health scores. The Safety Filter scans the input content for policy violations before forwarding the request upstream. The Stream Proxy establishes a connection to the selected provider and forwards the transformed request. As response tokens arrive, they are streamed back to the client in real time while simultaneously being buffered for the Output Guard. The Output Guard reviews the complete response for policy compliance. The Usage Meter records the request details including token counts, latency, and cost. Finally, the Response Transformer normalizes the output format and returns it to the client.
7. API Design & Protocol Layer
The API surface must be intuitive for developers while providing enough power for complex use cases. We support both Anthropic-compatible and OpenAI-compatible request formats, enabling seamless migration between providers.
Unified Inference Endpoint
C#
[ApiController]
[Route("v1")]
public class InferenceController : ControllerBase
{
private readonly IGatewayOrchestrator _orchestrator;
private readonly IAuthValidator _authValidator;
private readonly IUsageMeter _usageMeter;
private readonly ISafetyFilter _safetyFilter;
private readonly ILogger<InferenceController> _logger;
[HttpPost("messages")]
[HttpPost("chat/completions")]
public async Task CreateMessage(
[FromBody] InferenceRequest request,
[FromHeader(Name = "Authorization")] string authHeader,
CancellationToken cancellationToken)
{
var apiKey = await _authValidator.ValidateAsync(authHeader);
if (apiKey == null)
{
Response.StatusCode = 401;
await Response.WriteAsJsonAsync(new ErrorResponse
{
Error = new ErrorDetail
{
Type = "authentication_error",
Message = "Invalid API key provided"
}
}, cancellationToken);
return;
}
var requestId = Guid.NewGuid().ToString("N");
var startTime = Stopwatch.StartNew();
try
{
var safetyResult = await _safetyFilter.CheckInputAsync(
request, apiKey.OrgId);
if (safetyResult.IsBlocked)
{
Response.StatusCode = 400;
await Response.WriteAsJsonAsync(new ErrorResponse
{
Error = new ErrorDetail
{
Type = "content_policy_violation",
Message = safetyResult.Reason
}
}, cancellationToken);
return;
}
var cacheKey = PromptCache.ComputeKey(request);
var cached = await _orchestrator.TryGetCachedResponseAsync(cacheKey);
if (cached != null)
{
await _usageMeter.RecordAsync(new UsageRecord
{
RequestId = requestId,
OrgId = apiKey.OrgId,
ApiKeyId = apiKey.Id,
Model = request.Model,
InputTokens = cached.InputTokens,
OutputTokens = cached.OutputTokens,
LatencyMs = 0,
IsCached = true,
CostCents = 0
});
await Response.WriteAsJsonAsync(cached.Response, cancellationToken);
return;
}
if (request.Stream == true)
{
Response.ContentType = "text/event-stream";
Response.Headers.Add("Cache-Control", "no-cache");
Response.Headers.Add("X-Request-Id", requestId);
await foreach (var chunk in _orchestrator.StreamAsync(
request, apiKey, requestId, cancellationToken))
{
await Response.WriteAsync(
$"data: {JsonSerializer.Serialize(chunk)}\n\n",
cancellationToken);
}
await Response.WriteAsync("data: [DONE]\n\n", cancellationToken);
}
else
{
var result = await _orchestrator.CompleteAsync(
request, apiKey, requestId, cancellationToken);
await _usageMeter.RecordAsync(new UsageRecord
{
RequestId = requestId,
OrgId = apiKey.OrgId,
ApiKeyId = apiKey.Id,
Model = result.Model,
InputTokens = result.Usage.InputTokens,
OutputTokens = result.Usage.OutputTokens,
LatencyMs = (int)startTime.ElapsedMilliseconds,
IsCached = false,
CostCents = CalculateCost(result)
});
await Response.WriteAsJsonAsync(result, cancellationToken);
}
}
catch (ProviderException ex)
{
_logger.LogError(ex, "Provider error for request {RequestId}", requestId);
Response.StatusCode = 502;
await Response.WriteAsJsonAsync(new ErrorResponse
{
Error = new ErrorDetail
{
Type = "upstream_error",
Message = "The model provider returned an error"
}
}, cancellationToken);
}
catch (RateLimitExceededException)
{
Response.StatusCode = 429;
Response.Headers.Add("Retry-After", "30");
await Response.WriteAsJsonAsync(new ErrorResponse
{
Error = new ErrorDetail
{
Type = "rate_limit_error",
Message = "Rate limit exceeded. Please retry after 30 seconds"
}
}, cancellationToken);
}
}
}API Endpoints Summary
| Method | Path | Description |
|---|---|---|
| POST | /v1/messages | Create a message (Anthropic format) |
| POST | /v1/chat/completions | Create a chat completion (OpenAI format) |
| GET | /v1/models | List available models |
| GET | /v1/models/{model} | Get model details |
| GET | /v1/usage | Query usage statistics |
| GET | /v1/usage/summary | Aggregated usage summary |
| POST | /v1/keys | Create a new API key |
| GET | /v1/keys | List API keys |
| DELETE | /v1/keys/{keyId} | Revoke an API key |
| PATCH | /v1/keys/{keyId} | Update API key settings |
| GET | /v1/audit | Query audit logs |
| GET | /v1/health | Health check endpoint |
Error Response Format
All errors follow a consistent JSON structure that mirrors the Anthropic API error format. This consistency makes it easy for developers to handle errors across different providers without learning new patterns. Every error includes a machine-readable type, a human-readable message, and optionally additional context like retry-after headers for rate limit errors.
8. Model Registry & Versioning
The model registry is the central catalog of all AI models available through the gateway. It stores metadata about each model including pricing, capabilities, rate limits, and version information. The registry is critical for routing decisions, billing calculations, and developer portal display.
Model Registration Service
C#
public class ModelRegistryService
{
private readonly IDbConnection _db;
private readonly IDatabase _redis;
private const string CachePrefix = "model:registry:";
public async Task<ModelInfo> GetModelAsync(string provider, string modelId)
{
var cacheKey = $"{CachePrefix}{provider}:{modelId}";
var cached = await _redis.StringGetAsync(cacheKey);
if (cached.HasValue)
return JsonSerializer.Deserialize<ModelInfo>(cached!);
var model = await _db.QueryFirstOrDefaultAsync<ModelInfo>(
@"SELECT id, provider, model_id as ModelId, display_name as DisplayName,
version, input_price_per_1k_tokens as InputPricePer1kTokens,
output_price_per_1k_tokens as OutputPricePer1kTokens,
max_context_tokens as MaxContextTokens,
max_output_tokens as MaxOutputTokens,
supports_tools as SupportsTools,
supports_vision as SupportsVision,
supports_streaming as SupportsStreaming,
capabilities, is_active as IsActive
FROM models
WHERE provider = @Provider AND model_id = @ModelId AND is_active = true
ORDER BY version DESC LIMIT 1",
new { Provider = provider, ModelId = modelId });
if (model != null)
{
await _redis.StringSetAsync(
cacheKey,
JsonSerializer.Serialize(model),
TimeSpan.FromMinutes(5));
}
return model;
}
public async Task<IEnumerable<ModelInfo>> ListModelsAsync(string orgId)
{
var org = await GetOrganizationAsync(orgId);
var allModels = await _db.QueryAsync<ModelInfo>(
@"SELECT id, provider, model_id as ModelId, display_name as DisplayName,
version, input_price_per_1k_tokens as InputPricePer1kTokens,
output_price_per_1k_tokens as OutputPricePer1kTokens,
max_context_tokens as MaxContextTokens,
max_output_tokens as MaxOutputTokens,
supports_tools as SupportsTools,
supports_vision as SupportsVision,
supports_streaming as SupportsStreaming,
capabilities, is_active as IsActive
FROM models
WHERE is_active = true
ORDER BY provider, model_id");
if (org.ModelWhitelist != null && org.ModelWhitelist.Length > 0)
{
allModels = allModels.Where(m =>
org.ModelWhitelist.Contains($"{m.Provider}/{m.ModelId}"));
}
return allModels;
}
public async Task<ModelInfo> RegisterModelAsync(ModelRegistrationRequest request)
{
var model = await _db.QueryFirstOrDefaultAsync<ModelInfo>(
@"INSERT INTO models (provider, model_id, display_name, version,
input_price_per_1k_tokens, output_price_per_1k_tokens,
max_context_tokens, max_output_tokens, supports_tools,
supports_vision, supports_streaming, capabilities)
VALUES (@Provider, @ModelId, @DisplayName, @Version,
@InputPrice, @OutputPrice, @MaxContext, @MaxOutput,
@SupportsTools, @SupportsVision, @SupportsStreaming, @Capabilities::jsonb)
ON CONFLICT (provider, model_id, version) DO UPDATE SET
input_price_per_1k_tokens = @InputPrice,
output_price_per_1k_tokens = @OutputPrice,
is_active = true
RETURNING *",
request);
await InvalidateModelCacheAsync(request.Provider, request.ModelId);
return model;
}
private async Task InvalidateModelCacheAsync(string provider, string modelId)
{
await _redis.KeyDeleteAsync($"{CachePrefix}{provider}:{modelId}");
}
}Version Management
Model versioning follows semantic versioning principles adapted for AI models. A major version change indicates breaking changes in output format or behavior. A minor version indicates capability additions like new tool support. A patch version indicates internal improvements that do not affect the external interface. The registry maintains all active versions and supports pinning to specific versions or tracking the latest version automatically.
| Version Example | Meaning | Impact |
|---|---|---|
| claude-3-5-sonnet-20241022 | Full model release | New capabilities, potentially different behavior |
| claude-3-5-sonnet-latest | Alias to latest patch | Transparent upgrade, same behavior |
| claude-3-5-sonnet-20241022-v2 | Internal revision | Bug fixes, minor quality improvements |
9. Inference Routing & Load Balancing
Intelligent routing is one of the most valuable features of an AI gateway. Rather than blindly forwarding every request to the same provider, the router makes dynamic decisions based on multiple factors including model availability, cost, latency, and user preferences.
Router Implementation
C#
public class InferenceRouter
{
private readonly ModelRegistryService _registry;
private readonly IHealthChecker _healthChecker;
private readonly IRateLimitService _rateLimiter;
private readonly ILogger<InferenceRouter> _logger;
public async Task<RouteDecision> SelectRouteAsync(
InferenceRequest request, ApiKeyInfo apiKey)
{
var candidates = await GetCandidateRoutesAsync(request, apiKey);
if (!candidates.Any())
{
throw new NoAvailableRouteException(
$"No routes available for model {request.Model}");
}
var scored = candidates.Select(c => new
{
Route = c,
Score = CalculateRouteScore(c, request)
})
.OrderByDescending(x => x.Score)
.ToList();
var selected = scored.First().Route;
_logger.LogInformation(
"Routed request to {Provider}/{Model} with score {Score}",
selected.Provider, selected.ModelId, scored.First().Score);
return new RouteDecision
{
Provider = selected.Provider,
ModelId = selected.ModelId,
Endpoint = selected.Endpoint,
ApiKey = selected.ProviderApiKey,
Timeout = selected.Timeout,
RetryPolicy = selected.RetryPolicy
};
}
private double CalculateRouteScore(
ModelCandidate candidate, InferenceRequest request)
{
double score = 100.0;
var health = _healthChecker.GetHealthScore(
candidate.Provider, candidate.ModelId);
score *= health;
var costFactor = 1.0 - (
candidate.InputPricePer1k / MaxPriceAcrossProviders);
score += costFactor * 20;
var latencyFactor = 1.0 - (
candidate.P50LatencyMs / MaxLatencyAcrossProviders);
score += latencyFactor * 15;
if (request.MaxTokens != null &&
request.MaxTokens > candidate.MaxOutputTokens)
{
score = 0;
}
if (request.Tools != null && !candidate.SupportsTools)
{
score = 0;
}
if (request.Stream == true && !candidate.SupportsStreaming)
{
score = 0;
}
return score;
}
private async Task<IEnumerable<ModelCandidate>> GetCandidateRoutesAsync(
InferenceRequest request, ApiKeyInfo apiKey)
{
var models = await _registry.ListModelsAsync(apiKey.OrgId);
return models.Where(m =>
m.IsActive &&
_healthChecker.IsHealthy(m.Provider, m.ModelId) &&
(apiKey.ModelWhitelist == null ||
apiKey.ModelWhitelist.Contains($"{m.Provider}/{m.ModelId}"))
);
}
}Circuit Breaker Pattern
When a provider experiences failures, the circuit breaker prevents cascading problems by temporarily removing the provider from routing consideration. The circuit breaker tracks the failure rate over a rolling window and transitions between closed (healthy), open (failing), and half-open (testing recovery) states.
C#
public class ProviderCircuitBreaker
{
private readonly ConcurrentDictionary<string, CircuitState> _circuits = new();
private readonly int _failureThreshold = 5;
private readonly TimeSpan _recoveryWindow = TimeSpan.FromSeconds(30);
private readonly TimeSpan _halfOpenDuration = TimeSpan.FromSeconds(10);
public CircuitState GetState(string provider, string modelId)
{
var key = $"{provider}:{modelId}";
if (!_circuits.TryGetValue(key, out var state))
return CircuitState.Closed;
if (state.Status == CircuitStatus.Open &&
DateTime.UtcNow - state.OpenedAt > _recoveryWindow)
{
state.Status = CircuitStatus.HalfOpen;
state.HalfOpenedAt = DateTime.UtcNow;
}
if (state.Status == CircuitStatus.HalfOpen &&
DateTime.UtcNow - state.HalfOpenedAt > _halfOpenDuration)
{
state.Status = CircuitStatus.Closed;
state.FailureCount = 0;
}
return state;
}
public void RecordFailure(string provider, string modelId)
{
var key = $"{provider}:{modelId}";
var state = _circuits.GetOrAdd(key, _ => new CircuitState());
lock (state)
{
state.FailureCount++;
state.LastFailureAt = DateTime.UtcNow;
if (state.FailureCount >= _failureThreshold &&
state.Status != CircuitStatus.Open)
{
state.Status = CircuitStatus.Open;
state.OpenedAt = DateTime.UtcNow;
}
}
}
public void RecordSuccess(string provider, string modelId)
{
var key = $"{provider}:{modelId}";
if (_circuits.TryGetValue(key, out var state))
{
lock (state)
{
state.FailureCount = Math.Max(0, state.FailureCount - 1);
}
}
}
}10. API Key Management & Authentication
API keys are the primary authentication mechanism. Keys must be secure, manageable, and support fine-grained permissions. The system never stores raw API keys; instead it stores salted hashes and maintains a prefix for key identification.
Key Lifecycle
C#
public class ApiKeyService
{
private readonly IDbConnection _db;
private readonly IDatabase _redis;
private const string KeyCachePrefix = "apikey:";
public async Task<ApiKeyCreateResult> CreateKeyAsync(
CreateApiKeyRequest request, string actorId)
{
var rawKey = GenerateSecureKey();
var keyHash = HashKey(rawKey);
var keyPrefix = rawKey.Substring(0, 10);
var keyRecord = await _db.ExecuteAsync(
@"INSERT INTO api_keys
(org_id, key_hash, key_prefix, name, scopes,
model_whitelist, rate_limit_rps, budget_cents, expires_at)
VALUES (@OrgId, @KeyHash, @KeyPrefix, @Name, @Scopes,
@ModelWhitelist, @RateLimitRps, @BudgetCents, @ExpiresAt)
RETURNING id",
new
{
request.OrgId,
KeyHash = keyHash,
KeyPrefix = keyPrefix,
request.Name,
Scopes = request.Scopes,
ModelWhitelist = request.ModelWhitelist,
RateLimitRps = request.RateLimitRps,
BudgetCents = request.BudgetCents,
ExpiresAt = request.ExpiresAt
});
await _redis.HashSetAsync(
$"{KeyCachePrefix}{keyHash}",
new HashEntry[]
{
new("org_id", request.OrgId),
new("scopes", JsonSerializer.Serialize(request.Scopes)),
new("rate_limit_rps", request.RateLimitRps?.ToString() ?? ""),
new("budget_cents", request.BudgetCents?.ToString() ?? "")
});
return new ApiKeyCreateResult
{
Id = keyRecord,
RawKey = rawKey,
KeyPrefix = keyPrefix,
Warning = "Store this key securely. It will not be shown again."
};
}
public async Task<ApiKeyInfo> ValidateKeyAsync(string authHeader)
{
var rawKey = ExtractKeyFromHeader(authHeader);
if (string.IsNullOrEmpty(rawKey)) return null;
var keyHash = HashKey(rawKey);
var cached = await _redis.HashGetAllAsync($"{KeyCachePrefix}{keyHash}");
if (!cached.Any()) return null;
var orgId = cached.First(e => e.Name == "org_id").Value;
var isRevoked = await _redis.KeyExistsAsync(
$"{KeyCachePrefix}{keyHash}:revoked");
if (isRevoked) return null;
var keyRecord = await _db.QueryFirstOrDefaultAsync<ApiKeyInfo>(
@"SELECT id, org_id as OrgId, key_prefix as KeyPrefix,
scopes, model_whitelist as ModelWhitelist,
rate_limit_rps as RateLimitRps, budget_cents as BudgetCents,
expires_at as ExpiresAt
FROM api_keys
WHERE key_hash = @KeyHash AND is_revoked = false",
new { KeyHash = keyHash });
if (keyRecord == null) return null;
if (keyRecord.ExpiresAt.HasValue &&
keyRecord.ExpiresAt.Value < DateTime.UtcNow)
return null;
await _db.ExecuteAsync(
"UPDATE api_keys SET last_used_at = NOW() WHERE id = @Id",
new { keyRecord.Id });
return keyRecord;
}
private string GenerateSecureKey()
{
var bytes = new byte[32];
using var rng = RandomNumberGenerator.Create();
rng.GetBytes(bytes);
return $"sk-ant-{Convert.ToBase64String(bytes)
.Replace("+", "-").Replace("/", "_").TrimEnd('=').Substring(0, 40)}";
}
private string HashKey(string key)
{
using var sha256 = SHA256.Create();
var hash = sha256.ComputeHash(Encoding.UTF8.GetBytes(key));
return Convert.ToBase64String(hash);
}
}Key Scoping & Permissions
| Scope | Description | Example Use |
|---|---|---|
inference:read | Read inference responses | Standard API usage |
inference:write | Create inference requests | Standard API usage |
models:read | List available models | Dashboard display |
usage:read | View usage statistics | Billing integration |
keys:manage | Create and manage API keys | Admin portal |
org:admin | Full organization management | Organization owner |
11. Usage Metering & Billing
Accurate metering is the financial backbone of the platform. Every inference request must be tracked with precise token counts and cost calculations. The metering pipeline must be durable, exactly-once, and capable of handling high throughput without data loss.
Metering Pipeline
C#
public class UsageMeterService : IUsageMeter
{
private readonly IKafkaProducer _kafkaProducer;
private readonly IDatabase _redis;
private readonly ModelRegistryService _registry;
private const string UsageTopic = "usage.records";
private const string RealtimeCounterPrefix = "usage:realtime:";
public async Task RecordAsync(UsageRecord record)
{
var modelInfo = await _registry.GetModelAsync(
record.Provider, record.Model);
record.CostCents = CalculateCost(record, modelInfo);
record.TotalTokens = record.InputTokens + record.OutputTokens;
var envelope = new UsageEnvelope
{
EventId = Guid.NewGuid().ToString("N"),
Timestamp = DateTime.UtcNow,
Record = record
};
await _kafkaProducer.ProduceAsync(UsageTopic,
record.OrgId.ToString(), envelope);
await _redis.HashIncrementAsync(
$"{RealtimeCounterPrefix}{record.OrgId}:tokens",
"total", record.TotalTokens);
await _redis.HashIncrementAsync(
$"{RealtimeCounterPrefix}{record.OrgId}:cost",
"total", record.CostCents);
await _redis.StringIncrementAsync(
$"{RealtimeCounterPrefix}{record.OrgId}:requests");
}
private int CalculateCost(UsageRecord record, ModelInfo model)
{
var inputCost = (record.InputTokens / 1000.0) *
(double)model.InputPricePer1kTokens;
var outputCost = (record.OutputTokens / 1000.0) *
(double)model.OutputPricePer1kTokens;
var totalCost = (inputCost + outputCost) * 100;
if (record.IsCached)
{
totalCost *= 0.1;
}
return (int)Math.Round(totalCost);
}
}
public class UsageAggregationWorker : BackgroundService
{
private readonly IKafkaConsumer _consumer;
private readonly IDbConnection _db;
private readonly BatchBuffer<UsageRecord> _buffer;
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var messages = await _consumer.ConsumeBatchAsync(
UsageTopic, batchSize: 1000,
timeout: TimeSpan.FromSeconds(5),
cancellationToken: stoppingToken);
if (messages.Any())
{
var records = messages.Select(m => m.Record).ToList();
await InsertBatchAsync(records);
await _db.ExecuteAsync(
@"UPDATE organizations
SET spent_cents = spent_cents + @Cost
WHERE id = @OrgId",
records.GroupBy(r => r.OrgId)
.Select(g => new
{
OrgId = g.Key,
Cost = g.Sum(r => r.CostCents)
}));
await _consumer.CommitAsync(messages);
}
}
}
private async Task InsertBatchAsync(List<UsageRecord> records)
{
var table = new DataTable();
table.Columns.Add("request_id", typeof(string));
table.Columns.Add("org_id", typeof(Guid));
table.Columns.Add("api_key_id", typeof(Guid));
table.Columns.Add("model_id", typeof(string));
table.Columns.Add("provider", typeof(string));
table.Columns.Add("input_tokens", typeof(int));
table.Columns.Add("output_tokens", typeof(int));
table.Columns.Add("total_tokens", typeof(int));
table.Columns.Add("latency_ms", typeof(int));
table.Columns.Add("is_streaming", typeof(bool));
table.Columns.Add("is_cached", typeof(bool));
table.Columns.Add("cost_cents", typeof(int));
table.Columns.Add("status", typeof(string));
table.Columns.Add("created_at", typeof(DateTime));
foreach (var r in records)
{
table.Rows.Add(r.RequestId, r.OrgId, r.ApiKeyId,
r.Model, r.Provider, r.InputTokens, r.OutputTokens,
r.TotalTokens, r.LatencyMs, r.IsStreaming,
r.IsCached, r.CostCents, r.Status, r.CreatedAt);
}
using var bulkCopy = new SqlBulkCopy(
(SqlConnection)_db.Connection);
bulkCopy.DestinationTableName = "usage_records";
bulkCopy.BulkCopyTimeout = 30;
await bulkCopy.WriteToServerAsync(table);
}
}Pricing Tiers
| Plan | Rate Limit | Monthly Budget | Models | Support |
|---|---|---|---|---|
| Free | 10 RPS | $5 | Haiku only | Community |
| Pro | 100 RPS | $500 | All models | |
| Team | 500 RPS | $5,000 | All models + priority | Priority |
| Enterprise | Custom | Custom | All + custom fine-tunes | Dedicated |
12. Constitutional AI Safety Layer
Constitutional AI is Anthropic's approach to training AI systems to be helpful, harmless, and honest without relying solely on human feedback. In the context of an AI gateway, the Constitutional AI Safety Layer enforces safety policies on both inputs and outputs, detecting and blocking harmful content before it reaches the model or the user.
Safety Pipeline Implementation
C#
public class ConstitutionalSafetyLayer : ISafetyFilter
{
private readonly IContentClassifier _classifier;
private readonly IPromptInjectionDetector _injectionDetector;
private readonly ISafetyPolicyRepository _policyRepo;
private readonly ILogger<ConstitutionalSafetyLayer> _logger;
public async Task<SafetyResult> CheckInputAsync(
InferenceRequest request, Guid orgId)
{
var policy = await _policyRepo.GetActivePolicyAsync(orgId);
var fullText = ExtractTextContent(request);
var injectionResult = await _injectionDetector.AnalyzeAsync(fullText);
if (injectionResult.Detected)
{
_logger.LogWarning(
"Prompt injection detected for org {OrgId}: {Technique}",
orgId, injectionResult.Technique);
return new SafetyResult
{
IsBlocked = true,
Reason = "Potential prompt injection attack detected",
Category = "prompt_injection",
Confidence = injectionResult.Confidence
};
}
var classification = await _classifier.ClassifyAsync(fullText);
foreach (var category in policy.BlockCategories)
{
var categoryScore = classification.Scores
.FirstOrDefault(s => s.Category == category);
if (categoryScore != null &&
categoryScore.Score > policy.MaxToxicityScore)
{
return new SafetyResult
{
IsBlocked = true,
Reason = $"Content violates policy: {category} " +
$"(score: {categoryScore.Score:F2})",
Category = category,
Confidence = categoryScore.Score
};
}
}
return new SafetyResult { IsBlocked = false };
}
public async Task<SafetyResult> CheckOutputAsync(
string response, Guid orgId)
{
var policy = await _policyRepo.GetActivePolicyAsync(orgId);
var classification = await _classifier.ClassifyAsync(response);
var highestViolation = classification.Scores
.Where(s => policy.BlockCategories.Contains(s.Category))
.OrderByDescending(s => s.Score)
.FirstOrDefault();
if (highestViolation != null &&
highestViolation.Score > policy.MaxToxicityScore)
{
return new SafetyResult
{
IsBlocked = true,
Reason = $"Output violates policy: {highestViolation.Category}",
Category = highestViolation.Category,
Confidence = highestViolation.Score
};
}
var piiResult = DetectPII(response);
if (piiResult.HasPII)
{
return new SafetyResult
{
IsBlocked = true,
Reason = "Output contains personally identifiable information",
Category = "pii_detection",
Confidence = piiResult.Confidence
};
}
return new SafetyResult { IsBlocked = false };
}
}
public class PromptInjectionDetector : IPromptInjectionDetector
{
private readonly IClassifierModel _model;
private static readonly string[] InjectionPatterns = new[]
{
"ignore previous instructions",
"you are now",
"disregard all prior",
"new instructions:",
"system prompt:",
"override your settings",
"act as if you have no restrictions",
"pretend you are",
"jailbreak",
"DAN mode",
"ignore the above",
"forget everything"
};
public async Task<InjectionResult> AnalyzeAsync(string text)
{
var lowerText = text.ToLowerInvariant();
foreach (var pattern in InjectionPatterns)
{
if (lowerText.Contains(pattern))
{
return new InjectionResult
{
Detected = true,
Technique = $"pattern_match:{pattern}",
Confidence = 0.85
};
}
}
var structuralFeatures = ExtractStructuralFeatures(text);
var modelScore = await _model.PredictAsync(structuralFeatures);
if (modelScore > 0.7)
{
return new InjectionResult
{
Detected = true,
Technique = "ml_classifier",
Confidence = modelScore
};
}
return new InjectionResult { Detected = false, Confidence = 0 };
}
}Safety Categories
| Category | Description | Action |
|---|---|---|
| Hate Speech | Content promoting hatred based on protected characteristics | Block |
| Violence | Glorification or promotion of violence | Block |
| Sexual Content | Explicit sexual content or exploitation | Block |
| Self-Harm | Content promoting self-harm or suicide | Block |
| Prompt Injection | Attempts to override system instructions | Block |
| PII Leakage | Unintended exposure of personal information | Flag / Mask |
| Misinformation | Verifiably false claims presented as fact | Warn |
13. Prompt Caching & Optimization
Prompt caching is one of the highest-impact cost optimization strategies for an AI gateway. Studies show that twenty to forty percent of LLM requests in production are semantically identical or highly similar to previous requests. By caching these responses, the gateway can reduce inference costs dramatically while also improving response latency from hundreds of milliseconds to single-digit milliseconds.
Cache Implementation
C#
public class PromptCacheService : IPromptCache
{
private readonly IDatabase _redis;
private readonly IDbConnection _db;
private readonly IContentHasher _hasher;
private readonly TimeSpan _defaultTtl = TimeSpan.FromMinutes(60);
public async Task<CachedResponse> GetAsync(string cacheKey)
{
var cached = await _redis.HashGetAllAsync($"pcache:{cacheKey}");
if (!cached.Any()) return null;
await _redis.HashIncrementAsync(
$"pcache:{cacheKey}", "hits");
return new CachedResponse
{
Response = JsonSerializer.Deserialize<InferenceResponse>(
cached.First(e => e.Name == "response").Value),
InputTokens = int.Parse(
cached.First(e => e.Name == "input_tokens").Value),
OutputTokens = int.Parse(
cached.First(e => e.Name == "output_tokens").Value)
};
}
public async Task SetAsync(
string cacheKey, InferenceResponse response,
int inputTokens, int outputTokens, TimeSpan? ttl = null)
{
var effectiveTtl = ttl ?? _defaultTtl;
var entryKey = $"pcache:{cacheKey}";
await _redis.HashSetAsync(entryKey, new HashEntry[]
{
new("response", JsonSerializer.Serialize(response)),
new("input_tokens", inputTokens),
new("output_tokens", outputTokens),
new("hits", 0),
new("created_at", DateTimeOffset.UtcNow.ToUnixTimeSeconds())
});
await _redis.KeyExpireAsync(entryKey, effectiveTtl);
await _db.ExecuteAsync(
@"INSERT INTO prompt_cache_entries
(cache_key, prompt_hash, model_id, response,
input_tokens, output_tokens, expires_at)
VALUES (@CacheKey, @PromptHash, @ModelId, @Response,
@InputTokens, @OutputTokens, @ExpiresAt)
ON CONFLICT (cache_key) DO UPDATE SET
response = @Response,
hit_count = prompt_cache_entries.hit_count + 1,
expires_at = @ExpiresAt",
new
{
CacheKey = cacheKey,
PromptHash = cacheKey,
ModelId = response.Model,
Response = JsonSerializer.Serialize(response),
InputTokens = inputTokens,
OutputTokens = outputTokens,
ExpiresAt = DateTimeOffset.UtcNow.Add(effectiveTtl)
});
}
public string ComputeKey(InferenceRequest request)
{
var sb = new StringBuilder();
sb.Append(request.Model);
sb.Append('|');
if (request.System != null)
{
sb.Append(request.System.ToString());
sb.Append('|');
}
foreach (var msg in request.Messages)
{
sb.Append(msg.Role);
sb.Append(':');
sb.Append(msg.Content?.ToString()?.Trim());
sb.Append('|');
}
sb.Append($"temp={request.Temperature ?? 1.0}");
sb.Append($"|max={request.MaxTokens ?? 0}");
var bytes = Encoding.UTF8.GetBytes(sb.ToString());
var hash = SHA256.HashData(bytes);
return Convert.ToHexString(hash).ToLowerInvariant();
}
}Cache Strategy Comparison
| Strategy | Hit Rate | Complexity | Staleness Risk |
|---|---|---|---|
| Exact Match (SHA-256) | 20-30% | Low | Medium |
| Semantic Similarity (Embedding) | 40-60% | High | High |
| Prefix Match (Shared System Prompt) | 30-45% | Medium | Low |
| Hybrid (Exact + Prefix) | 50-65% | Medium | Medium |
14. Streaming Response Handling
Streaming is the dominant interaction pattern for LLM applications. Users expect to see tokens appearing in real time rather than waiting for the complete response. The gateway must efficiently proxy Server-Sent Events from upstream providers to downstream clients while maintaining low first-token latency and minimal memory overhead.
Stream Proxy Implementation
C#
public class StreamProxyService
{
private readonly HttpClient _httpClient;
private readonly UsageMeterService _usageMeter;
private readonly OutputGuardService _outputGuard;
private readonly ILogger<StreamProxyService> _logger;
public async IAsyncEnumerable<StreamChunk> ProxyStreamAsync(
RouteDecision route,
InferenceRequest request,
string requestId,
[EnumeratorCancellation] CancellationToken ct)
{
var httpRequest = BuildUpstreamRequest(route, request);
var response = await _httpClient.SendAsync(
httpRequest,
HttpCompletionOption.ResponseHeadersRead,
ct);
response.EnsureSuccessStatusCode();
var stream = await response.Content.ReadAsStreamAsync(ct);
using var reader = new StreamReader(stream);
var buffer = new StringBuilder();
var inputTokens = 0;
var outputTokens = 0;
var firstTokenTime = Stopwatch.StartNew();
var isFirstToken = true;
while (!reader.EndOfStream && !ct.IsCancellationRequested)
{
var line = await reader.ReadLineAsync(ct);
if (string.IsNullOrEmpty(line)) continue;
if (line.StartsWith("data: "))
{
var data = line.Substring(6);
if (data == "[DONE]")
{
yield return new StreamChunk
{
Type = "message_stop",
RequestId = requestId
};
break;
}
var upstreamChunk = JsonSerializer
.Deserialize<UpstreamChunk>(data);
if (isFirstToken)
{
firstTokenTime.Stop();
isFirstToken = false;
}
var gatewayChunk = TransformChunk(
upstreamChunk, requestId);
yield return gatewayChunk;
if (upstreamChunk.Usage != null)
{
inputTokens = upstreamChunk.Usage.InputTokens;
outputTokens = upstreamChunk.Usage.OutputTokens;
}
else
{
outputTokens += CountTokens(
upstreamChunk.Delta?.Text ?? "");
}
buffer.Append(upstreamChunk.Delta?.Text ?? "");
var partialText = buffer.ToString();
var guardResult = await _outputGuard
.CheckPartialAsync(partialText);
if (guardResult.ShouldTerminate)
{
_logger.LogWarning(
"Stream terminated by output guard for {RequestId}",
requestId);
yield return new StreamChunk
{
Type = "content_block_stop",
StopReason = "content_policy",
RequestId = requestId
};
break;
}
}
}
await _usageMeter.RecordAsync(new UsageRecord
{
RequestId = requestId,
OrgId = route.OrgId,
Model = route.ModelId,
Provider = route.Provider,
InputTokens = inputTokens,
OutputTokens = outputTokens,
LatencyMs = (int)firstTokenTime.ElapsedMilliseconds,
TimeToFirstTokenMs = (int)firstTokenTime.ElapsedMilliseconds,
IsStreaming = true,
Status = "success"
});
}
}Streaming Metrics
| Metric | Target | Why It Matters |
|---|---|---|
| Time to First Token (TTFT) | < 100ms gateway overhead | Perceived responsiveness for the user |
| Inter-Token Latency | < 50ms p99 | Smoothness of text appearance |
| Stream Completion Rate | > 99.5% | Reliability of full response delivery |
| Concurrent Stream Capacity | 60K+ connections | Supports peak concurrent streaming load |
| Memory per Stream | < 50KB | Efficient resource utilization |
15. Multi-Model Orchestration
Enterprise AI platforms rarely rely on a single model. Different tasks require different models: simple classification may use a small, fast model while complex reasoning requires a large, expensive model. Multi-model orchestration manages fallback chains, model cascading, ensemble approaches, and automatic quality-based routing.
Orchestration Engine
C#
public class ModelOrchestrator
{
private readonly InferenceRouter _router;
private readonly ProviderCircuitBreaker _circuitBreaker;
private readonly RetryPolicyEngine _retryEngine;
private readonly ModelQualityTracker _qualityTracker;
private readonly ILogger<ModelOrchestrator> _logger;
public async Task<InferenceResponse> CompleteAsync(
InferenceRequest request, ApiKeyInfo apiKey, string requestId)
{
var fallbackChain = BuildFallbackChain(request, apiKey);
foreach (var candidate in fallbackChain)
{
if (!_circuitBreaker.IsAvailable(
candidate.Provider, candidate.ModelId))
{
_logger.LogInformation(
"Skipping {Provider}/{Model} - circuit breaker open",
candidate.Provider, candidate.ModelId);
continue;
}
try
{
var retryPolicy = _retryEngine.GetPolicy(candidate);
var response = await retryPolicy.ExecuteAsync(
async () =>
{
var transformedRequest = TransformRequest(
request, candidate);
var httpClient = GetProviderClient(candidate.Provider);
var upstreamResponse = await httpClient
.PostAsJsonAsync(
candidate.Endpoint,
transformedRequest);
upstreamResponse.EnsureSuccessStatusCode();
return await upstreamResponse.Content
.ReadFromJsonAsync<InferenceResponse>();
});
_circuitBreaker.RecordSuccess(
candidate.Provider, candidate.ModelId);
response.Model = candidate.ModelId;
return response;
}
catch (Exception ex)
{
_circuitBreaker.RecordFailure(
candidate.Provider, candidate.ModelId);
_logger.LogWarning(ex,
"Request {RequestId} failed on {Provider}/{Model}",
requestId, candidate.Provider, candidate.ModelId);
}
}
throw new AllProvidersFailedException(
$"All providers in fallback chain failed for request {requestId}");
}
private List<RouteCandidate> BuildFallbackChain(
InferenceRequest request, ApiKeyInfo apiKey)
{
var preferred = _router.GetPreferredRoute(request, apiKey);
var chain = new List<RouteCandidate> { preferred };
var alternatives = _router.GetAlternativeRoutes(request, apiKey)
.OrderBy(r => r.CostPerToken)
.ThenBy(r => r.AverageLatencyMs)
.ToList();
chain.AddRange(alternatives);
return chain;
}
}Orchestration Patterns
| Pattern | Description | Use Case |
|---|---|---|
| Fallback Chain | Try primary model, fall back to alternatives on failure | High availability requirements |
| Model Cascading | Start with cheap model, escalate to expensive model if quality is insufficient | Cost optimization with quality guarantees |
| Ensemble Voting | Send to multiple models, combine responses | High-stakes decisions requiring consensus |
| Load Splitting | Distribute traffic across models based on weights | A/B testing and gradual rollouts |
| Task-Based Routing | Route to different models based on detected task type | Optimizing for task-specific performance |
16. Guardrails & Output Filtering
Guardrails go beyond content safety filtering. They ensure that model outputs conform to expected formats, contain accurate information, do not hallucinate beyond specified knowledge bases, and comply with organizational policies. The output filtering pipeline runs after the model generates a response and before it reaches the client.
Guardrails Engine
C#
public class GuardrailsEngine
{
private readonly List<IGuardrail> _guardrails;
private readonly IGuardrailMetrics _metrics;
public GuardrailsEngine(IEnumerable<IGuardrail> guardrails)
{
_guardrails = guardrails.OrderBy(g => g.Priority).ToList();
}
public async Task<GuardrailsResult> EvaluateAsync(
InferenceResponse response, GuardrailsContext context)
{
var violations = new List<GuardrailViolation>();
var modifications = new List<OutputModification>();
foreach (var guardrail in _guardrails)
{
if (!guardrail.IsEnabledFor(context)) continue;
var result = await guardrail.EvaluateAsync(response, context);
_metrics.RecordEvaluation(
guardrail.Name, result.Passed, result.DurationMs);
if (!result.Passed)
{
violations.Add(result.Violation);
if (guardrail.Severity == GuardrailSeverity.Critical)
{
return new GuardrailsResult
{
Passed = false,
Violations = violations,
Action = GuardrailAction.Block
};
}
}
if (result.Modifications.Any())
{
modifications.AddRange(result.Modifications);
}
}
if (violations.Any())
{
return new GuardrailsResult
{
Passed = false,
Violations = violations,
Action = GuardrailAction.Warn
};
}
return new GuardrailsResult
{
Passed = true,
ModifiedResponse = ApplyModifications(response, modifications)
};
}
}
public class FormatGuardrail : IGuardrail
{
public string Name => "format_validation";
public int Priority => 10;
public GuardrailSeverity Severity => GuardrailSeverity.Warning;
public async Task<GuardrailEvaluationResult> EvaluateAsync(
InferenceResponse response, GuardrailsContext context)
{
if (context.ExpectedFormat == null)
{
return new GuardrailEvaluationResult { Passed = true };
}
switch (context.ExpectedFormat)
{
case "json":
var isValidJson = IsValidJson(response.Content);
return new GuardrailEvaluationResult
{
Passed = isValidJson,
Violation = !isValidJson
? new GuardrailViolation
{
Rule = "format_json",
Message = "Response is not valid JSON",
Severity = GuardrailSeverity.Warning
}
: null
};
case "email":
var emailPattern = @"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$";
var isEmail = Regex.IsMatch(response.Content.Trim(), emailPattern);
return new GuardrailEvaluationResult
{
Passed = isEmail,
Violation = !isEmail
? new GuardrailViolation
{
Rule = "format_email",
Message = "Response is not a valid email address",
Severity = GuardrailSeverity.Warning
}
: null
};
default:
return new GuardrailEvaluationResult { Passed = true };
}
}
}Guardrail Categories
| Guardrail | Severity | Description |
|---|---|---|
| Content Safety | Critical | Blocks harmful, hateful, or policy-violating content |
| PII Detection | Critical | Prevents exposure of personal identifiable information |
| Format Validation | Warning | Ensures output matches expected format (JSON, XML, etc.) |
| Hallucination Check | Warning | Flags claims not grounded in provided context |
| Token Limit | Info | Warns when output approaches configured token limits |
| Relevance Filter | Warning | Checks if output is relevant to the input query |
| Citation Required | Warning | Ensures claims include citations when policy requires |
17. Developer Portal & Documentation
The developer portal is the primary interface through which developers discover, test, and integrate with the AI gateway. A well-designed portal dramatically reduces time-to-first-API-call and improves developer satisfaction. The portal provides interactive API documentation, key management, usage dashboards, code examples, and troubleshooting guides.
Portal Features
- Interactive API Explorer: Developers can make live API calls directly from the browser with their API keys, seeing real responses without writing any code. This is built using OpenAPI specifications and Swagger UI.
- Key Management Dashboard: Create new keys, view existing keys, set budgets, configure permissions, and revoke keys. Each key shows its usage history and remaining budget.
- Usage Analytics: Real-time and historical usage charts showing request volume, token consumption, cost breakdown by model, latency percentiles, and error rates. Filters allow drilling down by time range, model, and key.
- Code Samples: Complete integration examples in Python, TypeScript, C#, Go, Java, Ruby, and curl. Each example is tested and maintained alongside the API.
- Playground: An interactive chat interface for testing models with custom system prompts, parameters, and tools. Playground sessions are logged and can be shared with team members.
- Webhook Configuration: Set up webhooks for usage threshold alerts, safety incidents, and billing events. Webhook payloads are signed for verification.
- SDK Downloads: Official client libraries for all major languages with versioning and changelog management.
Portal Architecture
C#
[ApiController]
[Route("api/portal")]
public class DeveloperPortalController : ControllerBase
{
private readonly IPortalService _portalService;
[HttpGet("dashboard/{orgId}")]
public async Task<PortalDashboard> GetDashboard(string orgId)
{
return new PortalDashboard
{
Usage = await _portalService.GetUsageSummaryAsync(
orgId, TimeSpan.FromDays(30)),
ActiveKeys = await _portalService.GetKeyCountAsync(orgId),
MonthlyCost = await _portalService.GetMonthlyCostAsync(orgId),
TopModels = await _portalService.GetTopModelsAsync(
orgId, top: 5),
ErrorRate = await _portalService.GetErrorRateAsync(
orgId, TimeSpan.FromDays(7)),
AverageLatency = await _portalService.GetAverageLatencyAsync(
orgId, TimeSpan.FromDays(7)),
Alerts = await _portalService.GetActiveAlertsAsync(orgId)
};
}
[HttpGet("usage/{orgId}/timeseries")]
public async Task<IEnumerable<UsageTimeseries>> GetUsageTimeseries(
string orgId,
[FromQuery] DateTime from,
[FromQuery] DateTime to,
[FromQuery] string granularity = "hour")
{
return await _portalService.GetUsageTimeseriesAsync(
orgId, from, to, granularity);
}
[HttpGet("playground/models")]
public async Task<IEnumerable<PlaygroundModel>> GetPlaygroundModels(
[FromHeader(Name = "Authorization")] string authHeader)
{
var apiKey = await ValidateAuthAsync(authHeader);
return await _portalService.GetPlaygroundModelsAsync(
apiKey.OrgId);
}
}Documentation Structure
| Section | Content | Audience |
|---|---|---|
| Getting Started | Quickstart guide, first API call in 5 minutes | New developers |
| API Reference | Complete endpoint documentation with examples | All developers |
| Guides | Deep-dive tutorials on streaming, tools, safety | Intermediate developers |
| Cookbooks | Complete application examples and patterns | All developers |
| SDK Reference | Language-specific documentation for each SDK | All developers |
| Migration Guides | Step-by-step migration from other providers | Migrating developers |
| Security | Security best practices, key management, compliance | Security engineers |
| Troubleshooting | Common errors, debugging guides, FAQ | All developers |
18. Enterprise Governance & Compliance
Enterprise customers require governance capabilities that go far beyond individual developer needs. This includes organization-wide policies, team management, cost allocation, data residency controls, and compliance certifications. The governance layer ensures that all AI usage within an organization complies with internal policies and external regulations.
Organization Management
C#
public class EnterpriseGovernanceService
{
private readonly IDbConnection _db;
private readonly IAuditLogger _auditLogger;
public async Task<GovernanceCheckResult> EvaluateGovernanceAsync(
InferenceRequest request, ApiKeyInfo key, Guid orgId)
{
var org = await GetOrganizationAsync(orgId);
if (org.SpentCents >= org.MonthlyBudgetCents &&
org.MonthlyBudgetCents > 0)
{
return new GovernanceCheckResult
{
Allowed = false,
Reason = "Monthly budget exceeded",
Code = "budget_exceeded"
};
}
var policy = await GetSafetyPolicyAsync(org.SafetyPolicyId);
var dataResidency = await CheckDataResidencyAsync(
request, org);
if (!dataResidency.Compliant)
{
return new GovernanceCheckResult
{
Allowed = false,
Reason = $"Data residency violation: {dataResidency.Reason}",
Code = "data_residency_violation"
};
}
var modelApproval = await CheckModelApprovalAsync(
request.Model, orgId);
if (!modelApproval.Approved)
{
return new GovernanceCheckResult
{
Allowed = false,
Reason = $"Model {request.Model} is not approved for this organization",
Code = "model_not_approved"
};
}
var contentPolicy = await EvaluateContentPolicyAsync(
request, policy);
if (!contentPolicy.Compliant)
{
return new GovernanceCheckResult
{
Allowed = false,
Reason = contentPolicy.Reason,
Code = "content_policy_violation"
};
}
await _auditLogger.LogAsync(new AuditEvent
{
OrgId = orgId,
ActorId = key.Id,
Action = "inference_request",
Resource = request.Model,
Details = new
{
requestId = request.Id,
tokens = request.EstimatedTokens,
model = request.Model
}
});
return new GovernanceCheckResult { Allowed = true };
}
public async Task<CostAllocation> AllocateCostsAsync(
string orgId, DateTime period)
{
var usageByTeam = await _db.QueryAsync(
@"SELECT t.name as TeamName,
SUM(u.cost_cents) as TotalCost,
SUM(u.total_tokens) as TotalTokens,
COUNT(*) as RequestCount
FROM usage_records u
JOIN api_keys k ON u.api_key_id = k.id
JOIN teams t ON k.team_id = t.id
WHERE u.org_id = @OrgId
AND u.created_at >= @PeriodStart
AND u.created_at < @PeriodEnd
GROUP BY t.name
ORDER BY TotalCost DESC",
new
{
OrgId = orgId,
PeriodStart = period.AddDays(-30),
PeriodEnd = period
});
return new CostAllocation
{
Period = period,
Teams = usageByTeam.ToList(),
TotalCostCents = usageByTeam.Sum(t => (long)t.TotalCost)
};
}
}Compliance Matrix
| Regulation | Requirement | Gateway Control |
|---|---|---|
| SOC 2 Type II | Access controls, audit logging, monitoring | API key scoping, full audit trail, real-time alerts |
| HIPAA | PHI protection, BAA, encryption at rest | Data encryption, PII detection guardrails, BAA support |
| GDPR | Data minimization, right to erasure, consent | Usage data deletion, data residency controls, retention policies |
| ISO 27001 | Information security management | Key rotation, access reviews, vulnerability scanning |
| CCPA | Consumer privacy rights | Usage transparency, opt-out mechanisms, data inventory |
19. Audit Logging & Observability
A comprehensive audit trail is essential for security investigations, compliance reporting, and operational debugging. Every significant action in the system is logged with who, what, when, where, and result details. The audit logging subsystem is designed for append-only immutability and efficient querying over large time ranges.
Audit Logger Implementation
C#
public class AuditLogger : IAuditLogger
{
private readonly IKafkaProducer _kafkaProducer;
private readonly IDbConnection _db;
private const string AuditTopic = "audit.events";
public async Task LogAsync(AuditEvent auditEvent)
{
auditEvent.Id = Guid.NewGuid();
auditEvent.Timestamp = DateTime.UtcNow;
await _kafkaProducer.ProduceAsync(
AuditTopic,
auditEvent.OrgId.ToString(),
auditEvent);
}
public async Task<AuditLogPage> QueryAsync(
AuditLogQuery query)
{
var sql = new StringBuilder(
@"SELECT id, org_id as OrgId, actor_id as ActorId,
actor_type as ActorType, action, resource_type as ResourceType,
resource_id as ResourceId, details, ip_address as IpAddress,
user_agent as UserAgent, created_at as CreatedAt
FROM audit_logs
WHERE org_id = @OrgId
AND created_at >= @From
AND created_at < @To");
var parameters = new DynamicParameters();
parameters.Add("OrgId", query.OrgId);
parameters.Add("From", query.From);
parameters.Add("To", query.To);
if (!string.IsNullOrEmpty(query.Action))
{
sql.Append(" AND action = @Action");
parameters.Add("Action", query.Action);
}
if (!string.IsNullOrEmpty(query.ActorId))
{
sql.Append(" AND actor_id = @ActorId");
parameters.Add("ActorId", query.ActorId);
}
if (!string.IsNullOrEmpty(query.ResourceType))
{
sql.Append(" AND resource_type = @ResourceType");
parameters.Add("ResourceType", query.ResourceType);
}
sql.Append(" ORDER BY created_at DESC");
sql.Append(" LIMIT @Limit OFFSET @Offset");
parameters.Add("Limit", query.PageSize);
parameters.Add("Offset", (query.Page - 1) * query.PageSize);
var logs = await _db.QueryAsync<AuditLogEntry>(
sql.ToString(), parameters);
var totalCount = await _db.ExecuteScalarAsync<long>(
@"SELECT COUNT(*) FROM audit_logs
WHERE org_id = @OrgId
AND created_at >= @From
AND created_at < @To",
new { query.OrgId, query.From, query.To });
return new AuditLogPage
{
Entries = logs.ToList(),
TotalCount = totalCount,
Page = query.Page,
PageSize = query.PageSize
};
}
}Observability Stack
| Layer | Tool | Purpose |
|---|---|---|
| Metrics | Prometheus + Grafana | Request rates, latencies, error rates, token usage |
| Tracing | OpenTelemetry + Jaeger | Distributed tracing across gateway components |
| Logging | Structured JSON to stdout | Correlated logs with request IDs and org context |
| Analytics | ClickHouse | Fast analytical queries on usage data |
| Alerting | PagerDuty + Slack | Real-time alerts for SLO violations and incidents |
| Profiling | Continuous profiling (Pyroscope) | CPU and memory profiling in production |
Key Dashboards
The monitoring stack includes several critical dashboards. The operations dashboard shows real-time request rate, error rate, latency percentiles, and active connections across all gateway instances. The business dashboard displays total API calls, revenue, active organizations, and growth metrics. The model performance dashboard compares latency, error rates, and cost across different providers and model versions. The safety dashboard tracks blocked requests, policy violations, and prompt injection attempts over time. Each dashboard is built in Grafana with consistent color coding and alert thresholds.
20. Cost Optimization Strategies
AI inference is expensive. Enterprise organizations can spend hundreds of thousands of dollars per month on LLM API calls. The gateway plays a crucial role in reducing these costs through multiple optimization strategies that are transparent to the applications using the gateway.
Optimization Strategies
| Strategy | Potential Savings | Implementation Complexity |
|---|---|---|
| Prompt Caching | 25-40% | Medium |
| Intelligent Model Routing | 15-30% | High |
| Token-Aware Batching | 5-10% | Medium |
| Response Compression | 3-5% | Low |
| Cost-Based Model Selection | 20-35% | Medium |
| Prompt Optimization | 10-20% | High |
| Usage Quotas & Alerts | 10-15% | Low |
Cost Optimization Service
C#
public class CostOptimizationService
{
private readonly ModelRegistryService _registry;
private readonly UsageAnalyticsService _analytics;
public async Task<CostOptimizationReport> AnalyzeAsync(
string orgId, DateTime from, DateTime to)
{
var usage = await _analytics.GetDetailedUsageAsync(
orgId, from, to);
var recommendations = new List<CostRecommendation>();
var cacheableRequests = usage
.Where(u => !u.IsCached && IsCacheable(u))
.GroupBy(u => u.PromptHash)
.Where(g => g.Count() > 3)
.Sum(g => g.Count() * g.First().CostCents);
if (cacheableRequests > 0)
{
recommendations.Add(new CostRecommendation
{
Type = "prompt_caching",
Title = "Enable prompt caching for repeated prompts",
EstimatedSavingsCents = (int)(cacheableRequests * 0.7),
Confidence = 0.85,
Implementation = "Add cache key generation for " +
$"{usage.Count(u => IsCacheable(u))} cacheable request patterns"
});
}
var expensiveModelUsage = usage
.Where(u => u.CostPerToken > GetMedianCostPerToken(usage) * 3)
.ToList();
if (expensiveModelUsage.Any())
{
var cheaperAlternatives = FindCheaperAlternatives(
expensiveModelUsage);
if (cheaperAlternatives.Any())
{
recommendations.Add(new CostRecommendation
{
Type = "model_downgrade",
Title = $"Route {expensiveModelUsage.Count} requests " +
$"to more cost-effective models",
EstimatedSavingsCents = cheaperAlternatives
.Sum(c => c.SavingsCents),
Confidence = 0.70,
Implementation = "Configure routing rules for " +
string.Join(", ", cheaperAlternatives
.Select(c => c.CurrentModel))
});
}
}
var totalCurrentCost = usage.Sum(u => u.CostCents);
var totalPotentialSavings = recommendations
.Sum(r => r.EstimatedSavingsCents);
return new CostOptimizationReport
{
Period = new DateRange(from, to),
TotalCostCents = totalCurrentCost,
PotentialSavingsCents = totalPotentialSavings,
SavingsPercentage = totalCurrentCost > 0
? (double)totalPotentialSavings / totalCurrentCost * 100
: 0,
Recommendations = recommendations.OrderByDescending(
r => r.EstimatedSavingsCents).ToList()
};
}
}21. Monitoring, Alerting & SLOs
Reliable monitoring is the eyes and ears of the AI gateway platform. Without comprehensive observability, issues go undetected until customers report them, and root cause analysis becomes a time-consuming guessing game. The monitoring system is built around service level objectives that define what "good" looks like for each critical metric.
SLO Definitions
| SLO | Target | Error Budget | Window |
|---|---|---|---|
| Availability | 99.99% | 4.32 min/month | 30 days rolling |
| Latency (p50) | < 200ms | 5% of requests > 200ms | 7 days rolling |
| Latency (p99) | < 2000ms | 1% of requests > 2000ms | 7 days rolling |
| TTFT (p50) | < 100ms | 5% of streams > 100ms | 7 days rolling |
| Error Rate | < 0.1% | 0.1% of requests fail | 7 days rolling |
| Data Durability | 99.999999999% | Near zero | Annual |
Alert Configuration
YAML
# prometheus/alerts.yml
groups:
- name: ai_gateway_alerts
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.001
for: 5m
labels:
severity: critical
annotations:
summary: "Error rate exceeds 0.1% SLO threshold"
- alert: HighLatency
expr: |
histogram_quantile(0.50,
rate(http_request_duration_seconds_bucket[5m])
) > 0.2
for: 10m
labels:
severity: warning
annotations:
summary: "P50 latency exceeds 200ms SLO threshold"
- alert: ProviderCircuitBreakerOpen
expr: |
ai_gateway_circuit_breaker_state{state="open"} == 1
for: 1m
labels:
severity: warning
annotations:
summary: "Circuit breaker open for provider {{ $labels.provider }}"
- alert: HighTokenConsumptionRate
expr: |
sum(rate(ai_gateway_tokens_total[1h])) > 1000000
for: 30m
labels:
severity: info
annotations:
summary: "Token consumption rate is unusually high"
- alert: CacheHitRateDrop
expr: |
rate(ai_gateway_cache_hits_total[15m])
/ rate(ai_gateway_cache_requests_total[15m]) < 0.15
for: 30m
labels:
severity: warning
annotations:
summary: "Cache hit rate dropped below 15% threshold"
- alert: BudgetApproachingLimit
expr: |
ai_gateway_org_spent_cents
/ ai_gateway_org_budget_cents > 0.9
for: 5m
labels:
severity: warning
annotations:
summary: "Organization {{ $labels.org_id }} spent 90% of budget"Metrics Collection
All gateway components emit Prometheus metrics using the standard naming conventions. Key metric families include http_requests_total (counter), http_request_duration_seconds (histogram), ai_gateway_tokens_total (counter by model and org), ai_gateway_cost_cents_total (counter by model and org), ai_gateway_cache_hits_total (counter), ai_gateway_safety_blocks_total (counter by category), and ai_gateway_stream_active_connections (gauge). These metrics are scraped by Prometheus at fifteen-second intervals and retained for ninety days. Long-term aggregation into ClickHouse provides month-over-month and year-over-year trend analysis.
22. Testing, Load Testing & Chaos Engineering
Testing an AI gateway requires a multi-layered strategy that covers unit behavior, integration contracts with upstream providers, load capacity, and resilience under failure conditions. The testing strategy must account for the non-deterministic nature of LLM responses and the variability of upstream provider behavior.
Testing Pyramid
| Test Layer | Scope | Tools | Frequency |
|---|---|---|---|
| Unit Tests | Individual components (router, meter, cache) | xUnit, Moq | Every commit |
| Integration Tests | API endpoints with mocked providers | WebApplicationFactory, Testcontainers | Every PR |
| Contract Tests | Provider API compatibility | Pact, WireMock | Nightly |
| Load Tests | Throughput and latency under load | k6, NBomber | Weekly |
| Chaos Tests | Resilience under failure | Chaos Monkey, Toxiproxy | Weekly |
| E2E Tests | Full pipeline with real providers | Playwright, custom harness | Pre-deploy |
Load Test Example
C#
// NBomber load test configuration
public class GatewayLoadTest : NBomberScenario
{
protected override void Configure()
{
var httpClient = new HttpClient
{
BaseAddress = new Uri("https://gateway.example.com")
};
var inferenceStep = Step.Create("inference_request",
async context =>
{
var request = new HttpRequestMessage(
HttpMethod.Post, "/v1/messages")
{
Content = JsonContent.Create(new
{
model = "claude-sonnet-4-20250514",
max_tokens = 100,
messages = new[]
{
new { role = "user",
content = "Explain quantum computing in 3 sentences" }
}
})
};
request.Headers.Add("Authorization",
$"Bearer {TestApiKey}");
var response = await httpClient.SendAsync(request);
return response.IsSuccessStatusCode
? Response.Ok()
: Response.Fail();
});
var streamStep = Step.Create("streaming_request",
async context =>
{
var request = new HttpRequestMessage(
HttpMethod.Post, "/v1/messages")
{
Content = JsonContent.Create(new
{
model = "claude-sonnet-4-20250514",
max_tokens = 200,
stream = true,
messages = new[]
{
new { role = "user",
content = "Write a haiku about software engineering" }
}
})
};
request.Headers.Add("Authorization",
$"Bearer {TestApiKey}");
var response = await httpClient.SendAsync(request,
HttpCompletionOption.ResponseHeadersRead);
return response.IsSuccessStatusCode
? Response.Ok()
: Response.Fail();
});
var loadSimulations = new[]
{
Simulation.RampingInject(rate: 100,
interval: TimeSpan.FromSeconds(1),
during: TimeSpan.FromMinutes(5)),
Simulation.Inject(rate: 500,
interval: TimeSpan.FromSeconds(1),
during: TimeSpan.FromMinutes(10)),
Simulation.RampingInject(rate: 0,
interval: TimeSpan.FromSeconds(1),
during: TimeSpan.FromMinutes(2))
};
WithLoadSimulations(loadSimulations);
WithSteps(inferenceStep, streamStep);
}
}Chaos Engineering Scenarios
| Scenario | Expected Behavior | Verification |
|---|---|---|
| Primary provider goes down | Automatic failover to secondary provider | Request succeeds, latency within SLO |
| Redis cluster partition | Local rate limit fallback, degraded caching | Requests still processed, rate limits approximate |
| Database becomes slow | Auth cache serves requests, metering buffers | Auth latency within 2x normal |
| Network latency spike to provider | Circuit breaker opens, requests rerouted | No requests timeout, fallback engaged |
| Memory pressure on gateway instance | Graceful shedding of lowest-priority traffic | Core requests still served, p99 latency increases |
| Kafka broker failure | Metering events buffered locally, flushed on recovery | No data loss after recovery |
23. Interview Q&A
These questions cover the most common and challenging interview topics for AI gateway and platform engineering roles. Each answer demonstrates senior-level understanding of the trade-offs and design decisions involved.
Q1: How would you design the rate limiting system to handle burst traffic while remaining accurate?
Answer: I would implement a multi-layer rate limiting approach. At the primary layer, I use a Redis-backed sliding window counter algorithm implemented with a Lua script for atomicity. The sliding window provides smoother rate limiting than fixed windows by considering both the current window and the previous window weighted by how far we are into the current window. This prevents the burst-at-boundary problem. At the secondary layer, each gateway instance maintains local token bucket counters as a fallback when Redis is temporarily unavailable. These local counters are approximate but prevent unbounded traffic during Redis failures. The reconciliation layer periodically syncs local counters back to Redis. For burst handling, I support a separate burst allowance parameter that permits short spikes above the sustained rate, similar to how TCP congestion control allows bursts. The burst budget refills gradually over time. The Lua script for the Redis implementation ensures that the check-and-increment operation is atomic, preventing race conditions under concurrent requests from multiple gateway instances.
Q2: How do you handle streaming response failures midway through a response?
Answer: Mid-stream failures require careful handling to avoid data loss and provide a good client experience. When a streaming connection to the upstream provider drops, the gateway first checks whether the partial response is still usable. If the client has already consumed tokens via the SSE stream, we cannot retry from scratch because the client has already displayed partial content. Instead, we log the failure with the partial content received and return a stream termination event with a specific stop reason. For non-displayed streams where the client has not yet consumed the partial data, we can attempt a retry with a modified request that includes a continuation marker if the provider supports it. We track partial response state in memory with a per-connection context object that stores the accumulated tokens, the last upstream chunk ID, and the elapsed time. The retry policy considers the time already spent and the number of tokens already generated to decide whether a retry makes sense or would waste more tokens. Circuit breaker state is updated based on the failure pattern to avoid hammering a failing provider.
Q3: Explain how you would design the prompt caching system for maximum hit rates.
Answer: Maximizing cache hit rates requires a multi-strategy approach. The foundation is exact-match caching using SHA-256 hashes of the normalized prompt content, model, temperature, and max tokens parameters. Normalization is critical and includes trimming whitespace, normalizing Unicode characters, and sorting JSON keys. Beyond exact match, I implement prefix caching where prompts with identical system prompts and conversation history but different final user messages share a prefix cache entry for the system prompt portion. This is particularly effective for applications that use large system prompts with small variations in user input. The third layer is semantic similarity caching using vector embeddings of the prompt content. When an exact or prefix match is not found, we compute an embedding of the new prompt and search for similar cached prompts above a configurable similarity threshold. This catches paraphrased versions of the same question. The cache key also includes model-specific parameters like temperature and top_p to ensure cached responses are only served for identical generation settings. TTL is set based on content freshness, with longer TTLs for factual queries and shorter TTLs for time-sensitive content. Cache warming pre-populates the cache with frequently accessed prompts during off-peak hours.
Q4: How do you ensure the safety layer does not add unacceptable latency to the request path?
Answer: Safety layer latency is a critical concern because it sits directly in the hot path of every request. I address this through several strategies. First, the safety classifier runs as a lightweight model optimized for speed, not accuracy at the cost of latency. I use a distilled model that is specifically trained for content classification with a target inference time under 20 milliseconds. Second, I implement a tiered approach where pattern-matching rules run first in under 1 millisecond for obvious violations, and the ML classifier only runs when patterns do not match. Third, for known-safe content patterns like code generation requests or structured data transformations, I maintain a bypass list that skips the full safety check. Fourth, I use async processing where the input safety check runs concurrently with the cache lookup, and we only block on the result before forwarding to the upstream provider. The output safety check runs concurrently with the streaming response delivery, terminating the stream only if a critical violation is detected mid-stream. Finally, I maintain a separate safety model inference pool with dedicated GPU resources to prevent safety processing from competing with other workloads for compute resources.
Q5: Design the data model for supporting multi-tenancy with complete isolation between organizations.
Answer: Multi-tenant isolation is achieved through a combination of data partitioning, access control, and network isolation. At the data layer, every row in the database includes an org_id column as a mandatory filter. Row-level security policies in PostgreSQL enforce that queries cannot access data from other organizations, even if application code has bugs. API keys are bound to their parent organization and cannot be used across org boundaries. For cache isolation, Redis key namespaces are prefixed with the org_id, and Redis ACLs prevent cross-namespace access. At the network level, organization-specific API keys route to organization-specific rate limit pools, preventing one organization's traffic from affecting another's rate limit budget. For enterprise customers requiring stronger isolation, I support dedicated database schemas or even separate database instances, with a routing layer that directs the organization's traffic to their dedicated infrastructure. Audit logs include org_id on every entry, and the query layer always filters by org_id even for platform administrators who need cross-org visibility through a separate admin interface with elevated permissions and additional audit logging of their access.
Q6: How would you handle a scenario where a model provider changes their API format or behavior without notice?
Answer: Provider API instability is a real and common problem. I handle this through a combination of contract testing, version detection, and graceful degradation. The adapter layer for each provider is versioned independently, so when a provider changes their API, I can ship a new adapter version without redeploying the entire gateway. Contract tests run nightly against the live provider APIs and detect breaking changes immediately. When a contract test fails, an alert triggers and the on-call engineer can assess whether to pin to a specific API version, update the adapter, or route traffic away from the affected provider. For runtime detection, I implement response validation that checks whether the upstream response matches the expected schema. If validation fails, the request is routed to a fallback provider and the anomaly is logged for investigation. The model registry includes a capabilities negotiation feature where the gateway probes the provider endpoint with a test request on startup and determines the actual capabilities and format supported, rather than assuming based on the model name. This dynamic capability detection provides a safety net against silent API changes.
Q7: Explain how you would implement cost allocation for a shared organization with multiple teams.
Answer: Cost allocation for multi-team organizations requires tagging at the API key level. Each API key belongs to a team within the organization, and the team association is immutable after creation. When usage records are generated, they inherit the team_id from the API key used. The billing aggregation pipeline groups usage by team within each organization and produces team-level cost reports. For shared resources like cached responses, I allocate the cost proportionally based on team consumption. If Team A generates 60% of cache hits for a particular cached prompt, they are charged 60% of what the uncached cost would have been. The organization admin dashboard provides drill-down views showing cost by team, by model, by time period, and by individual API key. Budget limits can be set at the organization level, the team level, and the individual key level, creating a hierarchical budget management system. Alerts trigger when any level approaches its budget limit, giving team leads early warning before they exceed their allocation.
Q8: How do you test the non-deterministic nature of LLM responses in your integration tests?
Answer: Testing non-deterministic outputs requires shifting from exact-match assertions to behavioral assertions. For integration tests, I use several strategies. First, I test structural properties rather than content properties. Instead of asserting the response contains a specific string, I assert the response is valid JSON, contains the expected fields, and the content field is non-empty. Second, I use snapshot testing with golden files for provider responses, but the snapshots are regenerated periodically to account for model updates. Third, I mock the upstream providers in most integration tests using WireMock or a custom test double that returns pre-recorded responses. This makes the tests deterministic and fast. Fourth, for tests that need to verify actual LLM behavior, like safety filtering, I use a set of canonical test prompts with known expected behaviors and assert against those. The test suite includes a classification of tests into deterministic tests that always produce the same result and probabilistic tests that verify statistical properties over multiple runs. The probabilistic tests run with a confidence interval and are flagged in CI so failures are not treated as blocking.
Q9: Describe the architecture for supporting both synchronous and streaming responses through the same endpoint.
Answer: The unified endpoint uses the request's stream parameter to determine the response mode. The initial processing pipeline including authentication, rate limiting, routing, and safety checking is identical for both modes. The divergence happens at the provider communication layer. For synchronous mode, the gateway sends the request, waits for the complete response, runs output safety checks, and returns the full response. For streaming mode, the gateway establishes a long-lived HTTP connection to the upstream provider and begins proxying SSE chunks as they arrive. The key architectural decision is how to handle the output safety check in streaming mode. I implement a parallel pipeline where the stream is simultaneously delivered to the client and buffered for output safety analysis. The safety analysis runs on chunks as they arrive, using a streaming classifier that can make decisions on partial content. If a critical violation is detected mid-stream, the gateway sends a stream termination event and closes the connection. This approach minimizes latency because the safety check does not delay the initial tokens while still catching violations. The connection management layer uses a separate thread pool for streaming connections to prevent long-lived streams from starving short synchronous requests.
Q10: How would you design the system to handle a sudden 10x spike in traffic from a viral application?
Answer: Sudden traffic spikes are the ultimate test of the gateway's architecture. I prepare for this through several mechanisms. First, the compute layer is container-based with horizontal pod autoscaling configured on both CPU utilization and custom metrics like requests per second per pod. The HPA is tuned with conservative stabilization windows to prevent thrashing but aggressive enough to scale within 60 seconds of a traffic spike. Second, I maintain warm standby capacity of at least 30% above current peak to absorb sudden spikes without waiting for new instances to start. Third, the load balancer is configured with connection draining so that new instances can start accepting traffic immediately. Fourth, I implement adaptive rate limiting that can be rapidly adjusted per organization. When a single application causes a spike, the per-key and per-org rate limits provide natural backpressure. Fifth, the data layer is designed for burst traffic with Kafka buffering metering events and Redis absorbing rate limit checks. Sixth, I have a runbook for traffic spikes that includes temporarily raising limits for the spiking application, scaling up provider connection pools, and activating additional gateway regions. The key insight is that the gateway must never be the bottleneck. If we cannot handle the traffic, we should gracefully shed load with clear 429 responses rather than degrading silently.
Conclusion
Building an Anthropic-style AI Gateway and Model Management Platform is a substantial engineering undertaking that touches every area of modern systems design: distributed computing, security, real-time data processing, machine learning operations, and developer experience. The platform serves as the critical infrastructure layer that enables organizations to safely, efficiently, and cost-effectively leverage the power of large language models.
The key architectural principles that make this system work are separation of concerns with dedicated services for authentication, routing, metering, safety, and streaming; horizontal scalability at every layer; defense in depth for security with multiple validation and safety checkpoints; comprehensive observability from the first byte of a request to the final audit log entry; and cost consciousness built into every decision from model routing to caching strategy.
As the AI industry continues to evolve rapidly with new models, new providers, and new use cases emerging constantly, the gateway platform must be designed for change. The adapter pattern for provider integration, the registry pattern for model management, and the plugin pattern for guardrails all ensure that new capabilities can be added without disrupting existing functionality. Organizations that invest in building or adopting a robust AI gateway will find themselves with a significant competitive advantage, able to iterate faster, reduce costs, maintain compliance, and scale their AI initiatives with confidence.
The patterns and techniques described in this guide are not theoretical. They represent the collective wisdom gained from building and operating production AI platforms that serve billions of requests. Whether you are preparing for a system design interview or building the actual infrastructure, these patterns give you a solid foundation for success in the exciting and rapidly growing field of AI platform engineering.