Model Deployment Tutorial: Serving, Docker, FastAPI, ONNX, TensorRT (2026)
Building a model is only half the battle. Deploying to production requires containerization, API design, performance optimization, and monitoring.
By 2026, FastAPI, Docker, ONNX, and TensorRT form a robust deployment stack. Model optimization reduces inference latency by 5-10x.
Serving with FastAPI
FastAPI provides automatic OpenAPI docs, Pydantic validation, and async support. Define input schemas, load the model at startup, and create /predict endpoints.
I built a model server handling 10,000 requests per minute using async endpoints with thread pools for CPU-bound inference.
from fastapi import FastAPI\nfrom pydantic import BaseModel\nimport joblib\napp = FastAPI(); model = joblib.load("model.pkl")\nclass PredictRequest(BaseModel):\n features: list[float]\n@app.post("/predict")\nasync def predict(req: PredictRequest):\n return {"prediction": model.predict([req.features]).tolist()}Containerization with Docker
Docker ensures identical environments across development and production. Multi-stage builds keep images small by separating build from runtime dependencies.
I reduced a model server image from 1.2GB to 280MB using multi-stage builds with python:3.11-slim base image.
FROM python:3.11-slim as builder\nWORKDIR /app\nCOPY requirements.txt .\nRUN pip install --user -r requirements.txt\nFROM python:3.11-slim\nCOPY --from=builder /root/.local /root/.local\nCOPY model.pkl app.py ./\nCMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "80"]Model Optimization with ONNX
ONNX enables interoperability and hardware-specific optimizations. Exporting to ONNX unlocks constant folding, operator fusion, and quantization.
Converting a BERT classifier to ONNX achieved 4x latency reduction on CPU using ONNX Runtime. Int8 quantization halved latency again with only 0.5 percent accuracy drop.
import torch.onnx\ndummy = torch.randn(1, 3, 224, 224)\ntorch.onnx.export(model, dummy, "model.onnx", input_names=["input"], output_names=["output"])\nimport onnxruntime as ort\nsession = ort.InferenceSession("model.onnx")GPU Optimization with TensorRT
TensorRT optimizes deep learning models for NVIDIA GPUs through layer fusion, precision calibration, and kernel auto-tuning.
I optimized a ResNet-50 model with TensorRT and FP16 precision, achieving 5x throughput improvement with less than 1 percent accuracy loss.
import tensorrt as trt\nlogger = trt.Logger(trt.Logger.WARNING)\nbuilder = trt.Builder(logger)\nnetwork = builder.create_network()\n# Parse ONNX and build engine\nwith builder.build_serialized_network(network, config) as engine:\n # Run inference with optimized engine\n passModel Monitoring and Drift Detection
Production models require monitoring for data drift, concept drift, and performance degradation. Log predictions, track distributions, and set alert thresholds.
I set up monitoring with Prometheus metrics for latency, error rate, and prediction distribution. Automated retraining triggers when drift exceeds thresholds.
from scipy.stats import ks_2samp\ndef detect_drift(reference, current, threshold=0.05):\n stat, p_value = ks_2samp(reference, current)\n return p_value < threshold # Drift detectedCI/CD for ML Models
ML CI/CD pipelines automate training, evaluation, testing, and deployment. GitHub Actions and Jenkins trigger pipelines on code or data changes.
Our pipeline: train on new data, evaluate against production metrics, run integration tests, deploy canary, then full rollout. Rollback is automated on metric degradation.
# .github/workflows/ml-pipeline.yml\n# Trigger: push to main or data update\n# Jobs: train -> evaluate -> test -> build -> deploy canary -> full rollout\n# Run: python train.py; python evaluate.py; pytest tests/Frequently Asked Questions
FastAPI vs Flask for ML?
FastAPI has automatic OpenAPI docs, Pydantic validation, and async support. Flask is simpler but lacks these features. FastAPI is the modern choice for ML serving.
When to use ONNX vs TensorRT?
ONNX for cross-platform deployment and CPU optimization. TensorRT for maximum GPU performance. Use ONNX as intermediate format for TensorRT conversion.
How to handle model versioning?
Use MLflow Model Registry or W&B Model Registry. Tag models with version, stage (staging/production), and metadata about training run and metrics.
What metrics to monitor in production?
Latency (p50, p95, p99), throughput, error rate, prediction distribution, feature drift, and actual performance when ground truth arrives.
Originally published on Ayodhyyya. Last updated June 1, 2026.