{ "type": "new-runtime-knowledge-guide", "version": 1, "generated_at": "2026-07-23", "volume": 26, "slug": "llm-service-deployment-devops-reference", "title": "Deployment & DevOps for an LLM Service", "description": "This volume is not about the code inside the agent, but about the operation outside: ingress, TLS, forwarded headers, “root path”, probes, secrets, queues, tracing, metrics, token/cost budgets, autoscaling, rollout/rollback and the place of “MCP” in the production circuit. This is a map of what it takes for an LLM service to not just run, but live and experience real traffic.", "track": { "slug": "application-engineering", "title": "Application Engineering", "route": "https://newruntime.com/learn/tracks/application-engineering/", "position": 6 }, "counts": { "sections": 7, "cards": 36 }, "routes": { "html": "https://newruntime.com/learn/llm-service-deployment-devops-reference/", "json": "https://newruntime.com/learn/llm-service-deployment-devops-reference.json" }, "sections": [ { "number": "01", "title": "Ops Frame", "intro": "", "cards": [ { "number": "01", "title": "LLM service in production = code + network + state + operation", "tag": "Ops Frame", "description": "When the service goes out, its reliability is determined not only by the graph code. There are edge proxies, TLS, readiness, steady state, queues, timeouts, token budget, tracing and rollback strategy. The real system becomes multi-layered.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "THE FULL CARTINE\n\nclient\n ↓\nedge / ingress / TLS\n ↓\nFastAPI app\n ↓\nLangGraph runtime\n ↓\nexternal state / queues / tools / models\n ↓\nlogs / metrics / traces / alerts" } ] }, { "number": "02", "title": "Production begins where the service has a life cycle.", "tag": "Ops Frame", "description": "In dev, you just run uvicorn. In production, you need to roll out, heat, check, scale, limit, observe and be able to roll back. This is the main difference between operation and local launch.", "blocks": [ { "kind": "list", "label": "production cycle", "items": [ "deploy", "serve", "observe", "rollback" ] } ] }, { "number": "03", "title": "Not only uptime, but also token economics are especially important for LLM services.", "tag": "Ops Frame", "description": "A conventional API often has pain in the CPU or DB. The LLM service adds to this tokens, latency external model providers, the cost of reranker/tool loops and the length of the spoken state. The ops model is closely related to budgets and guardrails.", "blocks": [ { "kind": "code", "label": "What Really Limits the System", "language": "text", "value": "request rate\n× average tokens\n× model latency\n× tool count\n× retry policy\n= real operational price of the service" } ] }, { "number": "04", "title": "LLM service is more convenient to design as stateless compute + external state", "tag": "Ops Frame", "description": "That's a key operational idea. If the web-process is almost stateless, and threads, checkpoints, queues and artifacts are brought out, then the service is easier to scale, transfer between nodes and experience restarts. For LLM runtime, this is especially important because of the ‘thread id’ and durable execution.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Desirable model\n\nreplica A ─┐\nreplica B ─┼──→ shared persistence / queue / object store\nreplica C ─┘\n\nThen any instance can continue the same conversation thread." } ] }, { "number": "05", "title": "In an LLM service, “alive” and “ready” rarely mean the same thing.", "tag": "Ops Frame", "description": "The process may be alive, but not ready to receive traffic: graph/service objects are not yet available, persistence is not available, configuration is not warmed up, client adapters are not raised. Therefore, “healthz” and “readyz” should almost always be separated.", "blocks": [ { "kind": "list", "label": "That'll come in handy.", "items": [ "liveness", "readiness", "startup probe" ] } ] } ] }, { "number": "02", "title": "Edge, Ingress and HTTPS", "intro": "", "cards": [ { "number": "06", "title": "Ingress/reverse proxy is an external login to a service, not just another YAML.", "tag": "Edge", "description": "Kubernetes Ingress and similar edge layers solve several tasks at once: set HTTP/HTTPS routes outward, route traffic to the internal Service, know TLS termination, name-based routing and load balancing. This is a separate, very important operational layer before the application.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "EDGE PATH\n\ninternet\n ↓\ningress / load balancer\n ↓\nservice\n ↓\npod / app" } ] }, { "number": "07", "title": "TLS is usually terminated at the edge, and the application already comes HTTP.", "tag": "Edge", "description": "FastAPI docs separately remind: HTTPS and certificates live below the HTTP layer. In practice, this often means that TLS ends on proxy/ingress, and before Uvicorn, the application receives internal HTTP traffic. Therefore, certificates, renewals, and SNI are usually served not by an application, but by an edge component.", "blocks": [ { "kind": "list", "label": "practical conclusion", "items": [ "TLS certs more often on proxy", "More often than not, FastAPI app", "The app needs to understand the original scheme/host." ] } ] }, { "number": "08", "title": "Forwarded headers and 'root path' break URLs if not configured", "tag": "Edge", "description": "Behind-proxy is not just about beauty. If a proxy puts HTTPS and path prefix, and the application doesn’t trust forwarded headers or doesn’t know about “root path”, it starts generating incorrect redirect URLs and documentation along the wrong paths.", "blocks": [ { "kind": "code", "label": "what is important to remember", "language": "python", "value": "proxy adds:\n X-Forwarded-For\n X-Forwarded-Proto\n X-Forwarded-Host\n\napp must trust them:\n --forwarded-allow-ips\n\nif proxy strips/adds prefix:\n configure root_path" } ] }, { "number": "09", "title": "For SSE and long responses edge timeouts more important than it seems", "tag": "Edge", "description": "LLM services often stream tokens, tool progress, and updates. This means that ingress, proxy and client timeouts must be aligned with streaming pattern. Otherwise, the service “works”, but edge cuts off the connection before the user sees the answer.", "blocks": [ { "kind": "list", "label": "routine", "items": [ "proxy idle timeout less than generation time", "Disconnect client without correct cancellation", "Long-lived HTTP stream support" ] } ] }, { "number": "10", "title": "CORS, auth and rate limits are often more logical to put closer to the edge.", "tag": "Edge", "description": "Some policies can be done in the application, but operationally more convenient when at least part of the protection and normalization of traffic is at the input: origin policy, TLS, IP filtering, basic rate limiting, size limiting. Especially if the service is open to the browser or external customers.", "blocks": [ { "kind": "list", "label": "frequent edge concerns", "items": [ "CORS", "TLS", "request size caps", "basic throttling" ] } ] } ] }, { "number": "03", "title": "Runtime, Processes and Probes", "intro": "", "cards": [ { "number": "11", "title": "In a container environment, it is often easier to process one process per container and scale replicas.", "tag": "Runtime", "description": "FastAPI docs recommends looking at the deployment context. In orchestrators, it is usually more convenient to run one Uvicorn process per container and scale the number of replicas. It is easier for memory model, ‘lifespan’, graceful shutdown and durability logic around thread state.", "blocks": [ { "kind": "list", "label": "routine", "items": [ "1 process / container", "replicas for scale", "Workers within the process are not always needed." ] } ] }, { "number": "12", "title": "If you use a worker model, remember process semantics.", "tag": "Runtime", "description": "Uvicorn and Gunicorn solve the problem of process control, but this does not change the fact that every worker has his own memory. For an LLM service, this is important: in-memory state, local caches, and lazily initialized clients must be designed with multiprocess behavior in mind.", "blocks": [ { "kind": "code", "label": "important", "language": "text", "value": "worker A memory != worker B memory\n\nTherefore:\n state\n shared persistence\n shared queues\n Carefully with local caches" } ] }, { "number": "13", "title": "LivenessProbe, ReadinessProbe, StartupProbe solve different problems", "tag": "Runtime", "description": "Kubernetes docs are very clear about their roles. Readiness is responsible for receiving traffic. Liveness is responsible for restarting a really unhealthy container. Startup probe is useful when the application takes a long time to climb, so that premature checks do not consider it a breakdown.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Three samples\n\nstartupProbe\n Let the app start quietly\n\nreadinessProbe\n Do not allow traffic until the service is ready\n\nlivenessProbe\n Restart the container if it really freezes or breaks" } ] }, { "number": "14", "title": "‘readyz’ for an LLM service must check not only the process but also the critical wiring", "tag": "Runtime", "description": "Hello-world readiness can be a formality. For an LLM service, it is useful to check that the main service objects are collected, persistence is available and the application can safely accept new run-s. It doesn’t have to be a full synthetic chat, but it’s not just a ‘return 200’ without meaning.", "blocks": [ { "kind": "list", "label": "minimally check", "items": [ "service objects initialized", "checkpointer reachable", "critical config loaded" ] } ] }, { "number": "15", "title": "Graceful shutdown is especially important for streams and queues", "tag": "Runtime", "description": "If the replica leaves during a long stream or background job handoff, you need to correctly terminate connections, stop receiving new traffic and not lose your fortune. This is another reason to keep the application state and queue out of the process and respect the lifecycle container.", "blocks": [ { "kind": "code", "label": "shutdown", "language": "text", "value": "stop accepting traffic\nfinish or safely abort in-flight work\nflush logs/traces\nexit cleanly" } ] } ] }, { "number": "04", "title": "Queueues, Durable Work and Retries", "intro": "", "cards": [ { "number": "16", "title": "Long and heavy LLM tasks often need to be removed from HTTP request lifecycle", "tag": "Queues", "description": "If a task can take tens of seconds or minutes, call a lot of tools, generate reports, or do reindex, it’s best to switch to the job/worker pattern. In this case, the HTTP endpoint creates a run and queues the task, and then the client receives status/result in a separate way.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "JOB FLOW\n\nPOST /jobs\n ↓\nvalidate + persist job\n ↓\nenqueue\n ↓\nworker executes graph\n ↓\nstatus/result store\n ↓\nclient polls or streams status" } ] }, { "number": "17", "title": "Retry policy without idempotence quickly turns into a source of duplicates", "tag": "Queues", "description": "Queue retry, client retry and internal tool retry are easily overlapped. If side-effect tools are non-idempotent, you can send a letter several times, create a ticket or write off money. Therefore, retry should be designed with idempotency strategy.", "blocks": [ { "kind": "list", "label": "routine", "items": [ "idempotency key", "upsert semantics", "limited retry budget" ] } ] }, { "number": "18", "title": "Not only retries, but also dead-letter/failure handling", "tag": "Queues", "description": "Some tasks should not be retracted indefinitely: prompt bug, malformed payload, denied external dependency, broken tool schema. In such cases, you need a clear path for a failed job: error status, DLQ or manual parsing, rather than perpetual retry storm.", "blocks": [ { "kind": "code", "label": "normalcy", "language": "text", "value": "try job\n→ retry N times on transient failures\n→ mark failed on terminal error\n→ send to DLQ / alert / manual review" } ] }, { "number": "19", "title": "Interrupt/resume and queue-based execution are well combined.", "tag": "Queues", "description": "Human-in-the-loop flow doesn’t have to live in a synchronized API. Very often, the pause occurs inside the worker-run, and the “resume” comes later. Durable execution and queue here complement each other, not conflict.", "blocks": [ { "kind": "code", "label": "mental model", "language": "text", "value": "worker executes graph\n→ interrupt reached\n→ state persisted\n→ UI asks human\n→ human confirms\n→ resume event arrives\n→ worker or service continues same thread" } ] }, { "number": "20", "title": "Queue depth is one of the most useful operational metrics.", "tag": "Queues", "description": "LLM systems often rely not only on CPU/Memory, but on the accumulation of heavy tasks. Therefore, queue length, age of oldest job, retry count and throughput workers often speak about the health of the system better than just HTTP latency web service.", "blocks": [ { "kind": "list", "label": "track down", "items": [ "queue depth", "oldest job age", "retry rate", "worker success rate" ] } ] } ] }, { "number": "05", "title": "Observability and Budgets", "intro": "", "cards": [ { "number": "21", "title": "Three Observability Signals: Logs, Metrics, and Tracks", "tag": "Observability", "description": "OpenTelemetry ecosystem is useful precisely because it helps to think not about one “logger”, but about three classes of signals. Logs are good for a specific incident. Metrics are good for aggregate health. Traces are good for a request that has passed through API, graph, tool calls and external dependencies.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Three Signals\n\nlogs, what exactly happened?\nHow the system behaves on average\nHow one request went through the system" } ] }, { "number": "22", "title": "For an LLM service, trace must go through HTTP, graph, and tool layer.", "tag": "Observability", "description": "A simple request trace is not enough. It is useful to see at least: HTTP span, graph invocation, model call, tool calls and external I/O dependencies. Then you can understand where the latency was born: in the model, retriever-e, tool chain or persistence.", "blocks": [ { "kind": "code", "label": "example of a span chain", "language": "text", "value": "HTTP request\n→ chat service\n→ graph run\n→ model call\n→ tool call\n→ db / external api\n→ response serialization" } ] }, { "number": "23", "title": "Not only latency metrics, but also token/cost metrics.", "tag": "Observability", "description": "This is something that teams often forget without LLM experience. Metrics like 'request duration seconds' are useful, but without 'prompt tokens', 'completion tokens', 'tool calls per run', 'cost per request' and 'cost per tenant' picture operational remains blind.", "blocks": [ { "kind": "list", "label": "Very useful LLM metrics", "items": [ "tokens per request", "cost per request", "tool calls per run", "interrupt rate" ] } ] }, { "number": "24", "title": "Correlation IDs – mandatory boredom, without which everything breaks down when debunking", "tag": "Observability", "description": "It is almost always useful to carry through the system \"request id\", \"thread id\", \"run id\" and \"tool call id\". Then you can link the log, graph run, streaming session, alert and user complaint into one chain. Without this, debugging an LLM service turns into guesswork.", "blocks": [ { "kind": "code", "label": "simple circuit", "language": "text", "value": "request_id -> web request\nthread_id -> conversation line\nrun_id -> specific execution\ntool_call_id -> tool invocation" } ] }, { "number": "25", "title": "There are two types of budgets: latency budgets and cost budgets.", "tag": "Observability", "description": "For an LLM service, it is useful to decide in advance what maximum latency you allow and what price you are willing to pay for the request. This affects model choice, allowed tools, max history length, retry count, and when workflow should be led to queue instead of synchronous response.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "BUDGET THINKING\n\nlatency budget\n How many seconds can you wait?\n\ncost budget\n How many tokens/money can be spent\n\nbudget\n Form model selection policy and depth of workflow" } ] }, { "number": "26", "title": "Careful with cardinality: Don't make log storage metrics", "tag": "Observability", "description": "LLM systems are very tempting to drag in labels 'user id', 'prompt hash', 'tool name', 'tenant id', 'thread id' and everything. But high-cardinal metrics quickly become expensive and heavy. Deep context is often best left in logs and tracks rather than metric labels.", "blocks": [ { "kind": "list", "label": "rule", "items": [ "metrics = units", "Traces/logs = detail", "user id as metric label everywhere" ] } ] } ] }, { "number": "06", "title": "Scaling, Rollouts and Rollbacks", "intro": "", "cards": [ { "number": "27", "title": "HPA is a control loop, not a magical continuous elasticity.", "tag": "Scaling", "description": "Kubernetes HPA periodically looks at metrics and adjusts the desired replica count. This is useful for an LLM service, but it is important to remember that autoscaling is not instantaneous and does not solve everything by itself. If the queue is already crowded or the model provider has hit the rate limit, HPA will not fix the architecture.", "blocks": [ { "kind": "code", "label": "important idea", "language": "text", "value": "Autoscaling helps when:\n More replicas really increase throughput\n\nAutoscaling does not help when:\n bottleneck outside your pods\n shared provider rate limit\n queue policy broken\n state only in RAM" } ] }, { "number": "28", "title": "Concurrency limits and backpressure are needed before autoscaling", "tag": "Scaling", "description": "If a service can accept an infinite number of heavy requests, it will kill itself and external dependencies faster than it can scale. Therefore, it is useful to have limits on parallel runs, queue lengths, and a “fast reject rather than slow meltdown” strategy.", "blocks": [ { "kind": "list", "label": "normalization", "items": [ "max concurrent runs", "max queue depth", "429 / overload response", "per-tenant limits" ] } ] }, { "number": "29", "title": "Rolling deploy must respect readiness, otherwise you get false stability.", "tag": "Scaling", "description": "If the new version starts receiving traffic before it is actually ready, rollout will look “successful” on the processes, but customers will see errors. That’s why readiness probes and the right startup path are so important: they become part of a rollout strategy, not just health endpoints.", "blocks": [ { "kind": "code", "label": "healthy rollout", "language": "text", "value": "new pod starts\n→ startup probe passes\n→ readiness passes\n→ only then traffic shifts\n→ old pod drains and exits" } ] }, { "number": "30", "title": "Rollback should be simple and fast, otherwise it is useless.", "tag": "Scaling", "description": "Kubernetes Deployment knows how to roll out the previous revision, and this is a very practical idea: the rollback operation should be understandable and cheap in advance. For LLM-service, this also means the versioning of prompt/graph behavior, so that the rollback is not only in the image of the container, but also in the behavior of the system.", "blocks": [ { "kind": "list", "label": "worth knowing", "items": [ "rollback image revision", "rollback prompt/graph revision", "Quickly return to known-good state" ] } ] }, { "number": "31", "title": "Canary and staged rollout are particularly useful for LLM behavior changes.", "tag": "Scaling", "description": "When not only code changes, but also prompt policy, tools, routing logic, or model provider, it’s helpful not to transfer all traffic right away. Even a small percentage of canary can quickly show that latency, cost, or answer quality have left more than expected.", "blocks": [ { "kind": "list", "label": "look out", "items": [ "error rate", "latency", "cost/request", "human review rate" ] } ] } ] }, { "number": "07", "title": "Secrets, MCP and Final Frame", "intro": "", "cards": [ { "number": "32", "title": "Secrets is not just an env vars, but a separate risk zone.", "tag": "Security", "description": "Kubernetes docs specifically emphasize encryption at rest, RBAC, least privilege and external secret stores. For an LLM service, this is especially important because it often has provider API keys, DB creds, vector DB tokens, tool credentials, and sometimes MCP/OAuth secrets.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "A good model for SECRET-OVs\n\nstore securely\n ↓\nlimit access narrowly\n ↓\ninject into only needed containers\n ↓\nrotate when needed\n\nSecrets should not be smeared by image, code and random env dumps" } ] }, { "number": "33", "title": "Least privilege is important for both containers and tool credentials", "tag": "Security", "description": "An LLM tool layer often has access to external systems. Therefore, operational security here means not only “hide the key”, but also limit scope, namespace, container access and tools rights. Quick injection quickly turns into blast radius.", "blocks": [ { "kind": "list", "label": "limit", "items": [ "secret scope", "tool permissions", "namespace / tenant access", "avoid overpowered default tokens" ] } ] }, { "number": "34", "title": "If the system has an MCP, it must have a separate operational boundary.", "tag": "MCP", "description": "MCP transport and authorization are not “tiny parts of tools.” If your service uses remote MCP servers or displays the MCP interface itself, this separate operational surface: transport, auth/OAuth, trusted origins, allowed tools/resources, audit and rate limits should be designed separately from the conventional HTTP product API.", "blocks": [ { "kind": "code", "label": "right-hand", "language": "text", "value": "public app API\n !=\nMCP tool/data plane\n\nThey may have:\n clientele\n different models\n rate limits\n blast radii" } ] }, { "number": "35", "title": "MCP auth should be understood as a full protocol concern.", "tag": "MCP", "description": "The current MCP specification already describes the authorization flow separately. For production, this means that if you connect a remote MCP server, don’t leave auth for later. The protocol has its own security story, and it needs to be given as much attention as the bearer auth of a conventional FastAPI API.", "blocks": [ { "kind": "list", "label": "remember", "items": [ "auth flow is protocol-level", "token handling matters", "separate trust boundary" ] } ] }, { "number": "36", "title": "The main summary of the volume", "tag": "Ops Frame", "description": "For an LLM service to live, it is not enough to “raise FastAPI.” Need: edge with correct HTTPS/proxy behavior, honest probes, external durable state, queue for long tasks, tracing and budgets, controlled scaling/rollback and neat work with secrets and MCP boundary. This is the real devops framework around an LLM application.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "FULL DEVOPS FRAME\n\nedge + TLS + proxy correctness\n +\nruntime probes + graceful lifecycle\n +\nexternal state + queues\n +\nobservability + budgets\n +\nscaling + rollback\n +\nsecrets + MCP security\n =\nLive LLM service, not local demo" } ] } ] } ] }