Deployment & DevOps for an LLM Service

When the service goes out, its reliability is determined not only by the graph code. There are edge proxies, TLS, readiness, steady state, queues, timeouts, token budget, tracing and rollback strategy. The real system becomes multi-layered.

THE FULL CARTINE

client
  ↓
edge / ingress / TLS
  ↓
FastAPI app
  ↓
LangGraph runtime
  ↓
external state / queues / tools / models
  ↓
logs / metrics / traces / alerts

In dev, you just run uvicorn. In production, you need to roll out, heat, check, scale, limit, observe and be able to roll back. This is the main difference between operation and local launch.

A conventional API often has pain in the CPU or DB. The LLM service adds to this tokens, latency external model providers, the cost of reranker/tool loops and the length of the spoken state. The ops model is closely related to budgets and guardrails.

request rate
× average tokens
× model latency
× tool count
× retry policy
= real operational price of the service

That's a key operational idea. If the web-process is almost stateless, and threads, checkpoints, queues and artifacts are brought out, then the service is easier to scale, transfer between nodes and experience restarts. For LLM runtime, this is especially important because of the ‘thread id’ and durable execution.

Desirable model

replica A ─┐
replica B ─┼──→ shared persistence / queue / object store
replica C ─┘

Then any instance can continue the same conversation thread.

The process may be alive, but not ready to receive traffic: graph/service objects are not yet available, persistence is not available, configuration is not warmed up, client adapters are not raised. Therefore, “healthz” and “readyz” should almost always be separated.

Kubernetes Ingress and similar edge layers solve several tasks at once: set HTTP/HTTPS routes outward, route traffic to the internal Service, know TLS termination, name-based routing and load balancing. This is a separate, very important operational layer before the application.

EDGE PATH

internet
  ↓
ingress / load balancer
  ↓
service
  ↓
pod / app

FastAPI docs separately remind: HTTPS and certificates live below the HTTP layer. In practice, this often means that TLS ends on proxy/ingress, and before Uvicorn, the application receives internal HTTP traffic. Therefore, certificates, renewals, and SNI are usually served not by an application, but by an edge component.

Behind-proxy is not just about beauty. If a proxy puts HTTPS and path prefix, and the application doesn’t trust forwarded headers or doesn’t know about “root path”, it starts generating incorrect redirect URLs and documentation along the wrong paths.

proxy adds:
  X-Forwarded-For
  X-Forwarded-Proto
  X-Forwarded-Host

app must trust them:
  --forwarded-allow-ips

if proxy strips/adds prefix:
  configure root_path

LLM services often stream tokens, tool progress, and updates. This means that ingress, proxy and client timeouts must be aligned with streaming pattern. Otherwise, the service “works”, but edge cuts off the connection before the user sees the answer.

Some policies can be done in the application, but operationally more convenient when at least part of the protection and normalization of traffic is at the input: origin policy, TLS, IP filtering, basic rate limiting, size limiting. Especially if the service is open to the browser or external customers.

FastAPI docs recommends looking at the deployment context. In orchestrators, it is usually more convenient to run one Uvicorn process per container and scale the number of replicas. It is easier for memory model, ‘lifespan’, graceful shutdown and durability logic around thread state.

Uvicorn and Gunicorn solve the problem of process control, but this does not change the fact that every worker has his own memory. For an LLM service, this is important: in-memory state, local caches, and lazily initialized clients must be designed with multiprocess behavior in mind.

worker A memory != worker B memory

Therefore:
  state
  shared persistence
  shared queues
  Carefully with local caches

Kubernetes docs are very clear about their roles. Readiness is responsible for receiving traffic. Liveness is responsible for restarting a really unhealthy container. Startup probe is useful when the application takes a long time to climb, so that premature checks do not consider it a breakdown.

Three samples

startupProbe
  Let the app start quietly

readinessProbe
  Do not allow traffic until the service is ready

livenessProbe
  Restart the container if it really freezes or breaks

Hello-world readiness can be a formality. For an LLM service, it is useful to check that the main service objects are collected, persistence is available and the application can safely accept new run-s. It doesn’t have to be a full synthetic chat, but it’s not just a ‘return 200’ without meaning.

If the replica leaves during a long stream or background job handoff, you need to correctly terminate connections, stop receiving new traffic and not lose your fortune. This is another reason to keep the application state and queue out of the process and respect the lifecycle container.

stop accepting traffic
finish or safely abort in-flight work
flush logs/traces
exit cleanly

If a task can take tens of seconds or minutes, call a lot of tools, generate reports, or do reindex, it’s best to switch to the job/worker pattern. In this case, the HTTP endpoint creates a run and queues the task, and then the client receives status/result in a separate way.

JOB FLOW

POST /jobs
  ↓
validate + persist job
  ↓
enqueue
  ↓
worker executes graph
  ↓
status/result store
  ↓
client polls or streams status

Queue retry, client retry and internal tool retry are easily overlapped. If side-effect tools are non-idempotent, you can send a letter several times, create a ticket or write off money. Therefore, retry should be designed with idempotency strategy.

Some tasks should not be retracted indefinitely: prompt bug, malformed payload, denied external dependency, broken tool schema. In such cases, you need a clear path for a failed job: error status, DLQ or manual parsing, rather than perpetual retry storm.

try job
→ retry N times on transient failures
→ mark failed on terminal error
→ send to DLQ / alert / manual review

Human-in-the-loop flow doesn’t have to live in a synchronized API. Very often, the pause occurs inside the worker-run, and the “resume” comes later. Durable execution and queue here complement each other, not conflict.

worker executes graph
→ interrupt reached
→ state persisted
→ UI asks human
→ human confirms
→ resume event arrives
→ worker or service continues same thread

LLM systems often rely not only on CPU/Memory, but on the accumulation of heavy tasks. Therefore, queue length, age of oldest job, retry count and throughput workers often speak about the health of the system better than just HTTP latency web service.

OpenTelemetry ecosystem is useful precisely because it helps to think not about one “logger”, but about three classes of signals. Logs are good for a specific incident. Metrics are good for aggregate health. Traces are good for a request that has passed through API, graph, tool calls and external dependencies.

Three Signals

logs, what exactly happened?
How the system behaves on average
How one request went through the system

A simple request trace is not enough. It is useful to see at least: HTTP span, graph invocation, model call, tool calls and external I/O dependencies. Then you can understand where the latency was born: in the model, retriever-e, tool chain or persistence.

HTTP request
→ chat service
→ graph run
→ model call
→ tool call
→ db / external api
→ response serialization

This is something that teams often forget without LLM experience. Metrics like 'request duration seconds' are useful, but without 'prompt tokens', 'completion tokens', 'tool calls per run', 'cost per request' and 'cost per tenant' picture operational remains blind.

It is almost always useful to carry through the system "request id", "thread id", "run id" and "tool call id". Then you can link the log, graph run, streaming session, alert and user complaint into one chain. Without this, debugging an LLM service turns into guesswork.

request_id  -> web request
thread_id   -> conversation line
run_id      -> specific execution
tool_call_id -> tool invocation

For an LLM service, it is useful to decide in advance what maximum latency you allow and what price you are willing to pay for the request. This affects model choice, allowed tools, max history length, retry count, and when workflow should be led to queue instead of synchronous response.

BUDGET THINKING

latency budget
  How many seconds can you wait?

cost budget
  How many tokens/money can be spent

budget
  Form model selection policy and depth of workflow

LLM systems are very tempting to drag in labels 'user id', 'prompt hash', 'tool name', 'tenant id', 'thread id' and everything. But high-cardinal metrics quickly become expensive and heavy. Deep context is often best left in logs and tracks rather than metric labels.

Kubernetes HPA periodically looks at metrics and adjusts the desired replica count. This is useful for an LLM service, but it is important to remember that autoscaling is not instantaneous and does not solve everything by itself. If the queue is already crowded or the model provider has hit the rate limit, HPA will not fix the architecture.

Autoscaling helps when:
  More replicas really increase throughput

Autoscaling does not help when:
  bottleneck outside your pods
  shared provider rate limit
  queue policy broken
  state only in RAM

If a service can accept an infinite number of heavy requests, it will kill itself and external dependencies faster than it can scale. Therefore, it is useful to have limits on parallel runs, queue lengths, and a “fast reject rather than slow meltdown” strategy.

If the new version starts receiving traffic before it is actually ready, rollout will look “successful” on the processes, but customers will see errors. That’s why readiness probes and the right startup path are so important: they become part of a rollout strategy, not just health endpoints.

new pod starts
→ startup probe passes
→ readiness passes
→ only then traffic shifts
→ old pod drains and exits

Kubernetes Deployment knows how to roll out the previous revision, and this is a very practical idea: the rollback operation should be understandable and cheap in advance. For LLM-service, this also means the versioning of prompt/graph behavior, so that the rollback is not only in the image of the container, but also in the behavior of the system.

When not only code changes, but also prompt policy, tools, routing logic, or model provider, it’s helpful not to transfer all traffic right away. Even a small percentage of canary can quickly show that latency, cost, or answer quality have left more than expected.

Kubernetes docs specifically emphasize encryption at rest, RBAC, least privilege and external secret stores. For an LLM service, this is especially important because it often has provider API keys, DB creds, vector DB tokens, tool credentials, and sometimes MCP/OAuth secrets.

A good model for SECRET-OVs

store securely
  ↓
limit access narrowly
  ↓
inject into only needed containers
  ↓
rotate when needed

Secrets should not be smeared by image, code and random env dumps

An LLM tool layer often has access to external systems. Therefore, operational security here means not only “hide the key”, but also limit scope, namespace, container access and tools rights. Quick injection quickly turns into blast radius.

MCP transport and authorization are not “tiny parts of tools.” If your service uses remote MCP servers or displays the MCP interface itself, this separate operational surface: transport, auth/OAuth, trusted origins, allowed tools/resources, audit and rate limits should be designed separately from the conventional HTTP product API.

public app API
  !=
MCP tool/data plane

They may have:
  clientele
  different models
  rate limits
  blast radii

The current MCP specification already describes the authorization flow separately. For production, this means that if you connect a remote MCP server, don’t leave auth for later. The protocol has its own security story, and it needs to be given as much attention as the bearer auth of a conventional FastAPI API.

For an LLM service to live, it is not enough to “raise FastAPI.” Need: edge with correct HTTPS/proxy behavior, honest probes, external durable state, queue for long tasks, tracing and budgets, controlled scaling/rollback and neat work with secrets and MCP boundary. This is the real devops framework around an LLM application.

FULL DEVOPS FRAME

edge + TLS + proxy correctness
  +
runtime probes + graceful lifecycle
  +
external state + queues
  +
observability + budgets
  +
scaling + rollback
  +
secrets + MCP security
  =
Live LLM service, not local demo