Deployment & DevOps for an LLM Service

When the service goes out, its reliability is determined not only by the graph code. There are edge proxies, TLS, readiness, steady state, queues, timeouts, token budget, tracing and rollback strategy. The real system becomes multi-layered.

In dev, you just run uvicorn. In production, you need to roll out, heat, check, scale, limit, observe and be able to roll back. This is the main difference between operation and local launch.

A conventional API often has pain in the CPU or DB. The LLM service adds to this tokens, latency external model providers, the cost of reranker/tool loops and the length of the spoken state. The ops model is closely related to budgets and guardrails.

That's a key operational idea. If the web-process is almost stateless, and threads, checkpoints, queues and artifacts are brought out, then the service is easier to scale, transfer between nodes and experience restarts. For LLM runtime, this is especially important because of the ‘thread id’ and durable execution.

The process may be alive, but not ready to receive traffic: graph/service objects are not yet available, persistence is not available, configuration is not warmed up, client adapters are not raised. Therefore, “healthz” and “readyz” should almost always be separated.

Kubernetes Ingress and similar edge layers solve several tasks at once: set HTTP/HTTPS routes outward, route traffic to the internal Service, know TLS termination, name-based routing and load balancing. This is a separate, very important operational layer before the application.

FastAPI docs separately remind: HTTPS and certificates live below the HTTP layer. In practice, this often means that TLS ends on proxy/ingress, and before Uvicorn, the application receives internal HTTP traffic. Therefore, certificates, renewals, and SNI are usually served not by an application, but by an edge component.

Behind-proxy is not just about beauty. If a proxy puts HTTPS and path prefix, and the application doesn’t trust forwarded headers or doesn’t know about “root path”, it starts generating incorrect redirect URLs and documentation along the wrong paths.

LLM services often stream tokens, tool progress, and updates. This means that ingress, proxy and client timeouts must be aligned with streaming pattern. Otherwise, the service “works”, but edge cuts off the connection before the user sees the answer.

Some policies can be done in the application, but operationally more convenient when at least part of the protection and normalization of traffic is at the input: origin policy, TLS, IP filtering, basic rate limiting, size limiting. Especially if the service is open to the browser or external customers.

FastAPI docs recommends looking at the deployment context. In orchestrators, it is usually more convenient to run one Uvicorn process per container and scale the number of replicas. It is easier for memory model, ‘lifespan’, graceful shutdown and durability logic around thread state.

Uvicorn and Gunicorn solve the problem of process control, but this does not change the fact that every worker has his own memory. For an LLM service, this is important: in-memory state, local caches, and lazily initialized clients must be designed with multiprocess behavior in mind.

Kubernetes docs are very clear about their roles. Readiness is responsible for receiving traffic. Liveness is responsible for restarting a really unhealthy container. Startup probe is useful when the application takes a long time to climb, so that premature checks do not consider it a breakdown.

Hello-world readiness can be a formality. For an LLM service, it is useful to check that the main service objects are collected, persistence is available and the application can safely accept new run-s. It doesn’t have to be a full synthetic chat, but it’s not just a ‘return 200’ without meaning.

If the replica leaves during a long stream or background job handoff, you need to correctly terminate connections, stop receiving new traffic and not lose your fortune. This is another reason to keep the application state and queue out of the process and respect the lifecycle container.

If a task can take tens of seconds or minutes, call a lot of tools, generate reports, or do reindex, it’s best to switch to the job/worker pattern. In this case, the HTTP endpoint creates a run and queues the task, and then the client receives status/result in a separate way.

Queue retry, client retry and internal tool retry are easily overlapped. If side-effect tools are non-idempotent, you can send a letter several times, create a ticket or write off money. Therefore, retry should be designed with idempotency strategy.

Some tasks should not be retracted indefinitely: prompt bug, malformed payload, denied external dependency, broken tool schema. In such cases, you need a clear path for a failed job: error status, DLQ or manual parsing, rather than perpetual retry storm.

Human-in-the-loop flow doesn’t have to live in a synchronized API. Very often, the pause occurs inside the worker-run, and the “resume” comes later. Durable execution and queue here complement each other, not conflict.

LLM systems often rely not only on CPU/Memory, but on the accumulation of heavy tasks. Therefore, queue length, age of oldest job, retry count and throughput workers often speak about the health of the system better than just HTTP latency web service.

OpenTelemetry ecosystem is useful precisely because it helps to think not about one “logger”, but about three classes of signals. Logs are good for a specific incident. Metrics are good for aggregate health. Traces are good for a request that has passed through API, graph, tool calls and external dependencies.

A simple request trace is not enough. It is useful to see at least: HTTP span, graph invocation, model call, tool calls and external I/O dependencies. Then you can understand where the latency was born: in the model, retriever-e, tool chain or persistence.

This is something that teams often forget without LLM experience. Metrics like 'request duration seconds' are useful, but without 'prompt tokens', 'completion tokens', 'tool calls per run', 'cost per request' and 'cost per tenant' picture operational remains blind.

It is almost always useful to carry through the system "request id", "thread id", "run id" and "tool call id". Then you can link the log, graph run, streaming session, alert and user complaint into one chain. Without this, debugging an LLM service turns into guesswork.

For an LLM service, it is useful to decide in advance what maximum latency you allow and what price you are willing to pay for the request. This affects model choice, allowed tools, max history length, retry count, and when workflow should be led to queue instead of synchronous response.

LLM systems make it tempting to attach `user_id`, `prompt_hash`, `tool_name`, `tenant_id`, `thread_id`, and similar values to metric labels. These high-cardinality dimensions quickly become expensive and difficult to query. Keep deep request context in logs and traces rather than metric labels.

Kubernetes HPA periodically looks at metrics and adjusts the desired replica count. This is useful for an LLM service, but it is important to remember that autoscaling is not instantaneous and does not solve everything by itself. If the queue is already crowded or the model provider has hit the rate limit, HPA will not fix the architecture.

If a service can accept an infinite number of heavy requests, it will kill itself and external dependencies faster than it can scale. Therefore, it is useful to have limits on parallel runs, queue lengths, and a “fast reject rather than slow meltdown” strategy.

If the new version starts receiving traffic before it is actually ready, rollout will look “successful” on the processes, but customers will see errors. That’s why readiness probes and the right startup path are so important: they become part of a rollout strategy, not just health endpoints.

Kubernetes Deployment knows how to roll out the previous revision, and this is a very practical idea: the rollback operation should be understandable and cheap in advance. For LLM-service, this also means the versioning of prompt/graph behavior, so that the rollback is not only in the image of the container, but also in the behavior of the system.

When not only code changes, but also prompt policy, tools, routing logic, or model provider, it’s helpful not to transfer all traffic right away. Even a small percentage of canary can quickly show that latency, cost, or answer quality have left more than expected.

Kubernetes docs specifically emphasize encryption at rest, RBAC, least privilege and external secret stores. For an LLM service, this is especially important because it often has provider API keys, DB creds, vector DB tokens, tool credentials, and sometimes MCP/OAuth secrets.

An LLM tool layer often has access to external systems. Therefore, operational security here means not only “hide the key”, but also limit scope, namespace, container access and tools rights. Quick injection quickly turns into blast radius.

MCP transport and authorization are not “tiny parts of tools.” If your service uses remote MCP servers or displays the MCP interface itself, this separate operational surface: transport, auth/OAuth, trusted origins, allowed tools/resources, audit and rate limits should be designed separately from the conventional HTTP product API.

The current MCP specification already describes the authorization flow separately. For production, this means that if you connect a remote MCP server, don’t leave auth for later. The protocol has its own security story, and it needs to be given as much attention as the bearer auth of a conventional FastAPI API.

For an LLM service to live, it is not enough to “raise FastAPI.” Need: edge with correct HTTPS/proxy behavior, honest probes, external durable state, queue for long tasks, tracing and budgets, controlled scaling/rollback and neat work with secrets and MCP boundary. This is the real devops framework around an LLM application.