{ "type": "new-runtime-knowledge-guide", "version": 1, "generated_at": "2026-07-23", "volume": 24, "slug": "fastapi-langgraph-langchain-microservice", "title": "Mini-Repository: FastAPI + LangGraph + LangChain as a Service", "description": "This volume does not answer the question “how to write another agent”, but the question “how to make a live external application”. At the center is one solid Python mini-repo: ‘FastAPI’ as a public HTTP/SSE layer, ‘LangGraph’ as runtime, ‘LangChain’ as a layer of models, tools and middleware, external persistence for threads, scaling, auth, health/readiness and a separate place for ‘MCP’ as a tool/data plane.", "track": { "slug": "application-engineering", "title": "Application Engineering", "route": "https://newruntime.com/learn/tracks/application-engineering/", "position": 4 }, "counts": { "sections": 7, "cards": 36 }, "routes": { "html": "https://newruntime.com/learn/fastapi-langgraph-langchain-microservice/", "json": "https://newruntime.com/learn/fastapi-langgraph-langchain-microservice.json" }, "sections": [ { "number": "01", "title": "The Big Picture of Service", "intro": "", "cards": [ { "number": "01", "title": "Public application = ‘FastAPI’ outside, ‘LangGraph’ inside, ‘LangChain’ as a layer of components", "tag": "Architecture", "description": "If you remove the excess magic, the external LLM microservice looks like a normal backend. FastAPI accepts queries and gives answers/streams. LangGraph manages stateful execution: threads, routing, interrupts, checkpoints. LangChain provides models, messages, tools, middleware and structured output. Everything else is the infrastructure around this core.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "External layer and internal RUNTIME\n\nbrowser / frontend / other service\n ↓\n FastAPI\n routes / auth / schemas / stream\n ↓\n Agent Service\n ↓\n LangGraph\n state / checkpoints / interrupts\n ↓\n LangChain\n model / tools / middleware\n ↓\n DB / vector DB / queue / object storage / MCP / external APIs" } ] }, { "number": "02", "title": "What makes a service a “microservice”", "tag": "Architecture", "description": "It's not code size or Docker. Microservice is made by an external contract, a separate deploy lifecycle, its own health/readiness rules, its own persistence logic and the ability to safely serve customers from the outside as an independent service.", "blocks": [ { "kind": "list", "label": "minimum set", "items": [ "stable API", "separate deploy", "state/storage", "auth / monitoring / retries" ] } ] }, { "number": "03", "title": "This service usually has three modes of interaction.", "tag": "Architecture", "description": "Almost all production scenarios fit into three forms: the usual synchronous request/response, streaming response, and long-lived flow with interrupt/resume or background processing. If you think through all three modes in advance, the architecture becomes more sustainable.", "blocks": [ { "kind": "code", "label": "three modes", "language": "text", "value": "1. sync:\n I have been waiting for JSON to come back.\n\n2. stream:\n request → on the way to give tokens / updates\n\n3. durable:\n Request → saved state → pause / worker / resume later" } ] }, { "number": "04", "title": "Outside, it is better to expose business operations, not internal node-s of the count.", "tag": "Architecture", "description": "The client doesn’t usually need to know which “retrieve node”, “review node” or “tool router” you have. Externally, it is better to give APIs like \"chat\", \"resume review\", \"search knowledge\", \"summarize document\". The internal graph remains your implementation, not part of a public contract.", "blocks": [ { "kind": "list", "label": "signal", "items": [ "API = business activities", "API = internal node names" ] } ] }, { "number": "05", "title": "Thread id is quickly becoming part of an external API.", "tag": "Architecture", "description": "As soon as the service supports multi-turn, streaming history or ‘interrupt/resume’, the outside world starts to refer to specific threads. This means that ‘thread id’, ‘run id’ or their counterparts should be designed consciously, rather than generating randomly ‘inside’.", "blocks": [ { "kind": "code", "label": "practice", "language": "text", "value": "client sends:\n thread_id? maybe existing\n\nservice decides:\n new thread or continue old thread\n\nresponse returns:\n thread_id\n maybe interrupt payload\n maybe run metadata" } ] } ] }, { "number": "02", "title": "One Whole Mini-Repo", "intro": "", "cards": [ { "number": "06", "title": "One of the normal layouts for this repository", "tag": "Repo", "description": "The purpose of the structure is to separate the public API, LangGraph runtime, LangChain components and infrastructure. Then the service remains readable and does not turn into one giant 'main.py' with routes, prompts, tools and SQL in a jumble.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "REPO TREE\n\napp/\n main.py\n core/\n settings.py\n security.py\n observability.py\n api/\n deps.py\n routers/\n chat.py\n resume.py\n health.py\n agent/\n contracts/\n state.py\n context.py\n schemas.py\n components/\n models.py\n prompts.py\n tools.py\n middleware.py\n mcp_clients.py\n graph/\n nodes.py\n routes.py\n build.py\n services/\n chat_service.py\n stream_service.py\n infra/\n persistence.py\n queue.py\n mcp_server.py\ntests/\nDockerfile\npyproject.toml" } ] }, { "number": "07", "title": "'main.py' must raise resources once through 'lifespan'", "tag": "Repo", "description": "Graph, checkpointer, store, database pool, model clients and service objects are better collected at the start of the application, not for every request. For this, FastAPI has a ‘lifespan’, and it is one of the most important practical pieces for a production service.", "blocks": [ { "kind": "code", "label": "skeleton", "language": "python", "value": "@asynccontextmanager\nasync def lifespan(app: FastAPI):\n settings = load_settings()\n checkpointer, store = await build_persistence(settings)\n graph = build_graph(checkpointer=checkpointer, store=store)\n app.state.chat_service = ChatService(graph=graph, settings=settings)\n yield\n\napp = FastAPI(lifespan=lifespan)" } ] }, { "number": "08", "title": "‘api/routers/*.py’ – a thin layer above services", "tag": "Repo", "description": "An HTTP route does not need to know how a graph works. Its job is simpler: fail the input, pull out the auth/tenant context, invoke the service method, and return the JSON or stream. This keeps the HTTP code small and replaceable.", "blocks": [ { "kind": "list", "label": "good way", "items": [ "schema validation", "auth dependency", "service call", "complex orchestration logic" ] } ] }, { "number": "09", "title": "‘agent’ is the heart of the product, not the technical folder", "tag": "Repo", "description": "Inside the “agent” live domain contracts, models, tools, middleware, graph nodes and service orchestration. This folder is responsible for the business behavior of the application. This is where most of the real value of the service is concentrated.", "blocks": [ { "kind": "code", "label": "logic", "language": "text", "value": "contracts -> state / context / response schema\ncomponents -> model / tools / middleware / prompt layer\ngraph -> nodes / routes / build\nservices -> invoke / stream / resume orchestration" } ] }, { "number": "10", "title": "‘infra/’ is best highlighted explicitly because production dependencies live there.", "tag": "Repo", "description": "Persistence, queue, MCP server/client glue, metrics, logging setup, and other operational code are growing rapidly. If you do not take them into a separate layer, they will mix with the graph code and begin to complicate both the development and the depot.", "blocks": [ { "kind": "list", "label": "often lives infra", "items": [ "checkpointer/store builders", "queue adapters", "MCP server bootstrap", "logging/tracing setup" ] } ] } ] }, { "number": "03", "title": "Public HTTP API", "intro": "", "cards": [ { "number": "11", "title": "‘POST /v1/chat’ is the most understandable sync input", "tag": "HTTP API", "description": "If the user needs a regular response without streaming, it is convenient to have one sync endpoint. Inside, it builds a 'thread id', runtime context, calls 'graph.invoke(...)' through the service, and returns a validated 'response model' along with metadata.", "blocks": [ { "kind": "code", "label": "example", "language": "python", "value": "@router.post(\"/v1/chat\", response_model=ChatResponse)\nasync def chat(req: ChatRequest, svc = Depends(get_chat_service), ctx = Depends(get_request_context)):\n return await svc.chat(req, ctx)" } ] }, { "number": "12", "title": "Streaming is a separate endpoint rather than a “magic option”.", "tag": "HTTP API", "description": "LangGraph can stream tokens and state updates. Outwardly, it is convenient to give it through “StreamingResponse”, usually in the format of “text/event-stream” or chunked JSON lines. A separate endpoint makes the contract clearer and doesn’t force sync clients to understand the streaming protocol.", "blocks": [ { "kind": "code", "label": "example", "language": "python", "value": "@router.post(\"/v1/chat/stream\")\nasync def chat_stream(req: ChatRequest, svc = Depends(get_stream_service), ctx = Depends(get_request_context)):\n return StreamingResponse(\n svc.stream(req, ctx),\n media_type=\"text/event-stream\",\n )" } ] }, { "number": "13", "title": "If there is an ‘interrupt’, there is a ‘resume’ endpoint.", "tag": "HTTP API", "description": "This is one of the most common flaws. People add approval-node with ‘interrupt()’, but forget to design the external path of continuation. Production requires a separate endpoint that accepts 'thread id' and payload for 'Command'.", "blocks": [ { "kind": "code", "label": "example", "language": "python", "value": "@router.post(\"/v1/threads/{thread_id}/resume\")\nasync def resume(thread_id: str, req: ResumeRequest, svc = Depends(get_chat_service), ctx = Depends(get_request_context)):\n return await svc.resume(thread_id=thread_id, payload=req.payload, ctx=ctx)" } ] }, { "number": "14", "title": "`/healthz`, `/readyz`, `/version`, `/openapi.json` are not trifles", "tag": "HTTP API", "description": "A live service needs service entry points. Liveness says “process is alive,” readiness says “you can take traffic,” version says “what’s locked in right now,” OpenAPI says “how customers integrate.” These endpoints are quickly becoming a must for the platform team.", "blocks": [ { "kind": "list", "label": "minimum", "items": [ "GET /healthz", "GET /readyz", "GET /version", "GET /openapi.json" ] } ] }, { "number": "15", "title": "Auth, Tenant Context and CORS are better solved at FastAPI", "tag": "HTTP API", "description": "If a service lives on the outside, it almost always needs API keys, bearer tokens or other auth-mechanism. It usually gives rise to the request context: “user id”, “tenant id”, roles, feature flags. For browser clients, CORS is added. All this is more convenient to decorate the dependencies layer from FastAPI.", "blocks": [ { "kind": "code", "label": "idea", "language": "text", "value": "Authorization header\n→ verify principal\n→ build RequestContext(user_id, tenant_id, scopes)\nPut it in the service/graph context" } ] }, { "number": "16", "title": "'Idempotency-Key' is especially important if a graph can cause side effects.", "tag": "HTTP API", "description": "This is one of the “non-obvious but highly relevant” production units. If the client repeats the request because of the timeout, and inside the graph sends an email, creates a ticket or writes in CRM, without idempotence, you will easily get duplicate actions.", "blocks": [ { "kind": "list", "label": "particularly", "items": [ "resume after timeout", "tool with external side effect", "repeat client retries" ] } ] } ] }, { "number": "04", "title": "LangGraph and LangChain Inside the Service", "intro": "", "cards": [ { "number": "17", "title": "`state.py`, `context.py`, `schemas.py` are the main contracts of the service", "tag": "LangGraph", "description": "The internal runtime should be typed as well as the external HTTP API. “state” describes the living state of the graph, “context” – immutable launch data like “user id” and tenancy, “schemas” – the boundaries of structured output and API. This makes graph code less fragile and easier to test.", "blocks": [ { "kind": "code", "label": "state", "language": "python", "value": "class AgentState(TypedDict):\n messages: Annotated[list[AnyMessage], add_messages]\n retrieved_docs: list[str]\n final_answer: dict | None\n approval_required: bool | None" } ] }, { "number": "18", "title": "The `LangChain' within such a repos are models, tools, middleware and output contracts.", "tag": "LangGraph", "description": "Even if you have orchestration on LangGraph, LangChain is still very appropriate as a layer of finished components. It is home to ‘init chat model’, tools with ‘ToolRuntime’, middleware for model/tool hooks, messages and structured output schemes.", "blocks": [ { "kind": "code", "label": "component layer", "language": "text", "value": "model = init_chat_model(...)\ntools = [search_docs, create_ticket, get_profile]\nmiddleware = [dynamic_prompt, tool_error_handler]\nresponse_schema = AnswerSchema" } ] }, { "number": "19", "title": "The graph should be compiled once at the start, not for every HTTP request.", "tag": "LangGraph", "description": "`builder.compile(...)' assembles runnable graph with checkpointer and store. If you do this for every call, you’ll get extra complexity, lost productivity, and unstable resource initialization. The service usually receives a compiled graph from lifespan.", "blocks": [ { "kind": "code", "label": "build_graph", "language": "python", "value": "def build_graph(checkpointer, store):\n builder = StateGraph(AgentState)\n builder.add_node(\"agent\", agent_node)\n builder.add_node(\"review\", review_node)\n builder.add_conditional_edges(\"agent\", route_after_agent)\n return builder.compile(checkpointer=checkpointer, store=store)" } ] }, { "number": "20", "title": "ChatService links HTTP layer and graph runtime", "tag": "LangGraph", "description": "This is the place where HTTP payloads are born ‘inputs’, ‘config’ and runtime ‘context’, and then called ‘graph.invoke(...)’ or ‘graph.astream(...)’. Such a service layer is useful because it is convenient to hold thread policy, metadata, error mapping and response shaping.", "blocks": [ { "kind": "code", "label": "logic", "language": "typescript", "value": "config = {\"configurable\": {\"thread_id\": thread_id}}\ncontext = RequestContext(...)\n\nresult = await graph.ainvoke(\n {\"messages\": [{\"role\": \"user\", \"content\": req.message}]},\n config=config,\n context=context,\n)" } ] }, { "number": "21", "title": "Interrupt() is no longer an internal feature of the graph, but a part of the UX and API design.", "tag": "LangGraph", "description": "Once the graph can settle for human approval, the service should be able to beautifully return the payload to the UI or external client. This means a well-thought-out JSON contract: interrupt type, question text, draft answer, acceptable actions, and the way to resume.", "blocks": [ { "kind": "list", "label": "come back", "items": [ "thread_id", "interrupt type", "payload for UI", "resume action schema" ] } ] } ] }, { "number": "05", "title": "State, Scaling and Long Life", "intro": "", "cards": [ { "number": "22", "title": "In order for the service to live, the state must be removed from the process.", "tag": "Deploy", "description": "One of the most important production shifts: InMemorySaver and local dicts are well suited only for laptop and local development. A live service that restarts, scales and maintains multiple instances must store thread state and memory in external persistence.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Proper separation\n\napp instance A ─┐\napp instance B ─┼──→ external checkpointer / store / DB\napp instance C ─┘\n\nThen any copy can continue threading thread id" } ] }, { "number": "23", "title": "Horizontal scaling breaks if memory lives only in the workman", "tag": "Deploy", "description": "If one query hits worker A and the next thread id hits worker B, the in-memory state disappears. Therefore, for multi-worker and multi-replica scripts, checkpointer/store should be shared, and service instances should be as stateless as possible.", "blocks": [ { "kind": "list", "label": "red flag", "items": [ "thread history only in the RAM process", "Local disk as the only storage" ] } ] }, { "number": "24", "title": "Not every LLM-flow needs to live inside an HTTP request lifecycle.", "tag": "Deploy", "description": "This is another non-obvious, but really important block. If processing can take minutes, call a lot of tools, do reindex or generate a report, it is better to highlight the background job/worker pattern. HTTP endpoint then creates a run and puts the task in a queue, and the client then reads the status or streams.", "blocks": [ { "kind": "code", "label": "pattern", "language": "text", "value": "POST /v1/jobs\n→ validate request\n→ create run record\n→ enqueue task\n→ return 202 Accepted + run_id\n\nworker\n→ loads same graph/service layer\n→ executes durable flow\n→ writes status/results" } ] }, { "number": "25", "title": "Don’t put important artifacts on a local container disk.", "tag": "Deploy", "description": "Chats, traces, prompt snapshots, uploaded files, generated reports and other useful artifacts should be stored in an external storage: SQL / NoSQL, object storage, vector DB. The container or process does not have to be the only place where user data lives.", "blocks": [ { "kind": "list", "label": "dataplane", "items": [ "SQL / Postgres", "object storage", "vector DB", "queue broker" ] } ] }, { "number": "26", "title": "Timeout budgets, cancellations and backpressure are required in advance.", "tag": "Deploy", "description": "The real service has limits: upstream LLM timeout, tool timeout, client timeout, ingress timeout, limit on simultaneous streams. If they are not designed in advance, the service will begin to hang, accumulate hanging connections and fall under load.", "blocks": [ { "kind": "list", "label": "think through", "items": [ "request timeout", "tool timeout", "queue depth limit", "cancel on client disconnect" ] } ] } ] }, { "number": "06", "title": "Where is MCP Really Needed?", "intro": "", "cards": [ { "number": "27", "title": "MCP usually lies next to the microservice, rather than replacing its public API.", "tag": "MCP", "description": "One of the most useful shifts in understanding is that FastAPI and MCP are different. A public HTTP API is needed for browsers, backends, and business integrations. MCP is needed where LLM hosts and agent systems want standardized tools, resources, and prompts. It's not the same thing.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Two adjacent layers\n\npublic app API:\n browser / backend / mobile → FastAPI\n\ntool/data plane:\n host / agent platform / IDE → MCP server\n\nOften the same product has both layers." } ] }, { "number": "28", "title": "Pattern A: Your service is an MCP client to an external system n", "tag": "MCP", "description": "This is a good pattern if your LangGraph service needs standardized tools from the outside world: GitHub, Slack, files, CRM, internal knowledge, etc. Then your ‘LangChain’ tool layer can run through MCP clients rather than holding dozens of custom SDKs.", "blocks": [ { "kind": "code", "label": "what it looks like", "language": "text", "value": "FastAPI request\n→ graph node\n→ LangChain tool\n→ MCP client\n→ remote MCP server\n→ tool result back into graph" } ] }, { "number": "29", "title": "Pattern B: Your repository exports the MCP server to other agents", "tag": "MCP", "description": "If your product itself knows useful operations or has important data, you can add “mcp server.py” and give some of the features out through MCP. Then IDEs, external hosts or other agent platforms will be able to use your tools/resources/prompts in a standard way.", "blocks": [ { "kind": "code", "label": "sketch", "language": "sql", "value": "from mcp.server.fastmcp import FastMCP\n\nmcp = FastMCP(\"my-agent-platform\")\n\n@mcp.tool\ndef search_internal_docs(query: str) -> str:\n ...\n\n@mcp.resource(\"kb://policy/{name}\")\ndef get_policy(name: str) -> str:\n ..." } ] }, { "number": "30", "title": "'FastAPI' vs 'MCP' vs both at once", "tag": "MCP", "description": "It is better not to confuse these interfaces. FastAPI is good as a business API for applications and services. MCP is a standardized tool/data protocol for hosts and agents. Very often, the correct answer is to have both interfaces, but with different audiences and different areas of responsibility.", "blocks": [ { "kind": "table", "headers": [ "Interface.", "Better for you." ], "rows": [ [ "FastAPI", "frontend, backend, product API, auth, business flows" ], [ "MCP", "tools/resources/prompts for hosts and agent systems" ], [ "Both", "A platform that people and other agents need." ] ] } ] }, { "number": "31", "title": "MCP has its own security and its own trust boundary.", "tag": "MCP", "description": "If you are picking up a remote MCP server, think not only about transport, but also about auth, origin checks, tool descriptions, prompt injection through resources and rights to side-effect tools. An MCP server is a login to your tool/data plane, not just another JSON endpoint.", "blocks": [ { "kind": "list", "label": "really important", "items": [ "Streamable HTTP transport", "auth / OAuth", "least privilege tools", "prompt injection via tool/resource content" ] } ] } ] }, { "number": "07", "title": "Relevant production blocks that are often forgotten", "intro": "", "cards": [ { "number": "32", "title": "Observability should go all the way: 'request id', 'thread id', 'run id'", "tag": "Ops", "description": "Logs “received the request” and “response” are not enough. For an LLM service, it is very useful to carry correlation identifiers through the entire stack: HTTP request id, thread id, run id, tool call id. Then you can associate a specific client failure with a specific graph run and a specific execution tool.", "blocks": [ { "kind": "code", "label": "ids", "language": "text", "value": "request_id -> HTTP request\nthread_id -> conversation / state line\nrun id -> specific graph launch\ntool call id -> external tool invocation" } ] }, { "number": "33", "title": "Versioning not only API, but also graph behavior", "tag": "Ops", "description": "Even if the endpoint is called the same, the behavior of the service may change due to a new prompt, new tool rules, different routing logic, or other output schema. Therefore, it is useful to have a clear version of the API, prompt/graph revision and migration discipline for customers.", "blocks": [ { "kind": "list", "label": "version", "items": [ "HTTP API version", "response schema", "graph/prompt revision" ] } ] }, { "number": "34", "title": "Tests are required on three levels: HTTP, service, graph", "tag": "Ops", "description": "If you test only the endpoint, it will not be clear if the auth, service wiring, routing graph, or specific tool is broken. Therefore, unit tests tools/middleware, integration tests compiled graph and API tests for schemas, auth and stream/resume endpoints are useful here.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "TEST LAYERS\n\nHTTP tests\n ↓\nservice tests\n ↓\ngraph integration tests\n ↓\ntool / interrupt / resume tests" } ] }, { "number": "35", "title": "Deploy Path: Docker → reverse proxy / ingress → env-configured workers", "tag": "Ops", "description": "For external life, the service is usually containerized, raised behind proxy/ingress with HTTPS, throwing env vars and scale policy. An important engineering idea: once the state is outdoors, the application becomes much easier to scale and survive restarts.", "blocks": [ { "kind": "code", "label": "walkway", "language": "text", "value": "local dev\n→ Docker image\n→ staging with real DB/checkpointer\n→ prod behind ingress / load balancer\n→ replicas + shared persistence\n\nIn many clusters, it is easier to scale replicas than to build complex multiprocesses within a single container." } ] }, { "number": "36", "title": "The main summary of the volume", "tag": "Bridge", "description": "If you compress everything to the most useful engineering framework, the result is: FastAPI gives an external contract, LangGraph gives a long-lasting runtime, LangChain gives convenient building blocks, external persistence makes the service live, and MCP adds a standardized tool/data interface for other hosts and agents if necessary.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "One main picture\n\npublic clients\n ↓\nFastAPI API\n ↓\nservice layer\n ↓\nLangGraph runtime\n ↓\nLangChain components\n ↓\nexternal state + tools + MCP + data plane\n\nSuch a service can not only be run locally, but also actually operate." } ] } ] } ] }