--- type: "new-runtime-knowledge-guide" stable_id: "knowledge_guide:llm-frameworks" version: 1 generated_at: "2026-07-23" record_date: "2026-07-23" date_kind: "generated_at" slug: "llm-frameworks" title: "The LLM Framework Ecosystem" description: "Key abstractions, architectural patterns, pitfalls and selection criteria for the main frameworks of the LLM ecosystem. Each one is disassembled atomically." retrieval_nugget: "Key abstractions, architectural patterns, pitfalls and selection criteria for the main frameworks of the LLM ecosystem. Each one is disassembled atomically. LangChain is an abstraction layer on top of the LLM API. His philosophy: everything is a component, components are connected through pipes into chains (LCEL)." track: "application-engineering" volume: 3 --- # The LLM Framework Ecosystem ## Retrieval answer Key abstractions, architectural patterns, pitfalls and selection criteria for the main frameworks of the LLM ecosystem. Each one is disassembled atomically. LangChain is an abstraction layer on top of the LLM API. His philosophy: everything is a component, components are connected through pipes into chains (LCEL). ## 01. LangChain — Orchestration Foundation ### 01.01 LangChain - key abstractions LangChain is an abstraction layer on top of the LLM API. His philosophy: everything is a component, components are connected through pipes into chains (LCEL). The main value is not in the “magic”, but in uniform interfaces for 100+ providers and tools. #### Diagram ```text Key primitives: PromptTemplate → prompt template with variables ChatModel → wrapper for LLM API (OpenAI, Anthropic, ...) OutputParser → output parsing: str/JSON/Pydantic Retriever → search abstraction (vector, BM25, hybrid) Tool → called tool with description for LLM Memory → conversation context storage Chain → an outdated way to connect components LCEL pipe → prompt | model | parser ← modern way ``` #### When to buy LangChain - rapid prototype RAG - need integration with 50+ providers - the team already knows LangChain - complex stateful agent → LangGraph - production without overhead → pure SDK ### 01.02 LCEL — LangChain Expression Language A modern way to assemble chains is through the | operator. Provides streaming, batching, tracing and async out of the box for any chain. Replaces the old Chain classes. #### LCEL Syntax ```typescript # Simple RAG chain chain = ( {"context": retriever, "question": RunnablePassthrough()} | prompt_template | chat_model | StrOutputParser() ) # Everything works automatically: chain.invoke(query) # synchronously chain.stream(query) # token streaming chain.batch([q1, q2, q3]) # in parallel await chain.ainvoke(query) #async ``` ### 01.03 LangChain - pitfalls LangChain is often criticized for its abstractions that hide what's going on under the hood - and this is what causes most problems in production. #### Common problems ```text ✗ Magic prompts Default prompts inside Chain classes → unpredictable, impossible to control ✗ Hidden LLM challenges RetrievalQA does extra. calls unnoticed → unexpected costs, hidden latency ✗ Versioning Frequent breaking changes between minor versions → pin version, test updates carefully ✓ Solution: LCEL + explicit prompts + tracing everything ``` ## 02. LangGraph — Stateful Agent Graphs ### 02.04 LangGraph - philosophy and abstractions LangGraph is a way to describe agents as state machines with explicit state. Not chains, but a graph: nodes are actions/LLM calls, edges are transitions between states. This gives something that LangChain does not: loops, branches, human-in-the-loop, persistent state. #### Diagram ```text Agent graph: ReAct with human-in-the-loop [START] ↓ [agent] ← LLM decides: call the tool or give a response ↙ ↘ [tools] [END] ↓ [agent] ← tool result → next solution Key idea: cycles are possible - the agent can spin as much as needed until it solves the problem or hits the limit ``` #### When to take LangGraph instead of LangChain - agent with loops and branching - need human-in-the-loop (pause, confirmation) - persistent state between sessions - multi-agent with clear orchestration ### 02.05 State + Reducer is the heart of LangGraph Each node reads and writes to a common State. Reducer determines how the state is updated when there is a conflict (append vs replace). This makes the graph reproducible and debuggable. #### Example State ```sql from typing import Annotated from langgraph.graph import add_messages class AgentState(TypedDict): messages: Annotated[list, add_messages] # add_messages = reducer: append, not replace retrieved_docs: list[str] iteration_count: int final_answer: str | None # each node receives state and returns patch: def agent_node(state: AgentState) -> dict: response = llm.invoke(state["messages"]) return {"messages": [response]} ``` ### 02.06 Conditional Edges - flow control Edges can be conditional: a function looks at the state and decides which node to go to next. This allows you to build complex logic without imperative code inside nodes. #### Branching example ```python def should_continue(state) -> str: last_msg = state["messages"][-1] if last_msg.tool_calls: return "tools" # → tools node elif state["iteration_count"] > 5: return "force_end" # → force stop else: return "end" # → final response graph.add_conditional_edges("agent", should_continue, {"tools": "tools", "end": END, "force_end": END}) ``` ### 02.07 Checkpointing - persistence and HITL LangGraph can save state after each step in the database (Postgres, SQLite, Redis). This gives: continuation after a failure, human-in-the-loop (pause → wait for confirmation → continue), time travel (rollback to the previous state). #### Human-in-the-loop pattern ```python checkpointer = PostgresSaver(conn) graph = graph.compile( checkpointer=checkpointer, interrupt_before=["dangerous_action"] ← pause here ) # 1. Run before interrupt graph.invoke(input, config={"thread_id": "t1"}) # 2. Show the person a pending action state = graph.get_state(config) # 3. The person approves → continue graph.invoke(None, config) # resume ``` ## 03. LlamaIndex — RAG & Data Layer ### 03.08 LlamaIndex - philosophy and abstractions LlamaIndex is focused on one thing: connecting LLM with your data. While LangChain is a general-purpose orchestration tool, LlamaIndex is a specialized data layer. The main primitive is Index and QueryEngine. #### Diagram ```sql LlamaIndex abstractions: Document → raw document + metadata Node → document chunk (atomic index unit) Index → search structure (VectorStore, Summary, KG...) Retriever → strategy for retrieving from Index NodePostprocessor → reranker, filters, metadata enrichment QueryEngine → retriever + LLM + response synthesis ChatEngine → QueryEngine + dialogue history Pipeline → IngestionPipeline for indexing ``` #### When LlamaIndex beats LangChain - complex RAG logic (parent-doc, multi-hop) - many data sources (PDF, Notion, SQL, ...) - need advanced indexes (KG, Summary) - orchestration is more difficult → LangGraph is better ### 03.09 Types of indexes - not just vectors LlamaIndex offers several types of indexes for different tasks. The choice of index is an architectural decision that affects the quality of answers more than the choice of model. #### Types of Indexes ```text VectorStoreIndex standard RAG for embeddings SummaryIndex summarization of the entire corpus sequentially KeywordTableIndex keyword extraction → keyword search KnowledgeGraph entity graph → search by relationships DocumentSummary summary per doc → document selection → details TreeIndex hierarchical tree of summations Multi-Index: different indexes for different types of questions ``` ### 03.10 IngestionPipeline - correct indexing Document processing pipeline before indexing: transformations, splitters, metadata, deduplication. With caching - re-indexing skips already processed documents. #### Example ```text pipeline = IngestionPipeline( transformations=[ SentenceSplitter(chunk_size=512, overlap=64), TitleExtractor(), # LLM metadata QuestionsAnsweredExtractor(), # hypothetical questions OpenAIEmbedding(), ], vector_store=chroma_store, cache=IngestionCache(), # do not reindex old ) ``` ## 04. Multi-Agent Frameworks — CrewAI · AutoGen · Agno ### 04.11 CrewAI - roles, tasks, teams CrewAI models multi-agent systems as a team with roles. Each agent is a role + a set of tools + a goal. Tasks are assigned to agents and executed sequentially or in parallel. Very readable declarative API - good for business processes where roles are obvious. #### Diagram ```text CrewAI abstractions: Agent = role + backstory + goal + tools + llm Task = description + expected_output + agent Crew = [agents] + [tasks] + process + memory Process types: sequential agent 1 → agent 2 → agent 3 hierarchical manager LLM → delegates tasks to agents ↑ better quality control Crew example for WB seller analytics: researcher → collect data on competitors analyst → build BCG matrix writer → write a report in PDF ``` #### When CrewAI - business processes with clear roles - quick start of a multi-agent prototype - complex graph → LangGraph more precise - hidden prompts inside the framework ### 04.12 AutoGen — conversational multi-agent AutoGen (Microsoft) is based on the idea of interlocutor agents: agents exchange messages with each other, like in a chat. GroupChat allows multiple agents to discuss a task until consensus. #### Key primitives ```text ConversableAgent base agent with LLM + human_input_mode AssistantAgent LLM agent without human input UserProxyAgent executes code, optionally requests a person GroupChat multiple agents + next selection manager GroupChatManager LLM decides who speaks next Pattern: Coder + Reviewer + Executor Coder writes → Reviewer criticizes → Executor launches ``` ### 04.13 Agno (ex-Phidata) - lightweight agents Agno is a minimalistic framework for agents: minimum abstractions, maximum control. Agent = LLM + tools + storage. No magic. Easy to read and easy to debug. Built-in support for memory and knowledge bases. #### Minimal example ```text agent = Agent( model=Claude(id="claude-sonnet-4"), tools=[DuckDuckGoSearch(), PythonTool()], memory=AgentMemory(db=SqliteMemoryDb()), knowledge=PDFKnowledgeBase(path="docs/"), instructions=["Answer in Russian", "Use numbers"], show_tool_calls=True, markdown=True, ) agent.print_response("WB 2025 Market Analysis") ``` #### Applicable - transparent simple agents - quick start without learning the framework ### 04.14 Comparison of multi-agent frameworks The choice between frameworks is determined not by “which is more powerful”, but by “which abstractions coincide with your task.” | Criterion | CrewAI | AutoGen | LangGraph | Agno | | --- | --- | --- | --- | --- | | Abstraction | Roles / tasks | Agent chat | Graph / state | Agent + tools | | Control | Medium | Medium | High | High | | Learning curve | Low | Medium | High | Low | | Loops | No | Partial | Yes | No | | Production readiness | Use carefully | Use carefully | Yes | Yes, with caveats | ## 05. Eval Frameworks — RAGAS · ARES · DeepEval · Braintrust ### 05.15 RAGAS - standard for RAG eval RAGAS is a specialized eval framework for RAG systems. Uniqueness: reference-free metrics - you don’t need a dataset with GT answers for most metrics. Assessment via LLM judge with atomic approvals. #### Diagram ```text What RAGAS evaluates and how: Faithfulness answer → atomic statements → NLI vs context = proportion of statements supported by context Answer Relevance answer → LLM generates N questions → embed → cos(questions, query) = to what extent the answer falls within the topic of the question Context Precision Each chunk → LLM : “Is LLM useful for an answer?” → precision@k = share of useful chunks among retrieved Context Recall [requires GT] GT → atomic statements → NLI vs context = proportion of GT facts present in the context ``` #### Applicable - RAG assessment without GT dataset - diagnostics: retrieval vs generation problem - expensive for LLM-judge on large datasets ### 05.16 DeepEval - pytest for LLM DeepEval integrates into pytest and allows you to write eval tests as unit tests. Large library of ready-made metrics + support for custom ones. Fits well into CI/CD. #### Syntax ```python @pytest.mark.parametrize("case", test_cases) def test_rag_quality(case): actual_output = rag_pipeline(case.input) assert_test( test_case=LLMTestCase( input=case.input, actual_output=actual_output, retrieval_context=case.context ), metrics=[ FaithfulnessMetric(threshold=0.8), AnswerRelevancyMetric(threshold=0.7), HallucinationMetric(threshold=0.1), ] ) ``` ### 05.17 Braintrust - eval as a product Braintrust is a platform for eval with UI: launching experiments, comparing prompt versions, history of runs, drill-down on cases. Like Weights & Biases, but for LLM pipelines. The SDK works without a platform. #### Experiment concept ```typescript Eval( name="RAG pipeline v2 vs v3", data=dataset, # list {input, expected} task=rag_pipeline, # function → output scores=[ # metrics Faithfulness, AnswerRelevancy, NumTokens, # cost tracking Latency ], experiment_name="v3-with-rerank" ) ``` ### 05.18 ARES - autonomous eval system ARES (Stanford) is a system that trains small LM classifiers for evaluation instead of expensive LLM judges. You need a small number of human tags for training - further quickly and cheaply at any volume. #### Principle ```text # One-time use: human_labels = 150 labeled examples synthetic_data = LLM.generate_training_data(docs) classifier = finetune(small_lm, human_labels + synthetic) # Next - cheap and fast: scores = classifier.score(10_000_examples) # vs LLM-judge: 100x cheaper, comparable quality ``` #### Applicable - large volumes of assessment - eval cost reduction ## 06. Observability — Langfuse · Phoenix Arize · Helicone ### 06.19 Langfuse — open-source LLM observability Langfuse is the most complete open-source tool: traces, eval, datasets, prompt management, cost tracking - in one place. Self-hosted via Docker. Integration via SDK or OpenTelemetry. #### Diagram ```sql What Langfuse can do: Traces & Spans pipeline visualization, waterfall latency Scores eval-assessments attached to trace/span Datasets golden datasets + experiments (A/B eval) Prompt Management versioning prompts + fetch at runtime Cost Tracking tokens and $ by model, user, feature Users & Sessions grouping traces by user Evaluators LLM-as-judge directly in the interface Integration: 2 lines of code from langfuse.openai import openai # drop-in replacement ``` #### Strengths - self-hosted, GDPR-compliant - prompt management + tracing in one - integration with LangChain/LlamaIndex out of the box - UI is slower than Arize at high volumes ### 06.20 Phoenix Arize - ML observability for LLM Arize Phoenix - from ML observability to LLM. Strengths: visualization of embeddings, drift detection at the vector space level, UMAP query projections. The best choice if you need to analyze retrieval quality visually. #### Unique features vs Langfuse ```sql Embedding visualization UMAP/TSNE projections: see query clusters → find anomalies and OOD queries visually Retrieval analysis scatter plot: query embeddings vs doc embeddings → see where retrieval misses Drift detection comparison of distributions between periods → automatically finds data drift OpenInference open tracing standard LLM (like OTEL) ``` ### 06.21 Helicone — gateway + observability Helicone works as a proxy gateway between your code and the LLM API. Zero-instrumentation: you change one URL, you get all the metrics. Built-in caching, rate limiting, cost alerts. #### Connection in 1 line ```text client = OpenAI( base_url="https://oai.helicone.ai/v1", # ← instead of api.openai.com default_headers={ "Helicone-Auth": f"Bearer {HELICONE_KEY}", "Helicone-Cache-Enabled": "true", # cache "Helicone-User-Id": user_id, # per-user } ) ``` #### Applicable - no time for instrumentation - need cache + rate limit immediately - less flexibility than Langfuse ### 06.22 OpenTelemetry for LLM - tracing standard OpenInference (Arize) and OpenLLMetry (Traceloop) extend OpenTelemetry to be LLM-specific. The span attributes are standardized: input/output, model, tokens. Allows you to send traces to any OTEL-compatible backend. #### Standard attributes OTEL LLM span ```text gen_ai.system = "anthropic" gen_ai.request.model = "claude-sonnet-4" gen_ai.usage.input_tokens = 1240 gen_ai.usage.output_tokens = 312 gen_ai.prompt = [messages] gen_ai.completion = [response] # → goes to Langfuse / Jaeger / Grafana Tempo ``` ## 07. DSPy — Prompt Optimization as Code ### 07.23 DSPy — philosophy: program, don't prompt DSPy reverses the approach: instead of writing prompts manually, you describe the task signature (inputs → outputs) and quality metrics, and the optimizer itself finds the best prompts, few-shot examples and call chains. This is a compiler for LLM programs. #### Diagram ```typescript Traditional approach: the prompt was written manually → works on GPT-4 → the model was changed → the prompt is broken DSPy approach: signature = "question → answer" # what we do metric = lambda pred, gt: f1(pred, gt) # how to measure optimizer.compile(program, metric, trainset) # DSPy generates prompts itself + few-shot examples # re-optimizes when changing models in minutes Key primitives: Signature InputField + OutputField = declarative specification Module dspy.Predict / dspy.ChainOfThought / dspy.ReAct Optimizer BootstrapFewShot/MIPRO/BayesianSignatureOptimizer Metric function (prediction, ground_truth) → float ``` #### When DSPy beats manual prompting - the base model changes frequently - there is a dataset with a metric - the prompt can be optimized - multi-hop tasks with multiple LLM calls - no dataset → DSPy will not help - high entry threshold ### 07.24 DSPy Modules - building blocks DSPy modules are typed LLM call patterns. Each module is independently optimized and can be combined into programs. #### Main modules ```text dspy.Predict direct call by signature, without CoT dspy.ChainOfThought adds a reasoning field → response # prompt with CoT is generated automatically dspy.ReAct agent with tools based on the ReAct pattern dspy.Retrieve search in the connected knowledge base dspy.MultiChainComparison several reasoning paths → the best ``` ### 07.25 TextGrad & OPRO - DSPy alternatives TextGrad (Stanford) - "gradient descent" for text: LLM calculates the "gradient" (critique) and updates the prompt. OPRO (Google) - LLM as an optimizer: generates prompt candidates, evaluates, improves iteratively. #### TextGrad idea ```text loss = metric(output, gt) # "how bad" gradient = LLM( # "why is it bad" f"Output: {output} Loss: {loss} What's wrong with the prompt?" ) new_prompt = LLM( # "fix" f"Old prompt: {prompt} Gradient: {gradient} Improved prompt:" ) ``` ## 08. Navigating the ecosystem - when to take what ### 08.26 Decision Tree - choosing a framework Ask yourself these questions before choosing a framework. A wrong choice at the beginning costs correspondence later. #### Diagram ```text What are you building? ├── RAG with data → LlamaIndex (indices) + LangChain (orchestration) ├── Stateful agent with cycles → LangGraph ├── Multi-agent business process → CrewAI (fast) / LangGraph (reliable) ├── Research/agent dialogue → AutoGen ├── Simple agent, control is important → Agno / pure SDK └── Prompts to optimize auto → DSPy What are you assessing? ├── RAG (without GT) → RAGAS ├── LLM tests in pytest → DeepEval ├── Experiments / A/B prompts → Braintrust └── Small team, all at once → Langfuse (datasets + experiments) What are you monitoring? ├── Traces + prompts + cost → Langfuse (self-hosted) ├── Embeddings drift + retrieval → Phoenix Arize └── Zero instrumentation, tomorrow → Helicone ``` ### 08.27 Antipattern - a framework for the sake of a framework The most common mistake: they take LangChain/LlamaIndex “because everyone uses it” - and get an abstraction layer that hides bugs, slows down debugging and creates a dependence on other people’s breaking changes. #### When a framework is not needed ```text ✗ One or two LLM challenges → direct SDK (anthropic, openai) is better ✗ The team does not know the framework → time to study > time to write yourself ✗ Specific logic in every step → abstractions get in the way, you write workarounds ✓ The framework is justified when: → 10+ integrations are needed (providers, vector databases) → complex stateful graph (LangGraph) → the team already knows him ``` ### 08.28 Stack for 2025 - what really works Community opinion based on the results of production experience: which combinations give the least number of problems with the greatest flexibility. #### Recommended Stacks ```text Minimalistic (small team): SDK + Instructor + Langfuse RAG product: LlamaIndex + LangGraph (agent) + RAGAS + Langfuse Multi-agent enterprise: LangGraph + LangChain tools + Braintrust + Arize Research/Experiments: DSPy + Langfuse + DeepEval Main principle: start with the minimum → add frameworks only when the pain without them is obvious ```