What happened
In a Habr article, AI engineer Daniil Sheremet of Agentic Lab describes building entity extraction for Agata, the company’s legal assistant and workspace. The intended workload is large case folders containing contracts, amendments, payment documents, claims, motions, and correspondence. The system needs to identify parties, contracts, payments, breaches, claims, and other facts so lawyers can search across a matter without reading each file individually. The team says client materials cannot simply be sent to external APIs, and dedicated high-end GPU capacity is expensive.
The pipeline asks a model nine separate questions per document, one for each entity class: participants, document metadata, contracts, dispute objects, performance events, payment events, breach events, damages, and communications. Each request supplies a class-specific schema and the document text; the output is JSON. The team had initially used Ministral 3 14B on an A100, averaging about 33 seconds per document. It then fine-tuned Qwen2.5-1.5B-Instruct with a LoRA adapter, whose trainable addition is described as about 80 MB. Base weights remained frozen. The training set came from about 5,500 real documents supplied with client consent; a teacher model labeled the corpus, yielding roughly 45,000 document-and-class examples.
The reported comparison is on a golden set of 145 real documents excluded from training and synthetic-data generation. An evaluator distinct from the teacher compared extracted results with reference labels. The small model reached macro-F1 0.70 versus 0.71 for Ministral 3 14B. On an A5000, the tuned model processed a document 2.2 times faster than Ministral on an A100. These figures describe the team’s setup and test, not a general model ranking.
The article also details a failure in handling class imbalance. A naive attempt to balance data with synthetic examples did not solve the problem, and the model initially missed correspondence: in 94% of cases it said that correspondence was absent. The source emphasizes that this could look formally correct when examples are sparse, while being operationally wrong. It describes changes to the data approach, but the supplied excerpt does not include the full concluding account of those changes or enough information to reproduce all final training decisions. The article’s stated objective is an on-premise deployment that can fit smaller hardware, not a claim that legal documents should be processed without human review.
What the archive adds
This is a new deployment case rather than an update to our earlier model-evaluation coverage. The archive’s Artificial Analysis benchmark note argues that inspectable benchmark components matter, while their task mix may still not match a team’s deployment. Here, the test is more specific to one legal workflow, but likewise cannot establish performance elsewhere.
Trend signal
This case weakly supports agent economics moves to completed work: the comparison is framed around processing a document at a measured quality level, rather than model size alone. It does not report cost per verified completed legal task, so it is evidence of a practical direction, not proof that the economics have shifted to that unit.
Why it matters
For teams with sensitive documents, the choice is not simply between a large hosted model and no automation. This case suggests that task-specific extraction can be distilled into a smaller model and run locally, potentially reducing latency and hardware pressure while preserving near-baseline scores on a defined test. That can make it feasible to reserve expensive accelerators for interactive work instead of batch extraction.
The more important operational lesson is to evaluate the errors that affect the workflow, not only an aggregate score. Missing communications can hide relevant evidence even when overall macro-F1 looks strong. Teams should measure class-level recall, test realistic document mixes, and preserve human review for consequential legal outputs. The reported speedup is useful as a design signal, but no general savings or production reliability can be inferred from the comparison alone.
Limits and uncertainty
The evidence comes from one company’s article and a 145-document golden set; the source does not establish independent replication, confidence intervals, or performance across jurisdictions, firms, document formats, or languages. It reports macro-F1, but the supplied material does not fully show per-class metrics or the final correction strategy for correspondence. The latency comparison also changes both model and GPU, so it cannot isolate the effect of model size. The article does not provide a complete serving-cost analysis, a price, or production error rates. Client consent and the described data handling apply to this dataset, not automatically to other organizations.
Reader takeaway
A small tuned model may handle a narrow, privacy-sensitive extraction task well enough to be useful on more modest hardware—but only against a carefully defined test. Reproduce the evaluation on your own documents, inspect rare-class failures, and treat the F1 and speed figures as a case study rather than a promise.
Open question
How did Agentic Lab ultimately change its data strategy to address missed correspondence, and did that improvement hold on a larger, independently labeled test set?