What happened
AI Engg's account follows an eight-day November CodinGame contest. The author set three goals for a bot: understand the game board, run within 50 milliseconds, and perform better in real games when its network looked better during training. Claude agents wrote the code for the model, training loop and tests, while the author designed the harness. The transformer was not the whole bot: a search process improved plans, considered likely opponent replies, and used the network as a judge to score candidate positions.
The board became a sequence of tokens: one for each house, one for each triangle, and one for the whole board. House tokens represented units, ownership and position; triangle tokens represented ownership and whether either side could capture them. The model also received learned biases for relationships between tokens, such as distance between houses or a house being a triangle corner. It predicted a side's chance of winning and the likely margin. In a mid-game turn, the network evaluated about 64 positions.
The first training approach was distillation: a small network learned to score positions like a slower, stronger search. On 4 November, adding opponent plans helped that bot beat the previous version 177–79 and reach the author's best Silver score, 34.65. Self-play followed on 5 November. Those bots rose from a reference rating of 1,500 to 1,758, but the first online result was 17.54, placing 198th of 426 in Gold; the previous bot had scored 16.77 and placed 172nd. On 7 November, training instead used 1,918 games from top players. That bundle beat the previous build 248–120–32 and improved placement by 1.20 points, though a lead guard and timeout fix also contributed.
The contest required one source file of no more than 100,000 characters and a 50-millisecond turn. The live student had about 99,000 parameters, was trained for 7-bit weights, and packed into about 18,010 characters using an entropy coder. A variant exceeded the file limit at 102,999 characters and was discarded. A 42-minute final run distilled five smaller networks from 1.47 million positions, with 14,111 real positions held out for testing. The selected d24L2 model had 9,515 parameters, ran about nine times faster than the live student, chose the teacher's top move on 79.8% of held-out positions, and had 0.990 value correlation with the teacher.
Those offline measures did not settle usefulness. Against the live bot, d24L2 with deeper search won 64.5% in 96 games sharing a machine, but 55.7% in 48 games run one per core; it failed opening tests and placed 25.69 online. The author reports that only five of eleven network and reinforcement-learning ideas passed a test against the live bot, and only one improved placement. At equal compute, the bot's best-response decision rule won 67.6%; ten times the search won 54.8%.
What the archive adds
New Runtime's earlier coverage, “The model that didn't exist, so you made it yourself”, dated 2026-10-24, described using ML Intern to create and publish six small custom models, including a 0.8B model trained for $16 to run on a CPU. This is a new event, but a related practical account of building models around constraints; it adds contest-level evaluation, compression and runtime trade-offs rather than another model-creation workflow.
Trend signal
This does not yet fit a pattern New Runtime tracks. It is a detailed individual case in which full-system evaluation changes the conclusion drawn from standalone model metrics, but one contest bot is not enough to establish a broader shift.
Why it matters
A model's training score is an intermediate measurement, not proof that the application improved. Here, self-play raised the bots' internal rating but did not produce a better online rank. A policy that beat its parent 819–203 when both played alone scored only 46.0% when placed inside the search system. Even a compact student with strong agreement with its teacher stumbled in opening tests and online play.
For teams, the useful lesson is to evaluate the complete decision path under its actual constraints: inputs, model, search or orchestration, concurrency, latency budget and final outcome. The case also shows that model choice is only one lever. A better decision rule produced a larger gain than multiplying search compute by ten. This is not a general benchmark or proof that transformers are unnecessary; it is a concrete reminder to compare changes against a working baseline in the environment where users or competitors will experience them.
Limits and uncertainty
The results concern one CodinGame contest bot and its particular search, game and hardware setup. The account does not establish that the same model architecture, distillation approach or training sequence would work for another task. Some comparisons bundled multiple changes: the 7 November result also included a lead guard and timeout fix, so the network's individual contribution is not isolated. The author reports contest outcomes, but they are not an independent replication. Nor does the account quantify the total engineering time or cost of the experiments.
Skill to build now
This week, choose one model change in a latency- or resource-constrained workflow and test it in the full application against the current version. Record both model-level measures and end-to-end outcomes, such as win rate, task success and runtime under realistic concurrency. Keep a fixed test set and an explicit acceptance threshold. If offline scores improve but the complete system does not, treat that as a failed change rather than a reason to ship the model anyway.
Reader takeaway
The durable lesson is not that a small transformer will improve every application. It is that model quality has to be judged where the model is used. Translate the task into a suitable representation, respect deployment limits, and measure the complete system against a baseline before trusting training metrics.