🔍 Read the full analysis: How We Kept Legal Scores At Zero During AI Weight-Space Editing on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
A weight edit reduced looping in the local AI model Ornith 1.5 35B-A3B, allowing it to answer all 10 published Lexassay tasks instead of 4 in the comparison run. Its legal score remained zero, and its edited answers contained 55 wrong or invented citations. The experiment suggests the edit helped the model finish, not improve its legal knowledge.
A home experiment using a weight edit helped the local AI model Ornith 1.5 35B-A3B finish more legal benchmark tasks, but did not improve its legal score: it answered 10 of 10 published tasks instead of 4 in the reported comparison, while scoring zero both before and after the edit. The result, published by ThorstenMeyerAI.com and updated November 9, illustrates a limit of fixing model behavior: reducing repetitive loops can help a system complete an answer without making that answer accurate.
The experiment was inspired by an MIT CSAIL report on “weight-space programming,” but it used a different method. In the MIT work, GPT-6 Astra reasoned directly about another model’s weights. In this test, a Claude Opus 5.5 session wrote code that ran Ornith, measured a recurring internal signal, and removed that signal at half strength from the output weights of 27 of the model’s 40 layers. The authors describe this as directional ablation, a technique that does not require training or adding legal data.
On the 10 published Lexassay tasks, the edited model completed all 10, compared with four in the same-setup comparison. But the content remained poor: it met 7 of 50 checked criteria, up from three, and its legal score stayed at zero. The 10 edited answers included 55 wrong or invented citations, according to the report. Lexassay checks legal references against primary sources and covers German, U.S. and EU law.
The report also identifies a basic-settings issue: using the sampling settings recommended on the model card, the unedited model answered 9 of 10 tasks on its original setup. That result is not directly comparable with the four-of-10 figure because the runs used different setups. The experiment’s answers, grades and citation checks are publicly available on Lexassay, according to the author.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Completion Is Not Legal Accuracy
The central finding is a distinction between finishing a response and producing a reliable one. When a model repeatedly cycles through similar text, an edit that reduces the loop may make it usable for more requests. But this experiment offers no evidence that the edit supplied missing legal knowledge or improved the validity of legal authorities.
That distinction matters to anyone evaluating AI for legal work. A completed answer can look more useful than a cut-off one, yet the reported 55 faulty or invented citations show why completion counts cannot stand in for factual checks. Lexassay’s citation review provides a separate measure of whether legal references match primary sources. The reported zero score is a warning against treating a behavioral improvement as evidence of professional competence.
The result is also a small-scale test, not proof that directional ablation will reliably improve other models. It involved one run per setting and a limited set of public tasks. The report itself says the improvement was technical: the model stopped failing to finish as often, while the benchmark’s substantive result remained poor.
Popular right now
KitchenAid Artisan Mini Tilt-Head Stand Mixer, 3.5 Qt, Contour Silver
Roborock RockMow X130H Wire-Free 4WD Robot Lawn Mower
LG 77-Inch OLED evo C6 4K Smart TV
As an affiliate, we earn on qualifying purchases.
The MIT Method and Local Test
The MIT CSAIL work that prompted the test was reported by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. Its best-known example involved a Qwen model that had been stuck in loops. The MIT team reported that an edit removed a problematic direction and raised the model’s math score from 7.9% to 48.1%. The source article quotes the researchers cautioning that an edit might remove an obstacle hiding an ability, but cannot add an ability the model lacks.
The home experiment tested that idea against a different failure. Ornith was selected because its benchmark runs showed extended repetition: two answers did not finish within a 62,000-token limit, and the final portions of the transcripts contained few distinct lines. The model’s publisher describes it as a coding and agent model, rather than one built for legal work. Other models considered for the test either did not show the same looping behavior or could not be run under comparable conditions.
Lexassay asks models the same 15 questions across German, U.S. and EU law, without internet access or tools. Ten tasks are published and five are kept hidden, according to the report. The figures in this experiment refer to the published tasks only. The author also disclosed that Claude carried out the edit and Claude agents performed blind grading; GPT-6 Astra, from another company, reviewed the code, figures and claims and identified faults.
“The model finishes is not the answer is right.”
— Thorsten Meyer, author of the experiment report
Limits of the Reported Comparison
The figures do not establish how consistently the edit would work. The report says there was one run per setting, the development set was small, and the loop detector changed during the experiment. Three graders came from the same model family, while the independent review was a review of the work rather than a second experiment.
The reported completion counts also depend on setup. The unedited model answered eight of 10 tasks in a published run, but four of 10 in the other setup used for the direct comparison; with model-card sampling settings, it answered nine of 10 on its original setup. These results are not directly comparable. The article does not establish whether the edited model would retain its behavior across new prompts, hidden Lexassay tasks or other hardware and sampling configurations.
It remains unclear whether the slight increase in checked criteria, from three to seven of 50, reflects a repeatable improvement or variation in a single run. The source report also does not present evidence that the edit reduced legal citation errors; the edited answers still contained 55 wrong or invented citations. The article describes the work as an AI experiment, not legal advice.
The published runs provide a record of the model’s answers, grades and citation checks, allowing readers to inspect the reported comparison. Lexassay’s five hidden tasks remain separate from the 10 published tasks, according to the report, and could help test whether results extend beyond the development set without exposing those questions for tuning.
No follow-up experiment or broader evaluation is announced in the source material. Further runs under matched sampling settings, repeated trials and checks on hidden tasks would be needed to determine whether the edit consistently reduces looping. Any assessment of legal usefulness would also need to focus on answer accuracy and verified citations, not just whether the model completes its response.
Key Questions
What changed after the weight edit?
On the reported same-setup comparison, Ornith completed 10 of 10 published tasks, up from four. The experiment removed a measured internal direction from output weights; it did not train the model on legal material.Did the model become better at law?
The report does not show a meaningful legal-score improvement: the score remained zero. The edited answers met seven of 50 checked criteria and included 55 wrong or invented citations.Was this the same method used by the MIT team?
No. The MIT report describes an AI reasoning directly over model weights. This experiment used code to run the model, measure an internal direction and remove it through directional ablation.Can the results be generalized to other models?
Not from the evidence reported. The test involved one model and one run per setting, with a small task set and comparison conditions that were not all directly comparable.Where can readers check the experiment?
ThorstenMeyerAI.com says the runs are public on Lexassay, including answers, grades and citation checks. The reported figures refer to the 10 published tasks; five additional tasks are kept hidden.Source: ThorstenMeyerAI.com
