🔍 Read the full analysis: What A Second AI Review Revealed About Claude’s Model Edit on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
A review by OpenAI’s GPT-6 Astra found reproducible problems in a Claude-led experiment that edited the weights of the local model Ornith 1.5. The fixes addressed stale or empty benchmark runs, a broken command-line path and weight changes during conversion; the reviewer also found that four of six provenance checks failed in the second round. The review did not establish that the edit improved legal performance: the reported legal score remained zero.
A review by OpenAI’s GPT-6 Astra identified reproducible faults in an experiment led by Anthropic’s Claude Opus 5.5 to edit the weights of the local model Ornith 1.5. The review found that benchmark runs could reuse old results or finish with no evaluations, and that converting weights altered them even when the edit strength was set to zero; the reported legal score on the published tasks remained zero.
The experiment’s author, Thorsten Meyer, says the Claude session developed a directional weight edit intended to weaken a “loop direction” in Ornith 1.5, without training the model. In the updated account, the edited model answered all 10 published Lexassay tasks, compared with four for the unedited model, while its legal score stayed at zero. The runs are publicly available on Lexassay. These figures describe the reported benchmark outcomes, not evidence of improved legal ability.
GPT-6 Astra, running in Codex, was assigned a review role rather than a second model-improvement experiment. It wrote separate checks and reported findings; the Claude session made the changes. According to Meyer, the reviewer did not inspect hidden benchmark prompts, topics, answers or reference solutions. Its first review, on November 4, identified four problems: resuming could treat answers from a different setup as current, an unknown task ID could produce a successful run with no evaluations, a real command failed despite green parser tests, and converting weights changed values even at zero edit strength.
The author says the implementation was changed in response: unknown task IDs are rejected, the conflicting command-line separators were replaced, and edited tensors are stored in higher-precision bf16 rather than being re-quantized to the model’s compact 6-bit format. A later review round reported that all 12 of the author’s tests passed, while four of the reviewer’s six provenance checks failed. The figures concern checks of the experiment’s records and process, not the model’s legal accuracy.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
What the Review Changed
The review matters because a green test suite does not by itself show that a benchmark run measured what its operators intended. Reusing an answer from a different model or setting could make a comparison unreliable; accepting an unknown task ID could make an empty run look complete. Finding these faults before relying on the results gives readers a clearer account of the experiment’s limits.
The weight-conversion finding also bears on causal interpretation. Meyer reports that the first measurement at zero edit strength showed a 0.84% change in the matrix overall, with 105,315 of 1,048,576 reconstructed values differing. A later measurement, after the review and changes, found a different-format conversion changed 0.51% and 915,360 of 1,048,576 values. These are measurements of conversion, not legal quality or an error rate. They show why the edit’s effect cannot be separated from the handling of weights without suitable controls.
The review also challenged the wording used to describe the results. Meyer says claims that weights were “exactly identical at strength zero” and that the evidence established a causal effect from one control were too strong. The distinction matters to anyone evaluating model-editing claims: observed benchmark counts, a causal explanation and practical capability are separate conclusions, each requiring its own evidence.
Popular right now
De'Longhi Rivelia Automatic Espresso Machine with Grinder and Milk Frother, Beige
Bose QuietComfort Wireless Noise-Cancelling Headphones, Sandstone
Samsung 48 4K OLED Smart TV (2026) with WireLogic 8ft Ultra High Speed HDMI Cable
As an affiliate, we earn on qualifying purchases.
How the Two AI Roles Split
The work began on November 4, when two AI sessions were involved in the same broader effort. The roles were then divided. Claude Opus 5.5 handled the plan, test rig, proposed loop direction, weight edit, benchmark runs and blind grading by three Claude agents. GPT-6 Astra reviewed the implementation without altering the main rig and without running a competing model edit. The reviewer’s own report, as quoted in Meyer’s article, says: “Codex did not run a competing model-improvement experiment.”
The experiment was inspired by a post from MIT’s Computer Science and Artificial Intelligence Laboratory about programming in weight space. The source account describes the edit as a directional ablation, not a training run. Meyer says early preparatory files from the GPT-6 Astra session were set aside after the roles were divided, and that no second edited model resulted from them. The published benchmark consists of 10 Lexassay tasks; the updated article says the runs are public.
The source presents this as an AI experiment and review, not legal advice. Its benchmark result should be read within that scope: answering a task is not the same as providing legally correct advice, and a reported legal score of zero does not support a claim of legal improvement.
“Review and validate the other session’s implementation.”
— GPT-6 Astra, in its review instructions as quoted by Meyer
Limits of the Review Evidence
The published account does not establish that the weight edit improved the model’s legal performance. Although the edited model answered all 10 tasks versus four for the unedited model, the legal score remained zero, and the source does not provide enough detail here to assess the tasks’ representativeness or the reliability of that result beyond the reported runs.
Meyer also identifies limits in the second review. GPT-6 Astra did not re-run every test during its final documentation step, was not a fourth blind grader, and measured weight conversion on a CPU rather than the production GPU path. The source does not say whether GPU measurements would match the reported CPU measurements. It also says four of six provenance checks failed, but the supplied account does not give the full findings for each check.
The reviewer’s checks and measurements are reported by the experiment’s author; the source does not describe an external audit of the review itself. It remains unclear how much of the benchmark difference, if any, can be attributed to the directional edit rather than other factors in the test setup. The reported control and the conversion issue make broad causal claims premature.
Public Runs and Further Checks
The immediate next step available to readers is to examine the public Lexassay runs, including the weight-edited results, the edit control and model-card sampling, as cited in Meyer’s updated article. Those records may help readers check the reported task counts and compare the experiment’s conditions.
The source does not announce a scheduled follow-up, a new model edit or a further review round. Additional checks on the production GPU path, fuller reporting of the provenance findings and controls that isolate the edit from weight conversion would help clarify the remaining questions. Until then, the most supported conclusion is narrower: the second AI review found process and measurement problems, prompted corrections, and did not establish improved legal performance.
Key Questions
What did the second AI review examine?
GPT-6 Astra, running in Codex, reviewed the implementation and experiment process used in a Claude-led edit of Ornith 1.5. It wrote separate checks rather than changing the main implementation or running a competing model edit.
What faults did the first review find?
It found that a resumed run could reuse answers from a different setup, an unknown task ID could yield a successful run with zero evaluations, the real command line could fail despite passing parser tests, and weight conversion changed values even when edit strength was zero.
Did the model become better at legal tasks?
The source reports that the edited model answered 10 of 10 published tasks, compared with four for the unedited model, but its legal score remained zero. The reported result does not establish improved legal capability.
What does the 0.51% figure measure?
It measures the reported change from storing a real weight matrix in a different number format: 0.51% of the matrix overall, with 915,360 of 1,048,576 values changed. It is not a legal score, an error rate or a measure of model quality.
What remains to be checked?
The source says the final review did not rerun every test, measurements used a CPU rather than the production GPU path, and four of six provenance checks failed. The extent to which the benchmark difference can be attributed to the edit remains unclear.
Source: ThorstenMeyerAI.com
