🔍 Read the full analysis: 42 AI Setups, 15 Legal Questions, And An 8/100 Score For The Law Model on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
A test run by ThorstenMeyerAI.com gave GLM-5.3 FlashX Legal 8 out of 100 across 15 questions on German, U.S. and EU law. The author says the model fully solved none of the tasks and that citation errors were widespread, but the results come from a small spot check with disclosed grading and ownership conflicts.
GLM-5.3 FlashX Legal scored 8 out of 100 in a legal AI test published November 4 by ThorstenMeyerAI.com, which says it ran 42 AI setups against 15 questions spanning German, U.S. and EU law. The test operator says the model fully solved none of the tasks and gave no answer on two, raising questions about how much confidence users should place in AI marketed for professional legal work.
The test, called Lexassay, gives models the same 15 legal questions: 10 published and five hidden. According to Meyer, the questions cover deadline calculations, recent legal changes, leading court decisions and false-premise questions. Models answer without internet access or tools, so the test is intended to measure what they can produce without searching documents or the web.
Meyer reports that GLM-5.3 FlashX Legal, which Z.ai describes as optimized for professional legal workflows, scored 8/100 overall and 0 on the hidden questions. Lexassay counted 21 citations as real, 25 as misattributed and one as invented. On two German-law tasks, the model used its stated 64,000-token reasoning budget and produced no answer, according to the report. Its base model scored 3, a three-point difference that does not establish a broad performance advantage.
The reported results also put other systems in a wide range. Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Mistral Large 4 scored 10; Meyer says it got about half the substance right but lost most of its points over citations. The article reports that the leading model fully met every scoring criterion on eight of the 10 published tasks, rather than all of them.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Citation Errors Matter
The results matter because legal answers can turn on a precise rule, a valid citation or a deadline calculated for a particular place. A confident but incorrect answer may mislead someone about whether they can appeal, what a court has decided, or what compensation might be available. The article’s example of German appeal deadlines illustrates the risk: a public holiday in one state can change the final date there without changing the date elsewhere.
Lexassay’s results are a warning about using a chatbot as a substitute for legal review, not proof that every answer from the tested systems is wrong. The test covers a limited set of questions, and its no-tools setup differs from legal products that retrieve and analyze supplied documents. Meyer advises readers to check cited laws and judgments against their sources and not to rely on a chatbot for deadlines or matters involving money or an authority.
The citation totals are also relevant to anyone evaluating AI for legal workflows. Across published runs, Meyer says Lexassay confirmed 2,644 citations as real and identified 804 as invented or not supporting the model’s claim. These are counts reported by the test operator; the article does not give a broader industry-wide rate or a comparison baseline.
Popular right now
KitchenAid 5.5-Qt Bowl-Lift Stand Mixer, Porcelain White (KSM55SXXX)
TCL 65-Inch T7 Series 4K QLED Smart Google TV, 144Hz (65T7)
De'Longhi Dinamica Plus Fully Automatic Espresso Machine
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Lexassay scores answers against a reference answer, or “gold answer,” using three to six weighted criteria per task. The method described by Meyer says citations are checked against official databases and classified as real, misattributed, invented or not findable. A citation that exists but does not support the claim is treated differently from one that cannot be found. The scoring rules also deduct points for citation problems and for answers that lack a legal-advice disclaimer or direction to consult a lawyer.
The 42 figure refers to AI setups, not necessarily 42 distinct model families: different effort settings are tested separately, and a model may therefore appear more than once. Meyer describes Lexassay as a spot check, with one answer per model and task, and cautions that small score differences may be noise. Five questions are hidden to reduce the chance that future systems can reproduce public answers found online.
The report discloses that Thorsten Meyer AI runs Lexassay and that several Claude agents graded the answers. Because Claude Opus 5.5 ranks first, Meyer identifies that as a conflict of interest. The site says its method, public answers, reference answers and citation checks are available for scrutiny. The results should be read with that disclosure and the test’s limited size in mind.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, as quoted in the report
Limits of the Reported Scores
The results do not establish how the systems would perform with web search, legal databases, supplied case files or human review, because Lexassay tested models without internet access or tools. The source also describes only one answer per model and task, so it does not show how consistently a system would answer across repeated runs. Five hidden tasks are scored, but their answers are not publicly available in the described format.
The report includes no independent replication, and the grading arrangement creates a disclosed conflict: Meyer’s operation runs the test, Claude agents graded answers, and a Claude model led the results. The source says public answers, reference answers and citation checks are published, but the material provided does not specify an external audit of the grading. It is also unclear from the supplied report how each score would change under a different weighting of legal substance, citations and disclaimers.
The findings are not a legal assessment of any particular user’s case, and they do not show that a product using retrieval or attorney review would perform the same way. Z.ai’s full response to the score is not included in the source material. The report’s reference to another legal benchmark, where the same models rank differently, further indicates that rankings depend on test design and task conditions.
What Readers Should Check Next
Meyer says the full Lexassay method and the public answers, reference answers and citation checks are available for readers to inspect. Those materials are the next point of reference for checking how individual questions were scored and whether cited authorities support the answers. The source does not announce a date for another test or a response from Z.ai.
For readers using AI on a legal question, the practical next step is to verify each cited statute or ruling in an authoritative source and seek advice from a qualified lawyer when a deadline, financial claim or communication with an authority is involved. The test’s scores describe performance on its particular questions; they should not be treated as a guarantee about a different system, tool setup or legal matter.
Key Questions
What did GLM-5.3 FlashX Legal score?
It scored 8 out of 100 overall and 0 on the five hidden questions, according to the Lexassay report. Meyer says it fully solved none of the 15 tasks.
What did the test measure?
Lexassay tested 42 AI setups on 15 questions about German, U.S. and EU law. The models answered without internet access or tools, and their answers were graded against reference answers with citations checked against official databases.
Does the result show that legal AI cannot be used?
No. It reports how selected setups performed on this test under its stated conditions. It does not measure every legal AI product or systems that can search documents, use legal databases or receive attorney review.
Why is the report’s grading conflict disclosed?
Thorsten Meyer AI runs Lexassay, and several Claude agents graded the answers; Claude Opus 5.5 ranked first. Meyer identifies this as a conflict and says the method and public scoring materials are available for review.
Should I rely on a chatbot for a legal deadline?
The report advises against taking a deadline from a chatbot. Check the relevant law and court information, and consult a lawyer when a deadline or legal claim could affect your rights.
Source: ThorstenMeyerAI.com
