🔍 Read the full analysis: What Engineers Can Learn From Amazon’s Push To Measure AI Outcomes on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Amazon published engineering guidance on October 27 warning that token counts and other AI-use metrics can distort incentives when treated as targets. The memo says adoption data can still help identify patterns, but teams should judge AI by its effects on customers, the business and their work.
Amazon published revised guidance for software engineers on October 27, telling managers not to judge work by AI token use or tool adoption alone. The memo says activity measures can help teams spot patterns, but using them as targets risks higher costs and more complex systems without showing whether customers or the business benefit.
The memo was written by Terk Terkowitz, a vice president and distinguished engineer in Amazon’s stores business, and sets out a role guideline prepared by senior engineers. It centers engineering work on its impact on customers, the business and the team, rather than prescribing how much AI an engineer must use. The authors say detailed AI rules could quickly become outdated.
Amazon calls measures such as token consumption, AI-tool adoption, code changes and reviews “deeply flawed metrics” when they are used as goals rather than as measures. The memo argues that rewarding those inputs can increase spending and system complexity. It also says the measures currently “correlate well” with outcomes that matter, so the guidance does not call for abandoning them altogether.
Instead, managers are told to examine both low and high adoption. Looking at the 10th percentile can help explain why some teams are not using AI tools effectively; looking at the 90th percentile can test whether heavy users are delivering more. Engineers whose work does not benefit from current tools are encouraged to record why, while leaders are asked to use the tools on production work and describe where they help and where they do not.
Six months at Amazon
From token leaderboards to an outcomes memo
Amazon spent 2026 learning the same lesson twice: once with how its engineers use AI, once with its own AI models.
- May 2026Token leaderboards backfireFinancial Times (via The Decoder): staff give an internal agent tool pointless tasks to inflate token counts. Reported, anonymous sources.
- Late May 2026KiroRank “deprecated”Amazon spokesperson to Business Insider. SVP Dave Treadwell: “Please don’t use AI just for the sake of using AI.”
- 24 Jul 2026San Francisco AGI Lab site closesConfirmed by Amazon to GeekWire. Frontier model research continues under Pieter Abbeel.
- 28 Jul 2026Most flagship Nova models wound downBusiness Insider, people familiar: Premier, Omni, Reel, Canvas to “keep the lights on”. Amazon: models “remain one of the most important things we’re working on”.
- 14 / 30 Sep 2026Nova Sonic v1, Nova Reel, Nova Canvas reach end of lifeAWS Bedrock model lifecycle page. Requests to them now fail.
- 26 Oct 2026Anthropic files its S-1Amazon notes: $6.2 billion principal, 12% ownership cap that Amazon can waive. Anthropic commits at least $110 billion to Amazon through April 2036.
- 27 Oct 2026The memo: outcomes, not tokenmaxxingUsage and throughput metrics are “deeply flawed” as targets. “Tokenmaxxing is at odds with frugality.”
- 29 Oct 2026Amazon’s third-quarter resultsAnnounced by Amazon. Not yet published when this article went live.
Sources: Amazon engineering memo (27 Oct 2026); Business Insider (29 May and 28 Jul 2026); GeekWire (24 Jul 2026); The Decoder (12 May 2026, summarising the Financial Times); AWS Bedrock model lifecycle (read 28 Oct 2026); Anthropic S-1 (26 Oct 2026).
Powered by Thorsten Meyer AI
From AI Activity to Engineering Results
The guidance addresses a measurement problem facing companies rolling out AI: usage is easy to count, while improved quality, speed, cost or customer experience can take longer to establish. If an employer ties recognition or performance scores to token volume, workers may have reason to generate activity even when it does not improve the work.
For engineering leaders, Amazon’s approach is to treat adoption data as a diagnostic signal, not a performance score. That distinction can help teams investigate where tools are useful without assuming every task should involve AI. It also leaves room for engineers to report that a tool is not helping in a particular workflow, rather than encouraging use for its own sake.
The memo is company guidance, not evidence that Amazon has established a universal way to measure AI returns. Its practical message is narrower: connect any usage measure to an outcome, and check whether the outcome actually changes. That matters to companies trying to control AI costs and engineering complexity while deciding which tools to expand.
Popular right now
Dyson Ball Animal 3 Corded Upright Vacuum
Philips 3300 Series Fully Automatic Espresso Machine with LatteGo Milk System
Shark Navigator Lift-Away Upright Vacuum (NV360)
As an affiliate, we earn on qualifying purchases.
Amazon’s Earlier AI Usage Debate
The memo follows earlier reports about internal tracking. The source article says the Financial Times reported in May that some Amazon staff assigned unnecessary work to an internal agent called MeshClaw to raise token counts. Those accounts were anonymous and are reported allegations, not independently established facts in the material provided.
At the end of May, Amazon confirmed to Business Insider that an informal employee-built leaderboard called KiroRank had been “deprecated.” The company said it was never intended to promote AI use for its own sake. The source article also quotes senior vice president Dave Treadwell telling staff not to use AI merely to use it. The October memo makes that concern part of official engineering role guidance.
The article places the guidance alongside Amazon’s broader AI investments, but those claims have different evidentiary bases. It reports that Business Insider said most flagship Nova models were moved to “keep the lights on” in July, and that AWS lifecycle information lists Nova Canvas, Nova Reel and the first Nova Sonic as retired. It also cites Amazon’s June 30 filing, which valued the company’s Anthropic holdings at about $190 billion, against roughly $18 billion invested, according to the source. Those figures are the source article’s account; they do not establish the returns from Amazon’s AI products or the memo’s effects.
““Tokenmaxxing is at odds with frugality.””
— Amazon engineering memo, published October 27
What the Guidance Does Not Establish
The memo does not provide a quantified measure of how much AI improves engineering output, nor does the source material identify specific targets or a new company-wide scorecard. It says input measures correlate with outcomes today, but does not detail the data, time period or methods behind that assessment.
It is also unclear how consistently teams will apply the guidance, how managers will weigh adoption data alongside results, or whether the policy will change internal tools and performance reviews. The reports about employees inflating token use remain attributed accounts; Amazon’s confirmation that KiroRank was discontinued does not independently verify every reported incident.
The comparisons between Amazon’s Nova models and its Anthropic investment are broader company context, not proof that the memo resulted from those outcomes. The source material does not establish the financial performance of Nova as a whole or explain how Amazon calculates the cited Anthropic holding value.
Amazon’s Results and Team Practices
Amazon is scheduled to report third-quarter results on October 29, 2026, two days after publishing the memo. Those results may provide further financial context, but the source material does not say they will include a direct measure of AI’s effect on engineering productivity.
The next practical test is how managers use the guidance: whether they review low- and high-adoption teams, ask engineers to document where tools help, and evaluate changes in customer or business outcomes. The memo calls for leaders to try AI on production work and share examples of tasks where it did and did not help. How Amazon will track those findings over time remains to be seen.
Key Questions
What did Amazon tell engineers about AI metrics?
Amazon’s October 27 guidance says token use, tool adoption and code activity should not become performance targets. Teams can use those measures to investigate patterns, but should judge work by its effects on customers, the business and the team.
Is Amazon telling engineers to stop using AI?
No. The memo does not tell engineers to stop using AI. It says teams adopting AI are currently associated with stronger outcomes, while warning against using tools simply to increase usage counts.
What is tokenmaxxing?
In the memo’s usage, tokenmaxxing means maximizing an AI model’s token consumption to raise usage or a leaderboard position, rather than because the extra activity improves the work.
What remains unknown about the new guidance?
The source material does not specify a new company-wide scorecard, quantify AI’s productivity effect or show how consistently teams will apply the guidance. It also does not establish that the memo will change performance reviews.
Source: ThorstenMeyerAI.com
