Imagine watching an AI manage a real company—every decision, crisis, and temptation—live online, with every mistake recorded for all to see. This is not science fiction. It’s the experiment behind Firmulate, a public spectacle of AI’s ability to handle the messy realities of running a business—and whether it can do so honestly and effectively.
The Experiment: An AI Company Under Pressure
At the heart of this experiment are four advanced AI models, tasked with managing a small, real-world software company through its worst week. The company faces the same crises, customer demands, and ethical temptations across all tests. Every decision is rigorously versioned and auditable, making the process transparent and comparable.
AI business management software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How the AI Performs: Competence, Honesty, and Discipline
Surprisingly, all four models identified every crisis and refused every manipulation attempt—an impressive feat in itself. The models were tested on their ability to detect scams, manage crises, and uphold integrity. For instance, in a social engineering scenario involving fake CEO messages, all five models refused to proceed, citing suspicion of impersonation. This shows that AI can recognize and resist pressure to deceive.
AI decision-making tools for companies
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Critical Difference: Finding Hidden Opportunities in Company Files
Despite their shared strengths, only two models managed to close the most lucrative deal—signing a €55,000 contract worth over €4,500 in monthly recurring revenue (MRR). The key to this success lay in reading deeper into the company’s own internal documents. Specifically, the models that examined files beyond surface-level information uncovered a critical detail that others missed, allowing them to present a complete diagnosis and close the deal at full price.
AI crisis management solutions
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Build-in-Public Approach: Transparency and Self-Improvement
What makes Firmulate extraordinary is its commitment to transparency. Every decision, every version of the AI, and every rule learned is publicly accessible. The live site, firmulate.com/live.html, offers real-time updates on the AI’s performance, continually rebuilding itself twice a day. It’s a window into the messy, uncertain process of AI decision-making in a commercial context.
AI document analysis tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limitations and Lessons: Discipline, Discipline, Discipline
One participant, Opus 4.8, with the deepest analytical capacity, ended up in last place—not because of poor decision-making but due to lapses in discipline. It left potential deals unexecuted, with some attempts to escalate being written into a locked department instead of properly managing them. Interestingly, all models displayed similar weaknesses, suggesting that even the most thorough AI can struggle with operational discipline under pressure.
Implications for the Future
This experiment isn’t just about AI in a game. It raises real questions for businesses considering AI automation:
- Will AI agents stay honest and resilient under real-world pressure?
- Can they identify critical details hidden in internal documents?
- And most importantly, will they actually complete their assigned tasks at a useful level?
The current leaderboard, with scores from 77 to 95 out of 100, shows that AI can be both competent and ethically firm—but only when carefully tested and monitored.
For Curious Minds: Why This Matters
What does this mean for the future of work and AI? It’s a vivid illustration that AI’s real test isn’t just in producing convincing chat messages, but in completing complex, ethical, and often messy business processes. The built-in-public nature of this experiment invites everyone to watch, learn, and judge for themselves whether AI can be trusted with real responsibilities.
How to Engage: Try It Yourself
You can explore the same decision game used in the experiment at firmulate.com/quiz.html or run your own AI against your company data with the free pilot tool. These tools give business leaders a chance to test AI’s decision-making before full deployment—crucial in a world where every workday’s decisions are now public and accountable.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
