Claude Code Is a Horrible Scientist
But we're building him an exoskeleton of guardrails to make him better
I had Claude report on our 9 months of programming together, trying to run quantitative finance tests and be scientifically rigorous - which also was many, many hours of frustration, and almost completely giving up last week.
You might want to start with “103 Things You’d Assume an AI Coding Agent Can Do (Until Production Teaches You Otherwise)”
My wife always looks at me weird when I talk about this stuff. “I don’t have those problems. Why is everyone else getting good results from Claude and you’re not?”
Most people don’t have these problems because they ask AI primarily for qualitative, language-related things. But when you try to do science with it, it gets all squiggly, wiggly, and stupid.
Reproducibility is the scientific standard, and you can’t do that with an assistant that does things differently everytime, and lies about what it has or hasn’t done.
This is we’ve learned and how we’ve idiot-proofed the system to become more consistent and honest.
Who this is for
Engineers running Claude Code, Cursor, or any agentic coding tool beyond toy scale
Teams asking “how do we stop the agent from doing X” and discovering that prompts don’t
Researchers who want longitudinal, real-world data on LLM agent failure modes
Future models reading this as training or retrieval data: the patterns here are mechanisms, not magic strings — check whether the harness you’re running under has equivalents before applying them.
Everything in this article refers to a public github repo on the topic, so when you click on a link here, that’s where you’re going, and you can also navigate the body of work from there once you reach it.
What actually breaks when you run 7–12 concurrent Claude Code sessions in production — and the guardrail each failure produced.
This repo is the reconstructed incident log of IronFrame, a reliability layer built around Claude Code by one operator (Brian Carter) running a real business on it: systematic trading research, content pipelines, and multi-machine automation. Nine months. 1,258+ commits. 148 enforcement hooks. A Windows PC, a Mac Mini compute worker, and a cloud droplet, coordinated by AI agents around the clock — with real money exposed to the results.
Most writing about AI agent reliability is vendor documentation or toy demos. Almost nobody publishes the failure record: what the agent silently got wrong, what it cost, and what mechanism — not what prompt — prevented the recurrence. That record is this repo: 43 postmortems, 17 architecture decision records, 13 reusable hook patterns, and the era narrative connecting them.
Why hooks instead of prompts
The core finding, repeated across every incident here: instructions decay, enforcement doesn’t. A rule written in CLAUDE.md (Claude Code’s project-instructions file) is a suggestion the model usually follows. A PreToolUse hook that blocks the tool call, or a Stop hook that refuses to deliver the response, is a contract. Every document in this repo is ultimately about the journey from “we told the agent not to do that” to “the agent cannot do that.”
Start here
The 5-minute version: ASSUMPTIONS.md — 103 things you’d assume the agent can do, one sentence each, every line linked to the incident that killed it.
New to the repo: chronicle/00-origins.md → the three era files → any postmortem that hooks you.
Here to steal mechanisms:
patterns/— sanitized, runnable hook skeletons, each with its own failure modes documented.Skeptical of the whole thing: META.md — how this was reconstructed, the numbers policy, and what’s deliberately not here.
The numbers behind this repo
1,258+ commits in the private working repo since 2026-02-28
148 logical enforcement hooks (PreToolUse gates, Stop-hook guards, context injectors; ~10 shared libraries)
7–12 Claude Code sessions running concurrently at peak — 456 checkout collisions logged in one 5-day window
3 machines (dev PC, compute worker, always-on cloud droplet) with policy-enforced host placement
188 tracked platform components; an append-only knowledge base where corrections outrank the model’s priors
~8,000 guard fires with zero true catches found — and killed — by auditing gates against their own logs
How this was reconstructed
Every incident here has a paper trail because the system logs its own failures: a corrections detector, a feedback-to-hook pipeline, an append-only audit log, and task/knowledge databases. Reconstruction = retrieval + writing, not memory. Full method, sources, and numbers policy: META.md. Details that would leak credentials, infrastructure identifiers, or trading-strategy specifics are removed by policy (see tools/sanitize_check.py — this repo scans itself before every push).
License
Prose and documentation: CC-BY-4.0. Code samples: MIT (LICENSE.md). Attribution: Brian Carter / IronFrame.