my hypothesis: sloppy grammar produce sloppy code. i were wrong, the typo's mattered more deeply

I had a theory that sounded obvious: sloppy grammar prompts produce sloppy code. This AI research side project spanned three weeks and cost some $500 in AI usage (API prices).

Building the experiment

Frontier LLM models are now capable of understanding that they are operating inside a benchmark. I took OpenAI Codex CLI (Command Line Interface) for coding and built out a real work environment for it. One where the AI would not see it was being tested. This would have been easy in a simple restricted chat exchange via API calls, but I built a real environment for it to operate in, still having realistic misconfiguration in my system that prevented it from using code-quality correcting scripts. These were tsc, ESLint and Prettier, that are:

• tsc= TypeScript's compiler. Checks your code's types before it runs. It catches things like calling a function with a text string when it expects a number. The safety net.
• ESLint= a linter. Catches code that's technically valid but almost certainly wrong. Unused variables, unreachable branches, == where you meant ===.
• Prettier = a formatter. Rewrites your code into one consistent style (spacing, quotes, line breaks) so code has the standardized shape.

These are the most common work tools that software developers use. Simply prohibiting them raised suspicion in my earliest MVP rounds. A powerful AI fixes the work environment basics before doing any work, but this time it couldn't regardless of its attempts.

Normal developer feedback loop: write code → run tsc to fix logic mistakes → run ESLint to catch dumb mistakes → run Prettierto fix style mistakes. Removing the loop makes the AI write code blind, just like humans did in the 80's.

I chose OpenAI GPT-5.4-mini-low and Codex CLI after testing different options (after burning all my weekly rate limits).

Why I ran this

My first pass with GPT-mini was five clean runs versus five sloppy runs. Sloppy finished with roughly 2x more TypeScript tsc compiler errors than clean, supporting my original hypothesis.

The outlier 2x tsc run came almost entirely from one catastrophic sloppy run that finished at 265 tsc errors. Published literature is single-turn against HumanEval-style benchmarks. I could not find anything inside a real coding harness like Codex, Claude Code or GitHub Copilot. Previous literature predates agentic coding harnesses entirely.

Academic literature: PromptBench (Zhu et al., 2023) measures character-level robustness on single-turn adversarial prompts. Sclar et al. (ICLR 2024) documented formatting swings of up to 76 points on accuracy benchmarks. Laban et al. (2025) showed consolidating a sprawling multi-turn conversation into one clean prompt recovers ~95% of single-turn performance. Useful, but none of it answers: across a normal 70-turn AI agent session, through four or five context compactions, with the same semantic intent held fixed, does prompt grammar measurably change what you ship?

I tripled the sample and re-ran everything.

The experiment

Codex CLI driving the GPT-5.4-mini LLM. In 70 chat messages the user builds a TypeScript CLI task tracker for a certain application. The initial 70 messages were built interactively based on what GPT answered back. Following run messages were duplicates fixed in order, with two variants:

• Clean. Standard punctuation, no typos, consistent formatting.
• Sloppy. Each user prompt was modified by Opus 4.7, taking inspiration from how many write without any punctuation and in low-effort English. Opus took close inspiration from how Dax Raad writes (@thdxr), to have a real-world software developer yardstick. Sloppy, while preserving every technical identifier verbatim. Misspelling "recurring" is fine, but misspelling --recurring is not. User intent preserved exactly.

Fifteen runs per variant, thirty full builds, 2,100 turns total. After the build phase, thirty Opus fixer subagents made each workspace compile and run with minimal git-diff interventions. That functional test harness ran 68 spec-derived tests against every program in an isolated temp directory. That gave compile-level data, commit-diff data, and runtime behavior on 68 observable capabilities per run.

What I expected

I expected sloppy runs to end with more errors, but they did not.

Clean and sloppy runs finished with a median of 40 tsc errors. Means were clean 51.4 vs sloppy 68.7, but Welch's t = 0.68 (p = 0.50) and Mann-Whitney p = 0.66.

Translation: Welch's t compares the averages of the two groups; Mann-Whitneycompares "which side usually wins head-to-head." The p-valueis the rough chance the gap is just noise, and below 0.05 is the conventional bar for "real"; 0.50 and 0.66 are well into "could easily be coincidence." Here both tests agree: the mean gap is noise.

At n = 15 (sample size per variant), one sloppy run pulled the mean, but three clean runs also ended above 100 tsc errors.

I expected sloppy to ship fewer features, but it did not.

Both variants implemented ~31 sub-commands, accepted ~28 unique --flags, and hit ~62 of 68 spec capabilities in a regex spot-check. (Regex = a search across the source for each expected feature name.) Mann-Whitney p > 0.5 on each. The clean CLI and sloppy CLI were indistinguishable in UX (User eXperience) and QA (Quality Assurance).

Where the grammar actually shows up

Clean carves code into smaller, named pieces

Clean writes about 5% more lines of code (6,136 vs 5,826). The difference is extra structure for the same amount of features.

• +4 files per run (p = 0.039)
• +59 named function declarations per run(p < 0.001, the strongest signal in the dataset)
• 46 functions per 1,000 lines, vs 39 in sloppy (p = 0.001)

Clean extracts helpers. Sloppy inlines expressions into long top-level consts. The extra lines are function boundaries and the blank lines around them.

Neither variant wrote comments, which deserves its own investigation and hints at a coming shift in what software engineering is. Across 30 runs neither variant wrote a single explanatory comment. The only comments the model produced were telling future readers why a catch block is empty. Zero architectural commentary on either side. Sloppy edges clean 12 vs 10 on that one axis. So if anything the sloppy runs are slightly more apologetic about swallowed errors.

Comment samples:

// Ignore cleanup failures; the restored snapshot is already in place.
// no-op if we cannot touch a stale file
// Ignore unreadable history for the chart
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment

Clean gets the boring basics right more often

Sloppy sometimes ships more ambitious features (undo in 10/15 sloppy vs 6/15 clean) but misses core basics, which clean hits reliably.

One clean run calls a printValidationError helper 76 times in one file, every time emitting the same shape: Error: <problem> Use <remedy>. No sloppy run has that kind of internal consistency.

Reasoning engagement flips over the session

On turn 1, sloppy runs spent 6.5× more reasoning tokens than clean (558 vs 86). Every sloppy run reasoned harder on turn 1 than every clean run. That is the disambiguation tax, where a typo'd prompt takes real work to parse. Same as with humans: low-effort grammar requires more effort to understand, and this has been studied well.

By the end of 70 turns, the sign has flipped. Clean runs burn 36% more reasoning tokens per run. Turn 1 is about 0.8% of the whole reasoning budget; the rest of the gap is not disambiguation.

Clean prompts get the model to lean in. They run ~20% more tool calls per turn, commit to richer interpretations, and do more work. Sloppy prompts get parsed into the simplest plausible intent and the model cruises, dreaming of the work shift just ending. Like humans do too when they lose motivation.

The catastrophic tail is where it matters

Both variants have runs that end with broken CLIs. How they break looks very different.

Each finished program was graded against 68 behavioral tests (does add work, does help list commands, does --jsonemit valid JSON, and so on). Clean's worst run passed 28 of 68. Sloppy had three runs below that floor, where two of them passed only 16 and 17, failing roughly three-quarters of the spec. Clean's next two worst runs still passed 43 and 47 of 68; sloppy's did not have a comparable middle ground.

And the texture of the failures is different:

• sloppy_run_9 writes task data to the install directory instead of the working directory.
• sloppy_run_2 silently drops writes. add appears to succeed. Nothing persists.
• sloppy_run_5 ships with no functioning --help flag.

All three programs compile cleanly. They pass the tsc gate. They run. They are also broken on edges, which is the kind of broken you discover when production crashes on Saturday morning.

No clean run in the experiment has an analogue to any of those three failures.

Why this matters at scale

The composite functional-test gap (clean 0.787 vs sloppy 0.703) is +8.3 percentage points, Mann-Whitney p = 0.17. Not significant at n = 15. Read the p-value alone and you round it to zero. Read the distribution and you see the gap comes entirely from sloppy's left tail, not from the middle.

A developer using an AI coding assistant seriously runs 50–100 sessions per day. A team of twenty: 10,000 AI-generated code sessions per week.

At that volume, a 5% silent-failure rate, below the floor I saw in sloppy, is 500 broken-but-passing programs per team per week. A 1% rate is 100. The question is whether you catch every broken-but-passing program before it ships. QA always misses at least one edge.

Compile errors are cheap. A 40 tsc-error run gets reverted in an hour to zero errors.

The sloppy_run_9 class of bug is the expensive class. It compiles and runs, but it writes to the wrong place. Nothing catches it except a user whose data is gone.

Caveats

• gpt-5.4-mini-low through Codex CLI. The "model disengages rather than hallucinates" pattern may not hold for larger models or different reasoning tiers.
• User prompt identifiers were preserved verbatim clean vs sloppy. If typos had hit --recurring as --recuring, I would not isolate grammar.
• n = 15 rules in some effects (reasoning tokens, function decomposition) with tiny p-values and rules out others (tsc errors, feature coverage). It does not statistically ratify the +8.3pp composite gap. Direction is stable; the tail is one-sided.
• Single domain, single task shape. Findings may transfer. I have not checked.

The bottom line

The first version of the hypothesis — clean grammar makes AI write better code — is directionally true, but statistically invisible at the metrics engineers usually reach for. Clean and sloppy produce the same feature surface and indistinguishable compile-error counts.

The version that actually matters — clean grammar changes the shape of what the model produces and avoids the worst tail — is supported. Clean runs carve the same functional surface into more, smaller, named pieces. They standardize error messages and remember the unknown-command handler. Clean runs do not ship programs that silently write to the wrong directory.

A clean prompt reads like you care, and the model meets you there. A sloppy one reads like the floor is fine, and the model parks there.