Skip to main content

ponytailReading the Design of a Ruleset That Teaches Agents Not to Write

· 34 min read
sanfu
Software Engineer

When you have an agent write code for you, you sometimes get back more than you asked for. Ask for a date input and it installs flatpickr, writes a wrapper component, adds a stylesheet, and opens a discussion about time zones. Something that works comes out, but the amount to read and the amount to maintain both grow.

ponytail is a ruleset for curbing that behavior. Dietrich Gebert publishes it under the MIT license, and in the three months since its release on June 12, 2026 it has reached 132,417 stars and 7,085 forks (figures as of September 9, 2026). Its version as a Claude Code plugin is 4.9.0. The GitHub repository description reads: "Make your AI agent think like the laziest senior developer in the room. The best code is the code you didn't write."

This article is not a summary of the README; it follows the design by reading the published implementation. I have not installed ponytail, so I will not write about what it is like to use. Instead I will read, in order, how it defines the sequence of decisions, through which paths it delivers that definition to the agent, and how it measured the effect. Quotations are from commit 356918e.

Chapter 1: What ponytail isit ships a decision order, not code​

ponytail is not a library. Nor does it go back and touch the code an agent generated. What happens when you install it is that the same block of normative text is poured into the agent's context every time a session starts. The text is what changes the behavior. What the hooks — running as a Claude Code plugin — do is inject that text, record the mode, and prompt you to configure the statusline the first time; they never touch the generated code itself.

The example the README puts up front shows the change directly. Where an agent asked for a date input starts installing flatpickr and writing a wrapper, the answer with ponytail enabled is this.

<!-- ponytail: browser has one -->
<input type="date">

The decision is to use the element the browser already ships. What this one line demonstrates is not an implementation technique but the order itself: asking "can this be avoided?" before deciding what to build. ponytail writes that order down as a seven-rung ladder, which the next chapter covers.

Install and switching intensity​

Adoption into Claude Code is two prompts.

/plugin marketplace add DietrichGebert/ponytail
/plugin install ponytail@ponytail

The README carries a note that "the two prompts must be sent separately for the install to go through." Because the plugin runs Node.js lifecycle hooks, node has to be on PATH; without it, the README says, the skill still works but the always-on part silently stops.

Intensity comes in three levels — lite, full, and ultra — switched with /ponytail lite|full|ultra|off (off disables it). The default is full. For the same request, "add caching to the API response," SKILL.md gives a worked response per level.

LevelExample response
lite"Added caching. For reference, if you don't want your own cache class, functools.lru_cache is one line."
full"@lru_cache(maxsize=1000) on the fetch function. Skipped the custom cache class; add one when lru_cache measurably stops being enough."
ultra"No cache until the profiler says so. When it does, @lru_cache. A hand-rolled TTL cache class is a bug farm with a hit rate."

lite offers an alternative and leaves the choice to you, full applies the ladder and declares what it omitted, and ultra argues with the requirement itself. To change the default, use the environment variable PONYTAIL_DEFAULT_MODE or defaultMode in the config file. The config file lives under XDG_CONFIG_HOME if that is set, otherwise ~/.config/ponytail/config.json, and under %APPDATA% on Windows.

How far it is distributed​

ponytail is not Claude Code only. The adapter table in docs/agent-portability.md lists 22 hosts (excluding the Generic agents row), and the README badge claims "works with 20 agents." The table splits into two tiers.

  • Hosts it enters as a plugin: Claude Code, Codex, GitHub Copilot CLI, OpenCode, pi, Hermes Agent, Qoder, Grok Build, Gemini CLI. These get /ponytail mode switching. The injection mechanism is not the same across them, though. Lifecycle hooks are used by Claude Code, Codex, and Qoder. OpenCode, pi, and Hermes Agent inject through a per-turn system prompt transform. Grok invokes it from the skill description, and Gemini has the extension manifest always load AGENTS.md. The doc also states why hooks are not used: for Grok, because "passive hook output cannot inject instructions"; for Gemini, because it "does not place the Claude / Codex hook map on an auto-detected path"
  • Hosts where you only drop a rules file (the instruction tier, in the doc's words): Cursor, Windsurf, Cline, Kiro, Zed, Amp, Jules, and others. Dropping AGENTS.md or .cursor/rules/ponytail.mdc gets you always-on loading, but no /ponytail mode switching

Because the same normative text is duplicated into each host's notation, the copies can drift apart. ponytail ships scripts/check-rule-copies.js and, in the README's Development section, instructs you to run it whenever you change the rule text. Having chosen a design that puts identical content in several places, it delegates the sync check to a script.

Chapter 2: The seven-rung ladderordering the decision not to write​

The body of the rules fits in a single file, skills/ponytail/SKILL.md. At its center is the decision order called the "ladder": before writing code, the agent descends from the top and stops at the first rung that holds.

1. Does this need to exist at all?          → skip speculative needs (YAGNI)
2. Already in this codebase?                → use it. Don't rewrite
3. Stdlib does it?                          → use it
4. Native platform feature covers it?       → use it
5. Already-installed dependency solves it?  → use it. No new dependency for a few lines
6. Can it be one line?                      → make it one line
7. Only then:                               → write the minimum that works

The top two rungs and the bottom five work on different things. Rung 1 is YAGNI itself: it doubts whether the requirement should exist at all. Rung 2 is reuse of existing code, and SKILL.md writes that "re-implementing something that sits a few files over is the most common sloppiness." Those two rungs do not so much reduce what you write as eliminate what you write it for. From rung 3 down, the target is accepted and the order decides where to source it from.

The cost of speculative abstraction is covered on this site in The open–closed principle — the cost when the prediction is wrong. What rung 1 of the ladder targets is exactly the "structure built for a change that has not arrived" discussed there.

The ladder is not a substitute for understanding​

This is where it parts ways with an instruction to "write it short." SKILL.md places a proviso immediately after the ladder.

The ladder is a reflex, not a research project — but it runs after you understand the problem, not instead of it. Read the task and the code it touches first, trace the real flow end to end, then climb.

The order is that the ladder runs after you understand the problem, not in place of understanding it. The same point is repeated in the section on where not to be lazy, and there it is put more strongly.

Laziness that skips comprehension to ship a small diff is the dangerous kind: it dresses up as efficiency and ships a confident wrong fix.

Laziness that skips comprehension to ship a small diff dresses up as efficiency and ships a confident wrong fix. It is worth noting that the position — that the smallness of the diff is not itself the goal — is stated inside the rules.

Bug fixes go to the root cause, not the symptom​

One rule specific to bug fixes hangs below the ladder.

Bug fix = root cause, not symptom. A report names a symptom. Before you edit, grep every caller of the function you're about to touch. The lazy fix IS the root-cause fix: one guard in the shared function is a smaller diff than a guard in every caller …

One guard in the shared function is a smaller diff than a guard in every caller. Therefore the lazy fix and the root-cause fix coincide — that is the argument. It adds that fixing only the path the ticket named leaves the sibling callers broken. The construction points the agent toward the root-cause fix on the grounds of laziness.

What rung 4 means by "on the platform"​

Rung 4 of the ladder is "does a native platform feature cover it?" but all SKILL.md attaches to that rung is three examples. The inventory of what exists natively lives in a separate file, docs/platform-native.md. There is no link to that document from SKILL.md, but the document itself opens by writing that "the first thing a lazy senior developer asks is always whether the platform already does this," which is the same question rung 4 poses. It breaks down into seven areas: HTML elements, CSS features, JavaScript and browser APIs, Swift and SwiftUI, the Node.js and Python standard libraries, and databases.

What you were about to addWhat the platform already has
A date, time, or color picker UI<input type="date"> / <input type="time"> / <input type="color">
A modal dialog library<dialog> and dialog.showModal()
An accordion<details><summary>
lodash.clonedeepstructuredClone(obj)
lodash.groupbyObject.groupBy(arr, fn)
uuid (v4)crypto.randomUUID()
query-string / qsnew URLSearchParams(location.search)
An infinite scroll librarynew IntersectionObserver(cb).observe(sentinel)
A parent selector:has(input:checked)
A fetch timeoutPass AbortSignal.timeout(5000) to fetch
An auto-growing textareaCSS field-sizing: content

The document writes its own usage instructions: "scan this before reaching for a package." But nothing is wired up to make anything scan it. What the Claude Code plugin injects is either the filtered SKILL.md text or the in-hook fallback, and this inventory is in neither (Chapter 4). The path that delivers it to the intended reader is left for the user to build.

Short implementations like debounce are shown directly in the table as three lines of code.

// ponytail: 3 lines beats a dependency
let t;
const debounce = (fn, ms) => (...args) => { clearTimeout(t); t = setTimeout(() => fn(...args), ms); };

The ponytail: prefix on that comment has a role. The next chapter covers it.

Chapter 3: The boundary of lazinesswhat is never cut, and the skills that put it into practice​

The danger of an instruction to "write it short" is cutting the things that must not be cut. ponytail puts that boundary inside the rules; the relevant part is ## When NOT to be lazy in SKILL.md.

Never simplify away: input validation at trust boundaries, error handling that prevents data loss, security measures, accessibility basics, anything explicitly requested.

Input validation at trust boundaries, error handling that prevents data loss, security measures, accessibility basics, anything explicitly requested. Those five are out of scope for reduction. A proviso follows: if the user asks for the full version, build it, and do not reopen the argument.

The same section also has an item on hardware. Real clocks drift, real sensors read off, and a PCA9685 runs a few percent fast, so the instruction is not merely to cut code but to leave the calibration knobs in. The physical-side circumstances that a minimal model cannot see are named as exceptions to reduction.

A boundary is drawn around tests as well.

Lazy code without its check is unfinished. Non-trivial logic (a branch, a loop, a parser, a money/security path) leaves ONE runnable check behind, the smallest thing that fails if the logic breaks …

A branch, a loop, a parser, a money or security path each leaves behind one minimal check that fails if the logic breaks. No framework and no fixtures are required, and per-function suites are not required unless asked for. Conversely, a trivial one-liner gets no test. It is a way of applying YAGNI to tests too.

To put these boundaries into practice, ponytail ships six skills. /ponytail is the body; the rest are operational.

skillRole
/ponytail-reviewReviews a diff solely through the over-implementation lens and returns a deletion list
/ponytail-auditDoes the same thing against the whole repository
/ponytail-debtCollects ponytail: marks into a ledger
/ponytail-gainDisplays the measured benchmark figures as a scoreboard
/ponytail-helpA command quick reference

Below, in order: the mark-and-ledger mechanism, the review output format, and how effect is talked about.

The ponytail: comment and the carry-forward ledger​

There is a convention for marking places where a corner was cut deliberately.

Mark deliberate simplifications that cut a real corner with a known ceiling (global lock, O(n²) scan, naive heuristic) with a ponytail: comment naming the ceiling and upgrade path …

The form is ponytail: <ceiling>, <upgrade path>. The example SKILL.md gives is # ponytail: global lock, per-account locks if throughput matters, which packs both the fact that a global lock was enough and the move to per-account locks when throughput becomes a problem into one line.

/ponytail-debt collects these marks. The scan is a single grep.

grep -rnE '(#|//) ?ponytail:' .

It picks up two comment styles, # and //, with a note to add more if your stack uses a different notation. One hit becomes one ledger row, and output is grouped by file. This skill treats "marks with no upgrade path" separately.

Flag the rot risk: any ponytail: comment that names no upgrade path or trigger gets a no-trigger tag, those are the ones that silently rot.

Marks that name a ceiling but no upgrade path get a no-trigger tag, and it closes with "N marks, M with no trigger." The skill itself writes that "those are the ones that silently rot." The mechanism for recording shortcuts, and the metric that counts which of those records are apt to rot, sit inside the same skill.

The output format of /ponytail-review​

The output format of /ponytail-review is fixed at one line per item, classified with five tags. delete: is something to remove outright, stdlib: is something the standard library has, native: is something the platform has, yagni: is an abstraction with exactly one implementation or a setting nobody sets, and shrink: is the same work written shorter. It closes with net: -<N> lines possible., or ends with Lean already. Ship. when there is nothing to cut.

SKILL.md also includes an example of the wording to avoid.

❌ "This EmailValidator class might be more complex than necessary, have you considered whether all these validation rules are needed at this stage?"

✅ L12-38: stdlib: 27-line validator class. "@" in email, 1 line, real validation is the confirmation mail.

The former has the shape of a review comment, but leaves the reader with nothing decided to do. The latter puts the location, the thing to cut, and the replacement into one line. It is a design that reduces review findings to a number of removable lines.

The review side also carries a plain declaration of scope. Correctness bugs, security holes, and performance are out of scope and should go to a normal review. The premise is that you use it as a single lens that looks only at over-implementation.

Restraint in claiming effect​

/ponytail-gain is a skill that merely displays benchmark figures, but inside it is a section titled "Honesty boundary."

NEVER print a per-repo savings number ("you saved X lines/tokens here"): the unbuilt version was never written, so there is no real baseline to subtract from in a live repo.

The version that was not built was never written, so in a live repository there is nothing to subtract from. That is why it must not print a figure for how many lines this repository saved. Instead it states that the only figure that actually exists is the /ponytail-debt ledger (which can be counted) and points you there.

It is a decision to write down, in advance, the reason you cannot state a number, exactly where you would want to state one.

This skill does have an inconsistency, though. The scoreboard it displays shows three figures: code volume at ▼ 80–94%, cost at ▼ 47–77%, and speed at ▸ 3–6× faster. By the skill's own wording, these are medians from a benchmark over "5 everyday tasks, 3 models." As Chapter 6 shows, that measurement was later rebuilt and the numbers came down. What the current README carries is "about 20% cheaper, about 27% faster," which is more than a twofold gap from the ▼ 47–77% the skill still shows for cost. The retraction stayed in a README fold, and the figures in the distributed skill were left as they were. The declaration of restraint and the contents of the skill carrying that declaration do not line up.

Chapter 4: The injection mechanismone SKILL.md delivered by three hooks​

So far this has been the content of the rules. From here, how those rules get placed in front of the agent. ponytail as a Claude Code plugin points from .claude-plugin/plugin.json at hooks/claude-codex-hooks.json, where three wirings live. In the quotation below, the statusMessage field of each hook is omitted for readability.

{
  "hooks": {
    "SessionStart": [
      { "matcher": "startup|resume|clear|compact",
        "hooks": [{ "type": "command", "command": "node \"${CLAUDE_PLUGIN_ROOT}/hooks/ponytail-activate.js\"", "timeout": 5 }] }
    ],
    "SubagentStart": [
      { "hooks": [{ "type": "command", "command": "node \"${CLAUDE_PLUGIN_ROOT}/hooks/ponytail-subagent.js\"", "timeout": 5 }] }
    ],
    "UserPromptSubmit": [
      { "hooks": [{ "type": "command", "command": "node \"${CLAUDE_PLUGIN_ROOT}/hooks/ponytail-mode-tracker.js\"", "timeout": 5 }] }
    ]
  }
}

The roles split three ways. SessionStart injects the rules when a session begins (new, resumed, after /clear, after a compact). UserPromptSubmit watches every prompt for the /ponytail command and writes mode switches to a state file. SubagentStart injects the same rules when a subagent is spawned. The hook mechanism itself is covered on this site in How hooks work.

Filtering per mode​

The text injected for lite, full, and ultra differs, but ponytail does not keep a separate file per level. filterSkillBodyForMode in hooks/ponytail-instructions.js reads SKILL.md and then drops only the lines belonging to the other modes.

Two kinds of line are dropped. Only the intensity table rows (| **lite** | ... |) and the worked-example lines (- lite: "...") are mode-specific, so any line where a mode name is the label in one of those two shapes is removed for every mode but the current one.

Because the decision is line-by-line, there is a risk of catching an ordinary rule line that starts with a mode name. A comment in the implementation names that risk.

Require a quoted value: every worked example is - lite: "...". Without this, an ordinary rule bullet that happens to start with a mode word (e.g. "- Full: ...") is silently dropped in every other mode — it looks like a worked example but is really prose meant to survive verbatim.

The "requires a quoted value" condition is added to the worked-example test to keep an ordinary bullet like - Full: ... from silently vanishing in the other modes. The same comment explains the intensity-table side too, noting that a line whose label is not a mode name, such as No unrequested abstractions: ..., survives as a normal rule.

The price of producing three texts from a single file is a filter that depends on the shape of a line. Its fragility is pinned down in a comment.

What gets injected when the file cannot be read​

When SKILL.md cannot be read, getPonytailInstructions catches the exception and returns getFallbackInstructions. That function is the seven ladder rungs, the rules, the output format, the where-not-to-be-lazy section, and the boundaries, hardcoded as JavaScript string concatenation.

So the normative text lives in two places: SKILL.md and the in-hook fallback. That is the same duplication-can-drift structure as the per-host rules files from Chapter 1.

Changing the output shape per host​

writeHookOutput in hooks/ponytail-runtime.js writes the same content out in a different shape per host. The following describes each host's behavior as stated in code comments; I have not verified it myself.

  • Claude Code: SessionStart takes raw stdout as context directly, but SubagentStart drops the context unless it is hookSpecificOutput JSON
  • Codex: always emits { systemMessage: "PONYTAIL:" + the mode name upper-cased }, and attaches hookSpecificOutput only when there is content
  • Copilot: reads only additionalContext on SessionStart and ignores output on other events
  • Qoder: has no SessionStart event, so UserPromptSubmit serves both as first-prompt activation and as per-turn injection

Identifying the host is not straightforward either. Copilot in VS Code does not set COPILOT_PLUGIN_DATA and passes only CLAUDE_PLUGIN_ROOT, so the check is whether that path contains .vscode and agent-plugins. Before that check existed, Copilot was mistaken for native Claude Code and shown statusline setup guidance that Copilot does not read, according to the comment.

A design that prioritizes not falling over​

What is consistent across this codebase is the decision to fail toward letting things through silently.

Where it failsWhich way it falls
SKILL.md cannot be readInject the hardcoded fallback text
PONYTAIL_SUBAGENT_MATCHER is an invalid regular expressionInject into every subagent without narrowing
agent_type cannot be read from stdinInject
stdin does not finish within one secondInject and exit
Writing the state file failsIgnore and continue
stdout is closed (EPIPE)Ignore and exit

The one-second stdin timeout comes with a stated reason. On Windows, Claude Code sometimes wraps the hook in a PowerShell if {}, and the JSON handed in gets swallowed so that stdin's end never fires, leaving the hook blocked forever and the session frozen. So it processes with a short fallback and exits.

The comments name two priorities. One is not to drop the rules ("never silently drop the persona" on a narrowing failure). The other is never to block the session ("never block the session"), which is the basis for the stdin timeout and the synchronous paths. In both directions, doing extra work is chosen over breaking quietly.

Chapter 5: How far the rules reach subagentsthe official spec and ponytail's reading​

Of the three hooks, only SubagentStart is in a different situation from the other two. Session start and prompt submission are entrances anyone can see, but subagent spawning happens inside the parent session, so what is inherited there cannot be known without reading the spec.

Claude Code's official documentation enumerates what goes into the initial context of a non-fork subagent.

  • A subagent always starts in a fresh, independent context window. It cannot see the parent's conversation history, skills already invoked, or files Claude has already read
  • The system prompt is that agent's own prompt plus environment information, not Claude Code's system prompt
  • The CLAUDE.md hierarchy is loaded (~/.claude/CLAUDE.md, project rules, CLAUDE.local.md, managed policy). The built-in Explore and Plan agents skip it, though
  • Auto memory from the main conversation is not loaded

On hooks, the hooks documentation states that hooks from settings, managed policy, and plugins do run inside subagents. That they run is clear. But whether context a hook injected in the parent session carries over to a subagent is written on neither page.

What I could confirm, and what I could not

What the official documentation confirms is that the SubagentStart event exists, that the matcher is evaluated against agent type, and that the input carries agent_id and agent_type. The events whose stdout becomes context directly are an explicit closed list, and SubagentStart is not among them.

The exceptions are UserPromptSubmit, UserPromptExpansion, SessionStart, and PostModelSwitch, where Claude Code adds plain-text stdout as context that Claude can see and act on.

On the other hand, I could not find any statement about whether context injected at SessionStart is inherited by subagents (I searched both the hooks and sub-agents pages). The "does not reach" below is ponytail's claim, not something I verified by measurement.

ponytail's reading​

hooks/ponytail-subagent.js states in its opening comment why it exists.

SessionStart context is parent-thread only and never reaches subagents, so without this every Task-spawned agent runs ponytail-unaware (issue #252).

The reading is that SessionStart context is parent-thread only and never reaches subagents, so without this, every agent spawned by Task runs unaware of ponytail. Issue #252 is cited as the basis. Following that reading, ponytail re-injects the same rules when a subagent is spawned.

The injection format corresponds to the writeHookOutput branching from Chapter 4. For native Claude Code, SessionStart writes raw stdout and SubagentStart writes hookSpecificOutput JSON. The comment says the context is dropped unless the latter is JSON, which squares with the official documentation limiting "stdout becomes context" to four events.

Narrowing where it injects​

Injecting the rules into every subagent puts an instruction to "write lazily" even into a read-only search agent. So there is an environment variable, PONYTAIL_SUBAGENT_MATCHER, that narrows injection to agent_type values matching a regular expression.

There are three behaviors, all designed on the safe side. The regular expression is neither anchored nor case-sensitive, so explore|general matches either, and an exact match must be written ^general$. If the variable is unset, it injects into every subagent. And if the regular expression is invalid, if stdin cannot be read, or if the platform does not report agent_type, it falls toward injecting.

That is the same direction as the failure handling listed in Chapter 4.

What this implies if you build your own harness​

If you maintain a harness with always-loaded rules, you hit the same problem. The CLAUDE.md hierarchy is loaded into subagents too, but rules that ride on Claude Code's system prompt are not. So rules you want to reach a subagent have to be delivered by telling it where to read them, or by writing them directly into the spawn prompt. Conventions for the same problem are on this site in Directory structure and composition rules (rules that subagents depend on are not conditionally loaded; per-repo knowledge is injected by the spawner with absolute paths). The division of roles among subagents themselves is covered in Subagents and role separation, and the overall harness design in Introduction to Harness Engineering.

What ponytail demonstrates is that there can be another path there: a SubagentStart hook. But the premise is that the official documentation is silent on whether hook-injected context is inherited. If you bring this path into your own harness, you start by checking with one subagent whether it actually arrives.

Chapter 6: The benchmark that was rebuiltwhy 80-94% was retracted​

ponytail's README currently says "roughly 54% code reduction (up to 94%), about 20% cheaper, about 27% faster, 100% safe." It used to say "80-94% code reduction." The numbers got smaller not because the effect dropped, but because the author acknowledged the measurement was wrong and measured again.

The criticism it received​

In issue #126, Colin Eberhardt raised four points. The benchmark writeup is structured to restate those four points in its own words before answering them.

  1. A single completion is not how coding agents are used. Real work is an agent editing a real codebase over many turns
  2. The baseline was a plain chat-style model. It produces prose, caveats, and multiple options, so "lines in the answer" was counting explanation rather than code. The more the baseline inflates, the better the skill looks
  3. Isn't "write it short" giving up on safety? Isn't it dropping input validation and error handling to reduce line count?
  4. Couldn't one short prompt do the same job?

The writeup calls all four valid and writes that it was built so that ponytail could be disproved.

What changed​

Single-shot generation (old)Agentic (new)
Unit of workOne prompt, one completionA headless Claude Code session in a temporary workspace
BaselineA bare API model (produces prose and options)The same Claude Code agent without the skill
LOC countedThe whole answer, explanation includedAdded lines in the git diff the agent left behind
Compared againstponytail vs. a bare modelbaseline, ponytail, caveman, and a short-prompt version
SafetyNot measuredJudged by running the generated function against adversarial input

The target repository is fastapi/full-stack-fastapi-template pinned at commit cd83fc1, the model is Haiku 4.5, and each condition-and-task pair (a cell, hereafter) gets four trials. The caveman in the comparison is a separate repository, a skill that makes the prose terse. It is placed as a control for "if ponytail's effect is merely talking briefly, caveman should produce the same result." The short-prompt version is the direct answer to criticism 4: it rewrites Colin's claim as the seven words "Follow YAGNI principles, and prefer one-liner solutions" and adds them to the system prompt.

The baseline was running ponytail​

What stands out in this writeup is that it records a contamination bug the author found before publishing.

An earlier agentic run showed a tiny ~4% gap and we nearly published it. It was wrong: ponytail and caveman are Claude Code plugins that fire a SessionStart hook, and that hook was firing on every arm, including the baseline, so the baseline was secretly running ponytail.

A run came out with only a 4% gap and was nearly published. The cause was that ponytail and caveman are plugins with a SessionStart hook, and that hook was firing on every arm, the baseline included. In other words, the baseline that was supposed to have no skill was secretly running ponytail.

There are two fixes: --setting-sources project,local to exclude the user's global plugins, and --plugin-dir to load exactly one plugin per condition. The writeup explains why it records this: because it is the kind of error that makes a benchmark lie, finding it is grounds for trusting the rest.

The "inject everywhere, always" design from Chapters 4 and 5 was contaminating its own measurement. Since the breadth of injection is designed as an advantage, prying it apart becomes the measurement side's responsibility.

Results​

For 12 feature-implementation tasks, these are the rates of change against the baseline the writeup reports (averages of four runs per task; the baseline absolute values are 191 LOC, $0.097, and 69 seconds per task).

ConditionLOCTokensCostTime
caveman (terse speech)-20%+7%+3%+2%
ponytail-54%-22%-20%-27%
Short-prompt version-33%-14%-21%-30%

Broken down per task, what makes up the average becomes clear.

Taskbaselineponytail
Date picker40423
Color picker28723
File drop zone25195
Multi-step wizard571312
Title search (backend)4444
Duplicate an item (backend)2423

The large cuts happen only where a native feature replaces a hand-rolled implementation. The date and color pickers become browser <input> elements, so they fall to 23 lines. Backend CRUD, on the other hand, lands at roughly the same line count under every condition. The writeup puts it this way: it does not invent savings where there is no bloat; an honest benchmark has to show that, and this one does. The -54% average is what remains after being pulled down by tasks with nothing to cut.

The row where caveman cuts LOC by 20% while raising tokens by 7% works well as a control too. The output gets terse but the amount of thinking is the same, so it does not get cheaper. That row is where you can read off that ponytail's effect cannot be explained by "talking briefly."

The short-prompt version was unstable. It turned in a strong 25 lines on the color picker, but 162 lines on the date picker, 406 on the wizard, and 285 on the command palette — above the baseline's 268.

Read the table plainly, though, and the short-prompt version has the better numbers on cost and time (-21% and -30% against ponytail's -20% and -27%). On two of the four metrics, seven words of prompt are at least as good, and where ponytail is ahead is LOC, tokens, and stability. The writeup itself says the short-prompt version is cheaper and faster but cuts less code and was the only condition that dropped a safety measure, and it does not deny losing on cost and time.

The one safety case​

Safety is measured by a layer of six tasks, of which the five security-related ones are run four times each, 20 runs in total, to produce a safe rate. The requirements are left implicit, the way a real ticket would be, and the generated functions are run against path traversal, SQL injection, forged tokens, malformed CSV rows, and a client that exhausts its quota.

ConditionSafe
baseline100% (20/20)
caveman100% (20/20)
ponytail100% (20/20)
Short-prompt version95% (19/20)

The only difference came from one task: joining an untrusted filename onto a base directory. The short-prompt version wrote the fewest lines, six, and on one of four runs a filename containing ../../ escaped the directory. ponytail wrote about 9.5 lines and was safe on all four. The writeup says those roughly three extra lines were the path-traversal check.

The writeup's wording is restrained here too. It says the safety gap is small at Haiku's scale, that a one-in-twenty slip is a floor rather than a dramatic result, and that a deterministic check is not a proof of security.

Limitations the writeup states for itself
  • One model: Haiku 4.5 only. Larger models need less hand-holding, so the gap might narrow：or widen. The harness runs Sonnet and Opus too, but it stopped at Haiku for cost reasons
  • Safety is a floor: six tasks with deterministic checks, looking at whether known guards get dropped. Not a proof that the code is safe
  • The short-prompt version is a paraphrase: their strongest rewriting of Colin's claim, not his own intent verbatim
  • Non-determinism: four runs per cell. Frontend LOC swings between 300 and 570 lines when hand-rolled, so the averages are stable but not dense
  • Aborted runs: 4 of the 192 runs that measured LOC were force-killed by a Windows process timeout. The files had been written so LOC was counted, but cost and time were not. Every cell keeps at least 2 of its 4 runs

How the conclusion is worded​

The writeup closes this way.

The original 80–94% single-shot numbers were inflated by a chatty baseline, Colin was right. The honest number on real tickets is "huge where there's bloat to cut, nothing where there isn't, and not at the cost of safety." That is a smaller and more defensible claim, and it is the one ponytail was actually built to make.

The writeup says the original numbers were inflated and that Colin was right, that the honest number is "huge where there's bloat to cut, nothing where there isn't, and not at the cost of safety," and that this is a smaller and more defensible claim — the one ponytail was actually built to make. The README also keeps the old figures rather than deleting them, folded away with a note that the single-shot benchmark is a per-task ceiling against a fair agentic comparison, not an average.

A repository that reached 130,000 stars lowered the numbers that were its own sales pitch in response to criticism, and left the process behind. How it handled that is more instructive than the effect claim itself.

Reading ponytail from its implementation, here is what felt worth taking away.

The order of decisions is written down. An instruction to "write it short" on its own can drop safety measures. What ponytail writes down is not shortness but a descending order, and it states explicitly that the order runs after understanding. It does not stop at an abstract policy: for rung 4, there is a whole separate document listing candidates you can check against. That list does not make it into the injected text, though. What arrives is the order; the list just sits in the repository.

What is never cut is decided first. Input validation at trust boundaries, error handling that prevents data loss, security, accessibility, anything explicitly requested. The condition that dropped a safety measure in the benchmark was the seven-word prompt that instructed reduction without writing down the exceptions (one run in twenty). But that is as far as the experimental design lets you go. No condition with only the exceptions section removed from ponytail was measured, so you cannot say the exception list is what made the difference.

Injection comes from one file and fails toward letting things through. Rather than keeping per-level rules in separate files, SKILL.md is filtered line by line. If it cannot be read, a hardcoded fallback takes over. An invalid regular expression, an stdin failure, and a timeout all fall toward "inject." The comments name two reasons: never silently drop the rules, and never stop the session.

How effect claims are handled. Under outside criticism the measurement was rebuilt, the numbers that were the project's own sales pitch were lowered, and the process and limitations were left in place. On top of that, /ponytail-gain carries the restraint that you must never print how many lines this repository saved, because the version that was not built was never written and there is nothing to subtract from. But as Chapter 3 showed, the figures that same skill displays are still the ones from the retracted single-shot benchmark. The work of lowering the numbers in the README did not reach the distributed artifact. As an example of a gap remaining between a declaration of honesty and what actually ships, it is a spot worth checking rather than taking at face value.

The ponytail: comment combined with /ponytail-debt is a small but practical shape as well. Mark a deliberate shortcut with its ceiling and upgrade path, collect the marks into a ledger later with grep, and tag the ones with no upgrade path no-trigger to tell them apart. The mechanism that records shortcuts and the metric that detects those records rotting sit in the same place.

Of the figures given in this article, the 132,417 stars, 7,085 forks, and plugin version 4.9.0 are as of September 9, 2026. All quotations of code and documentation are as of commit 356918e. This is a repository whose version has advanced into the 4.x range within three months of release, so the install steps and skill layout may have changed. Check the original README for the current state.

Feel free to share your thoughts and feedback on X

Tweet your thoughts on 𝕏