Keep the judgment, cut the slop

My AI harness had 57 agents on file, but the busiest worker in it was me.

Fixing that meant learning what to keep and what to cut.

Every task started with a choice: which agent should handle it.

Before I could hand off any work, I had to translate what I wanted into the harness's own names, and nothing in the interface explained them.

A one-off task had to become the right agent's name, and a repeatable workflow had to become a skill.

When the work came back, I made another choice: did I trust it?

I had to remember which gate could stop a task and what happened when one threw an exception.

I was the switchboard operator of my own automation, plugging every call into the right jack by hand, 57 jacks deep.

The limit was simple: how much of the harness I could carry in my head.

The first redesign made the size of that job visible.

57 agent files and 250+ skills had made the harness look capable.

Each agent file gave me another jack to remember, and each skill gave me another way a task could go wrong.

I cut them to fourteen roles, each one I could explain in a sentence: one for marketing, one for code review, one for design, and so on.

That number still came from a guess about what might get used.

The usage logs told a harder story.

Of the fourteen, only five had ever done real production work, and the other nine sat wired in like extensions nobody ever called.

An agent earns a permanent line through weeks of real use.

A perfect score that was quietly wrong

Fourteen roles made the harness easier to hold in my head, but it still needed proof that its decisions were sound.

One eval gave me that proof, and the proof was wrong.

I had inverted its grading rule by mistake, so it marked the wrong answer as correct.

It still passed 28 of 28 tests.

Every test in that suite shared the rule's mistake, so passing only proved the test agreed with itself.

That's slop with a green checkmark: confident and wrong, machine-approved.

The rule was supposed to catch a mistake before it reached someone relying on the harness's judgment. It rewarded whichever answer matched its own broken assumption instead.

Nothing in that loop checked the outcome the rule was built to protect.

I changed the standard after that.

A check has to start with the result a person cares about, then work backward to the decision that caused it.

A case the evidence already covers can move on its own.

A case that changes the outcome goes to a person whose judgment can carry it.

Less to think about

The next comparison covered seven kinds of work the harness does: writing, coding, producing video, running security reviews, and a few more.

None of the seven cleared the bar to replace the way that work already got done, so I kept the improvements inside the existing paths and dropped the replacements.

That's when I stopped counting what the harness could do and started counting what I no longer had to remember.

A better switchboard needs fewer things pulling at the operator's attention.

The real cost was always the number of steps between wanting something done and getting it.

Users need less to think about.

The same thing applies to product marketing.

The important question isn't "why is my team shipping so much and retaining so little?" It should be: "do we really know where the user gets stuck, or do we only know what we want to build?"

Keeping the judgment meant cutting every line that only looked useful.

What will your users get to forget?