We Published the Fix in July. Production Deleted It on Arrival.

In July we published a post about two broken metrics in our own scanner. One of the two fixes did not run. It was written, reviewed, tested, deployed and described in public, and for five weeks production threw half of it away on arrival.

We found it on 1 September. Here is what happened, why it stayed invisible, and why the same shape shows up in every founder-led company whose stack grew one tool at a time.

The fix that shipped

Buyer alignment is one of five layers in the score, weighted at twenty per cent. In July it was three yes-or-no questions: is the ideal customer clear, is there outcome language, is the category clear. Three booleans give eight possible scores, and any company that can afford a website answers yes to all three. Eleven of fourteen sites scored exactly 100.

So we added two questions that competent sites actually fail. Does the copy name a real role the reader occupies. Does the page state the reader’s problem before it describes the seller. Five booleans give thirty-two combinations and twenty-four distinct scores, which is enough to separate a category.

That is what the code said. What follows is what ran.

The two answers were deleted on arrival

The model answers the rubric questions and returns them as structured output. Before scoring reads that output, it passes through a zod schema for validation. The relevant behaviour of z.object() is that it silently drops any key the schema does not declare.

Five places were involved in adding those two signals. Four of them were updated.

Where	Declares the two new fields
The Rubric type	Yes
The tool schema sent to the model	Yes, with descriptions
The required list on that schema	Yes
The scoring formula	Yes, it reads both
The zod validator	No

So the model was asked both questions, answered both every time, and the validator removed both answers before the formula could reach them. The formula read undefined for each and scored them zero.

Buyer alignment shipped as three booleans wearing a five-boolean formula. Eight reachable values, with a ceiling of 70 in place of 100.

The complaint in the July post was that eight reachable values cannot separate fourteen sites. The fix for that produced eight reachable values. It lowered the ceiling by thirty points and kept the defect.

The database settles it

Twenty-four sites were scanned between the fix and its discovery. Zero of the stored rubrics contain either field. Six of those scans ran cleanly after the deploy, and every score they produced sits inside the dead eight-value set.

Site	Scanned	Buyer alignment
kartaro.studio	2026-07-30	42
ameriking.io	2026-07-31	18
diflumericonsulting.com	2026-08-04	18
garruda.co.in	2026-08-04	0
triage.nu	2026-08-14	70
mobilehomeliving.org	2026-08-23	0

Highest buyer alignment observed across five weeks: 70. The intended ceiling was 100. Every value in that column is a sum of the three surviving weights, and none of them could have been anything else.

Three things that should have caught it

The compiler. Both fields are declared on the Rubric type, so every read of them type-checks cleanly. The type was accurate about the shape the code expected. The parsed object was the thing that came up short, and a type cannot see inside a runtime parse.

The runtime. The helper that turns a boolean into a weight is typed to take a boolean. It received undefined, which is falsy, so it returned zero without complaint. Zero is a legal score. A layer returning 42 out of 100 reads as a finding about the site.

The test. There is a test called buyer alignment has more than eight reachable values. It builds a rubric object as a literal, sets the five keys directly, and calls the formula. It never runs the validator, because the validator was the part nobody suspected. It stayed green with twenty-four values for five weeks while production produced eight.

Each of those three was correct about its own job. The system was wrong.

The founder version of this is a stack, and it fails the same way

A founder-led company assembles its tooling one decision at a time. The store goes in first. Then a CRM, because the store cannot follow up. Then an ad account, a scheduler, an analytics property, an email tool with its own idea of what a contact is, and a spreadsheet somebody built in a hurry that half the company now depends on.

Every one of those is correct about what it can see. The store knows what it sold. The ad platform knows what it spent and what it believes it caused. The CRM knows what a human typed. The spreadsheet knows what someone pasted into it on a Tuesday in March.

There is no screen where those systems are forced to agree. So when two of them disagree, the disagreement has nowhere to surface. Every tool reports success. Each one keeps producing a confident number, and the decision gets made on whichever screen happened to be open.

That is the failure we shipped, at a larger scale. Ours had five participants inside one repository, written in the same week, by people who agreed about what they were building. It still lost half a fix for five weeks. Nine products from nine vendors who have never spoken to each other have considerably worse odds.

Slop is what this produces, and it reads well

Slop usually gets described as bad writing. The expensive version is fluent output resting on a number that stopped being true. A report that reads well, formats correctly, cites a figure and recommends something sensible, built on a metric that quietly went dead in July.

A number between 0 and 100 next to a confident label is very good at not being questioned. That was true of our scanner and it is true of any dashboard. Fluency is the cheap part to produce. Being right requires the system to disagree with itself out loud.

Why connecting everything is the wrong first move

The instinct when a stack contradicts itself is to integrate it. More connections, more sync, one source of truth. Integrations do help. They also add seams, and a seam is precisely where this class of failure lives. Ours was an integration. It sat between two files in the same folder.

Connecting two systems that disagree gets you a faster disagreement. The work that comes first is knowing what the numbers are supposed to be.

What we do instead, in order

Understand the business well enough to hold expectations. A wrong number is only catchable by someone who knows roughly what the right one looks like. We caught this because 70 is a strange ceiling for a metric scored out of 100. Somebody had to find that odd. That is a judgment about the business the tool serves, and it cannot be bought as a feature.

Read distributions before verdicts. A single report is unfalsifiable, because any one plausible number stays plausible. Run the same instrument across a set and look at the shape of the answers. If a column never moves, it is measuring nothing. If the maximum lands on an odd round figure, ask what that maximum is made of. This is the method that found the original two defects in July, and it remains the cheapest test available.

Make disagreement fail loudly. The guard we added is one test, and it deliberately says nothing about these two fields. It reads the scoring formulas, extracts every rubric field they reference, and asserts each one survives validation. Add a signal to a formula and forget the schema, and it goes red on the spot. We ran it against the old code first to confirm it fails there, because a test that cannot fail is decoration.

That third one is the transferable part. Wherever two systems have to agree, there should be one cheap check that breaks when they stop. For most founders that is rarely a test file. It is more often two numbers that should match, reconciled on a fixed day, written down, with one person responsible for looking. The mechanism matters less than whether disagreement produces a signal or silence.

What it cost, and what changed

Every site scanned between 28 July and 1 September received a buyer alignment score up to thirty points low, which is up to six points on the composite. The number was too low for everyone, including us. If you ran a scan in that window, run it again.

The scoring version is now 1.3.0. The formula and the weights are unchanged. Two answers now survive the trip. Anything scored under an earlier version is comparable only to itself, which is the reason we version the scoring at all.

The part worth keeping

We published a fix, in detail, with charts, and the fix did not run. That is embarrassing in a useful way, because carelessness was not the cause. Four of the five places were right. The tests were green. The deploy succeeded. Every part of the process reported success.

Systems built from correct parts fail quietly, and they keep sending you numbers while they do it. The defence is an expectation of what the number should be, and a cheap way to hear the parts disagree.

If you want to see what ours says about your site now, run the Scan. It is free and takes about thirty seconds. If you ran one before 1 September, the score you got was low. Run it again.