We Scored 14 B2B Sites. Two of Our Metrics Were Broken.

We built a scanner that scores a website's content across five layers and returns a number out of 100. It has been running for months. In July we pointed it at fourteen sites in our own category, one after another, and read the results as a distribution rather than as fourteen separate verdicts.

Two of the five layers were broken. Together they carried forty per cent of every score the tool had ever produced.

This is what they were doing, how the distribution gave them away, and what we changed. The numbers below are the real ones. The sites are anonymised because the point is not who scored what.

The tell was not a wrong score. It was a boring one.

No single result looked wrong. Scores landed between 48 and 80, which is a believable spread for a category of competent companies. If you look at one report you see nothing.

Then you line fourteen of them up and one column does not move.

Buyer alignment returned exactly 100 for eleven of the fourteen sites, and produced three distinct values in total. It is weighted at twenty per cent of the composite. So a fifth of every score the tool produced was, in practice, a constant that got added to almost everyone.

The cause was in the formula, and it is embarrassingly simple. Buyer alignment was three booleans: is the ideal customer clear, is there outcome language, is the category clear. Three yes-or-no questions have eight possible combinations, and any company that can afford a website answers yes to all three.

A measure with eight reachable values cannot separate fourteen things. It was not measuring the sites. It was measuring whether the site was made by professionals, and they all were.

The second one was worse. It measured the opposite of what it claimed.

The feature is called cross_page_similarity. The type declaration documents it as "TF-IDF cosine across pages". The layer built on it is called voice consistency and is weighted at twenty per cent.

It was a type-token ratio. It counted how many unique words appeared in a block of text, divided by the total, and inverted the result.

Read that curve again, because it is the whole problem. Text that repeats itself scored near 100. Text with a varied vocabulary scored near zero. The layer meant to reward a disciplined brand voice was rewarding repetition and punishing range.

It also could not have measured anything across pages. By the time the scoring code received the text, the scraper had already fetched every page, split them into paragraphs, removed duplicates, and joined the survivors into one string. Page boundaries did not exist at that point. The one signal a cross-page measure needs had been deliberately thrown away upstream, for perfectly good reasons that had nothing to do with scoring.

The distribution shows what that produced.

Nothing in a category of professionally built sites scored above 55 out of 100 on voice consistency. The median was 24.5. Two sites scored zero, one of which has some of the most disciplined product copy on the internet.

When every result in a category clusters at the bottom of a scale, the honest first assumption is not that the category is bad. It is that the instrument is.

What we changed

The scraper now keeps each page's text alongside the flattened version it already produced. Voice consistency is the mean pairwise cosine similarity between the term vectors of the pages actually read, with stopwords removed so the measure is about the words a brand chooses rather than the grammar every site shares.

One decision inside that is worth stating. If fewer than two pages are readable, the function returns the midpoint rather than zero. Returning zero would have scored every one-page site as maximally inconsistent, which is a claim the evidence does not support. Not enough data is a different answer from a bad result, and a measurement tool should be able to say so.

Buyer alignment went from three signals to five. The two additions are the questions competent sites actually fail: does the copy name a real role rather than addressing "companies", and does the page state the reader's problem before it describes the seller. On three sites that had never been scored, the new formula returned 70, 42 and 42. The old one would have returned 100 for all three.

The part that is still wrong

Fixing the mechanism is not the same as fixing the number. So we measured what a consistent site actually looks like, by taking several sites and computing the mean cosine between their own pages.

A genuinely consistent site sits around 0.55 to 0.68 against itself. Not 0.9, and nowhere near 1.0. Different pages talk about different things, which is what pages are for.

Our scoring feeds raw cosine straight into a 0 to 100 scale. So a well-run site earns roughly 40 out of 70 available points and the report tells almost every visitor they are inconsistent. The mechanism is now correct and the scale is not calibrated, and those are two different problems.

We are not going to invent the curve. Setting it properly means running enough scans to see the real distribution under our own extraction, then fixing the floor and ceiling from that. Until we do, the consistency number is directionally useful and should not be read to the decimal point. We would rather say that than ship a figure that looks authoritative and is not.

What this is actually about

Every content scoring tool, ours included, is a set of opinions with arithmetic on top. The arithmetic is easy to check. The opinions hide.

Both of these defects survived because each individual report was plausible. A number between 0 and 100 next to a confident label is very good at not being questioned. What exposed them was the cheapest possible test: run the thing on a set of comparable inputs and look at the shape of the answers.

If a column does not move, it is not measuring. If everything clusters at one end, suspect the ruler before the world. And if a metric rewards the opposite of the behaviour it is named after, no amount of confident presentation will save the conclusion built on it.

We publish the scoring version with every scan for this reason. These changes ship as 1.1.0, so anything scored under 1.0.0 is comparable only to itself.