Case study 01 · Web3Connect.com · Deep dive 1 of 3
Designing a ranking customers could trust
Competitors in this market sell the top spot in search results. I didn’t, because customers stop trusting a ranking once they know it is for sale. Instead I built a quality score out of verified reviews, put a hard limit on what money could do to it, and published the whole formula so anyone could check.
Part of a larger project
Web3Connect.com is a B2B reviews marketplace for the blockchain industry that I founded, designed and built on my own. This page covers the work that happened before any code was written: the research, the category structure, and the ranking model everything else was built on. For the background and the other two deep dives, start at the Web3Connect.com overview.
The problem nobody was solving
I spent six years within blockchain-related businesses buying B2B services. Finding a qualified supplier was hard enough, but working out whether to trust them was even harder. Teams often operate anonymously, companies are young, they are spread across dozens of countries, and the usual signals of credibility — a long trading history, a recognisable client list, a local registered company — mostly don’t exist.
The big B2B review sites had not moved into this industry. G2, Capterra and Clutch were all running successful businesses in adjacent markets and ignoring this one.
Rather than assume my own experience was typical, I went looking for evidence of how the industry actually recommends suppliers. I processed thousands of messages from industry Slack and Telegram groups to work out who was recommending whom, and for what, then mapped it:
That was the business case. Reliable information about who was good already existed, but it was stuck in private group chats and only helped the people who happened to be in them. The product was a way to make it public.
Building the categories, then testing them
I wrote the first version of the category structure by hand, based on six years of buying these services myself, and checked it against the small directories that already exist for individual niches like crypto accounting and security auditing.
The obvious risk is that categories built from my own experience only fit my own experience. So I tried to prove them wrong. I took the attendee list from a major industry conference — a few thousand companies — and checked whether each one had somewhere obvious to go.
It failed in a useful way. Ordinary suppliers fitted fine. What didn’t were incubators, foundations, investment funds and grant programmes — organisations that are clearly part of this industry but don’t sell a service in the normal sense. They have no price list, often no clients, and the people they deal with are receiving money rather than paying it.
The quick fix was to add more categories. The better fix was to change how the data was structured underneath.
Decision: one flexible record type instead of many rigid ones
Instead of designing a separate data structure for each category, I built one record type — an offering — holding everything every category needs: name, description, category, reviews and score. Individual categories then add their own extra fields on top. A security auditor gets fields for which blockchains it covers; a grant programme gets funding stage and typical cheque size.
This abstraction allowed larger partners (like Consensys) to list offerings across multiple offering types (services, products, grants, capital) within multiple categories using a consistent interface.
It improved the overall maintainability and scalability of the platform.
Why I didn’t sell the top spot
I looked at the competitors across four factors: how they make money, how they verify reviewers & partners, what standards they enforce, and where they have damaged their own reputation. The same problem came up repeatedly, and it comes from the obvious way to make money from a ranked list — selling the top position.
There are two problems with that.
It limits revenue up-side potential. There is only one top spot per category, so your income is limited by how many spots you have rather than by how many companies want to buy.
It destroys trust. Customers come to a review site for an honest ranking. A ranking you can simply buy is an advert, and buyers work that out quickly.
Nobody trusts the top result if it’s paid.
So I looked at industries that have solved this. Google Ads and Amazon both work the same way: you don’t buy a position, you buy a percentage improvement on a quality score you have to earn first. A poor advertiser pays more and gets less. That was the model I wanted.
The formula
The entire commercial model comes down to one line:
Final Rank = Quality Score × Paid Boost
Quality Score 0–100, earned. Paid Boost capped at 1.4×.
The Quality Score is built from four weighted factors, three of them on-site and one off-site:
-
Verified reviews45%
The average rating from verified client reviews of this specific service. The hardest part of the score to fake.
-
Trust tier25%
How many reviews, in bands: New (0–2) scores 20, Rising (3+) scores 60, Established (10+) scores 85, Top Rated (25+ at 4.5 stars or above) scores 100. Counting volume separately stops one five-star review beating a company with a long track record.
-
Profile completeness20%
How complete the service description is (10%) and the company profile (10%). This is the one part a brand-new company can improve immediately, for free — deliberately, so starting from zero isn’t hopeless.
-
Domain authority10%
A standard measure of how credible a company’s website is. This is the only part that comes from outside the platform. An established firm will likely already have a high score and a new firm can’t earn it quickly – which pushes them to get reviews instead.
The paid boost cap is the important part. Because payment multiplies a score you earned rather than buying you a position outright, money only ever gets you a percentage more than you already deserve. Past a certain gap, it stops helping at all.
What the 1.4× cap guarantees
To overtake a service scoring 71 — even with the highest paid boost plan — you'd have to score at least 51 yourself, because 51 × 1.4 is the lowest score that can be paid to pass 71. Below 51, no amount of money wins.
A badly-reviewed company cannot meaningfully buy its way past a well-reviewed one.
Here it is working on the live site. A category listing sorted by quality score, with a small firm called Block3 Finance at 71 on the strength of one review, and Deloitte at 34:
Rating the service, not the company
Every competitor rates the company. A firm gets one rating, and that rating follows it into every category it appears in. The problem is that firms are rarely equally good at everything they sell. An agency might be genuinely excellent at security audits and mediocre at marketing — and rating the company lets the reputation from the audits sell the marketing.
So I decided that reviews attach to individual offerings instead. One company can list many offerings, and each is reviewed, scored and ranked separately in its own category.
What it cost me
This caused the worst onboarding problem the platform had. Companies couldn’t tell the difference between their company profile and their individual offering listings — they provided thier company overview then often copy-pasted it into a single 'offering' mapped to broadly unrelated categories. This would cause irrelevant reviews downstream if not addressed. I found this by watching PostHog session recordings and reviewing new partner applications, and fixed it by adding educational content to the onboarding process, which fixed the issue.
The structure was right. The explanation was missing. Full account in deep dive 3.
Publishing the formula
A fair ranking that nobody can check undermines credibility.
Publishing the weightings does invite people to try to game them, however the protection is in the weightings themselves. The two biggest factors — average rating and number of reviews, 70% between them — are the two a company cannot award itself. Every review needs a LinkedIn-verified reviewer, an email address matching their company domain, and manual admin approval before it appears.
So I published all of it: the formula, every weighting, the thresholds, the cap, and a clear label for payment-boosted.
Sticking to it when it was expensive
After launch, I received three reviews for one firm citing fake websites, about projects I couldn’t verify, in the most valuable category on the site — at the exact point when reviews were the thing I most needed. I rejected all three. The platform operator is forced to maintain a high standard due to transparency, which enforces the trust proposition of the platform. See deep dive 3.
What partners could actually buy
The paid plan buys three things: a small 1.05× visibility boost across all the partner's offerings with the option to buy more up to the 1.40× cap, a detailed analytics view, and the removal of competitors from your own pages.
Free profiles show six competing vendors alongside your own. Paying doesn’t give you an advantage so much as remove a disadvantage. A paying partner has those competitors removed from their profile.
The model rests on three assumptions:
- Partners want and need more B2B leads.
- Partners will spend marketing budget competing for visibility in popular categories.
- Partners would rather not have competitors shown on their own profile pages.
The first two were tested properly after launch and held up considerably less well than I had assumed — that is the subject of deep dive 3. The third was doing the most commercial work and got the least testing, which in hindsight should have been validated earlier.
To hit the tight launch deadline, I postponed the build-out of the visibility boost purchasing flow. The rationale was it would be months before traffic & reviews critical mass attracted even a base paid account. It was a practical decision prioritising speed to market over a finished product.
What it produced
The ranking model changed how I positioned the whole thing. I had started with directory language — database, search, listings — but if the ranking is the point, that undersells it. The site launched as a trusted network instead, with how the ranking works stated on the homepage rather than buried in a help page.
It also produced something companies would promote for me. If the ranking is earned and the formula is public, a “#1 in category” badge is a credential rather than an advert — the kind of thing a firm puts on its own website without being asked:
Outcome
The ranking system shipped and ran across 2,500+ companies, with 136 competing services in the largest category alone. The formula, weightings, thresholds and cap were all published. It survived its first real test when I turned down fake reviews in my most valuable category rather than bend the rules.
One thing it never proved: I never got enough reviews onto the platform to find out whether companies would genuinely compete on quality at scale. The model held up. The market never got engaged enough to really test it.
What I learned
Five things
- Trust requires transparency. “We rank fairly” is a slogan. “Rank = Quality Score × Paid Boost, capped at 1.4×” is a claim anyone can test and hold me to. That is what made it credible.
- Test your assumptions early. My categories taxonomy looked complete until I tried mapping a conference vendors list to it. What that exposed changed how the data was structured, not just the category names — at the only point where changing it was still cheap.
- Rate the thing the customer is actually buying. Companies aren’t equally good at everything they sell. Rating at company level would have let a strong reputation in one area cover for a weak service in another, which is the exact problem the site existed to fix.
- Charging to remove a problem can work as well as charging for a benefit. Removing competitors from a company’s own page makes money without pushing anyone down the rankings.