Method
How we test, and how we score.
Same brief, same day, same laptop, same stopwatch. Every price re-checked by a human. Every interruption counted.
Weights
Accuracy35%
Usefulness30%
Friction20%
Hand-holding15%
Verdicts
- Cleared · overall 8.0 or above
- Works with caveats · 6.0 to 7.9
- Not yet · under 6.0
Rubric (each criterion scored 1 to 10)
| Criterion | 1 to 2 | 3 to 4 | 5 to 6 | 7 to 8 | 9 to 10 |
|---|---|---|---|---|---|
| Accuracy and verification · 35% | Under 50% of prices and availability match the live check | 50 to 69% match | 70 to 84% match | 85 to 94% match, nothing invented | 95%+ match within £10 per item, every link lands on the right page |
| Usefulness · 30% | No usable plan | Partial plan, major gaps | Complete but misses 2+ rules or over budget | Complete, all rules met, minor gaps | Complete, all rules met, under budget, a plan you’d actually book |
| Friction · 20% | 9+ interventions or 35+ minutes of human time, or a safety fail | 6 to 8, or under 35 min | 4 to 5, or under 20 min | 2 to 3 and under 10 min | 0 to 1 and under 5 min |
| Hand-holding · 15% | 6+ re-prompts or corrections, or gave up | 4 to 5 | 3 | 1 to 2 | None: one brief, one result |
Hand-holding counts help with thinking (re-prompts, corrections). Friction counts help with access (sign-ins, codes, CAPTCHAs, confirmations). Between bands, we use the worse one.
Rules we keep
- Every agent gets the brief verbatim. We only reply when it asks a question.
- All agents run on the same day, back to back, so prices drift as little as possible.
- Agents stop at the payment page. We never enter a card.
- Every price and availability claim is re-checked live within 30 minutes.
- We publish the logs, failures included, with the date tested.
- Sponsors never affect scores or see results early. A sponsor’s product is never a scored agent in an issue it sponsors.
- Sherbot, Hamish’s own AI agent, helps with research and drafting. When it competes, it’s labelled and scored like the rest.