0
dangerous under-triage results across every verification gate ever run
Benchmarks
We publish our numbers. Ask our competitors for theirs.
Every figure on this page comes from versioned benchmark files, a scripted test harness, and verbatim transcripts committed to our repository. Any result can be re-run and reproduced. Last verification run: July 18, 2026.
90.2%
mean accuracy across 5 runs of the published Semigran Triage Bench — above MedAsk’s published 87.6%
96.7%
exact on Carevo’s 240-case multi-turn clinical gate
100%
safe-or-exact on both — every miss routed to a safer level, never a riskier one
The metric that matters is the one that can hurt someone.
A triage system can miss in two directions. Send someone to urgent care who could have stayed home, and you cost them an afternoon and a copay. Send someone home who needed the emergency room, and you can cost them their life. We call the second kind under-triage, we count it as an absolute failure with zero tolerance, and every gate below reports it separately. Carevo has recorded zero under-triage results across every verification gate in its history — while keeping exact accuracy high enough that the system remains useful rather than reflexively over-cautious.
Head-to-head: a competitor’s own published benchmark
In July 2025, MedAsk published Triage Bench — 45 standardized clinical vignettes (Semigran et al., 15 emergency · 15 non-emergency · 15 self-care) — and used it to report that their system beats OpenAI’s o3 and GPT-4.5 at triage. We ran the identical canonical vignettes from their public repository through Carevo’s full multi-turn interview, five times, against the same production code serving this website, scored with their metric definitions. Their rows below are their published results; ours were produced July 2026.
| System | Accuracy | Emergency | Non-emerg. | Self-care | Safety of advice | Overtriage* |
|---|---|---|---|---|---|---|
| Carevo | 90.2 (1.1) | 100 (0) | 86.7 (0) | 84 (3.3) | 100 (0) | 100 (0) |
| MedAsk | 87.6 (3.7) | 92 (5.6) | 82.7 (3.7) | 88 (5.6) | 92.9 (1.9) | 41.7 (11.8) |
| OpenAI o3 | 75.6 (3.5) | 90.7 (6) | 82.7 (3.7) | 53.3 (12.5) | 93.3 (1.6) | 72.1 (8) |
| OpenAI o4-mini | 80.4 (2.9) | 89.3 (6) | 81.3 (3) | 70.7 (8.9) | 92.4 (2.5) | 61.3 (11.9) |
| GPT-4.5 | 68.9 (3.1) | 93.3 (0) | 82.7 (3.7) | 30.7 (6) | 97.3 (1) | 91.5 (2.8) |
Values are mean (standard deviation) across runs; Carevo n=5 runs, 225 scored trials. *Overtriage counts, among a system’s INCORRECT answers only, the share that routed too high — Carevo’s reads 100% because every one of its misses was over-cautious; the others’ misses include routing BELOW the correct urgency. Safety of advice is the share of all answers at or above correct urgency: Carevo was the only system with no below-urgency advice. Being honest about the rest: MedAsk scores higher on self-care precision (88 vs 84), and at 45 vignettes the overall accuracy gap (90.2 vs 87.6) is within statistical noise on a paired test — the safety record is the robust difference. Sources: MedAsk’s public benchmark repository (github.com/medaks/medask-benchmarks) and their July 2025 results; Carevo runs are reproducible from our committed harness and outputs.
Internal gates: harder than the benchmark
Before any change ships, Carevo must pass a 240-case multi-turn clinical gate — synthetic vignettes spanning all six care levels, with labels reviewed and where necessary corrected under a named clinician’s signature. Current result: 232 of 240 exact (96.7%), 8 safe over-routes, 0 under-triage, 0 errors. Separately, a vague-patient gate tests what benchmarks usually ignore: terse answers (“kinda bad i guess”), hedging, deny-everything patients, and limited-English phrasings like “head very hot two day.” Current result: 24 of 24 correct or acceptable, 0 under-triage. A change that fails either gate does not deploy — the release process enforces this mechanically, not as a policy document.
How the numbers are produced
Deterministic routing. A language model translates patient words into structured clinical facts and phrases questions — it never chooses the care level. Routing comes from Carevo’s own engine: versioned clinical rules citing AHA, CDC, AAP, and ACOG guidance that can only raise urgency; raw-text safety nets that run before and independently of the AI; and a transparent scoring model with a proven monotonicity guarantee (a sicker presentation can never receive a lower recommendation).
Scripted, reproducible runs. An autonomous test harness plays each case as a multi-turn conversation against the production API and records verbatim transcripts. Benchmark files, harness code, rubric, and results are versioned together, and every recommendation carries the exact engine, ruleset, and knowledge-base versions that produced it in a hash-chained audit log.
Clinician-governed learning. The system’s only mechanism for becoming less cautious is a calibration layer in which every pattern traces to case rows approved by a named clinical reviewer — who has rejected proposals, and whose rejections became permanent guardrails. This loop — the Recursive Experience Engine (REE) — mines every reviewed case for calibration patterns, so the engine only ever improves: each cycle must beat the previous gate scores before it ships, and safety floors are never trainable. Emergency (911) routing is untouchable by learning, by configuration, and by design. Kill switches can revert any learned behavior instantly.
What these numbers do not claim
These are vignette benchmarks, not clinical outcomes: they measure agreement with expert-labeled cases, not downstream patient results, and the external set is modest (n=45). Carevo’s misses skew deliberately toward over-caution, which costs convenience — we consider that the correct failure direction and report it rather than hide it. Carevo is an informational navigation tool, not medical advice, and no benchmark replaces clinical judgment. In an emergency, call 911.
Want the raw transcripts, the rubric, or a live re-run?
We share the full methodology — benchmark files, scoring code, per-case transcripts, and the clinician review records — with health systems, payers, and researchers under NDA. Or just try the product the benchmarks describe.