Use records the team already reviews.
Operative note, active order, therapy note, and discharge draft; no interface build is required for a silent retrospective study.
No-integration validation workspace
Prove it before integration.
Import consecutive, institutionally de-identified postoperative packets. LimbRelay checks each source set, two reviewers establish the reference answer, and the lab calculates whether the product passes or deserves to be killed.
No PHI. No chart action.Files are read in browser memory and are never submitted by this prototype.
Automate an acknowledged manual audit.
The FFPOM protocol explicitly asks reviewers to compare weightbearing in the discharge summary with the operative note.
Read the protocol ↗Separate system output from clinical truth.
Two independent reviewers decide whether each discrepancy is actionable. Disagreement requires adjudication before metrics count.
Browser-local batch audit
Turn a flat note export into reconciled packets.
Use one spreadsheet row per source document or the original JSON contract. Rows group by opaque packet ID; one invalid packet is rejected without hiding the remaining results.
Current source50 controlled synthetic fixturesSynthetic fixtures loaded. Reviewer fields are intentionally blank.
Locked 14-column source export100 packets · 8 MB · nothing transmitted
This is not an approved PHI system. The direct-identifier screen is a last guardrail, not a de-identification service.
1 row = 1 sourceRows join on packet_idQuoted line breaks supportedUnknown columns rejected
Current run50 packets
- Conflict packets
- 23
- Missing-field packets
- 17
- Terminology packets
- 5
- Exact quote support
- 100.0%
Packet-level evidence
Join the agent output to an independent reference.
Reviewer 1 and Reviewer 2 work separately on the source-only route. This workspace imports their locked files, joins them by assignment fingerprint, and requires consensus or adjudication before a packet contributes to performance metrics. Reviewer decisions must be locked before either file is imported here.
1
Export one blinded assignmentSynthetic labels, expected answers, and system output are stripped.
2
Review independentlyEach reviewer uses the source-only workspace and exports a separate locked result.
3
Open blinded reviewer workspace Join both locked files hereIncomplete or mismatched assignments fail closed.
SYN-001
Source check readyLeft tibial plateau fixation
Trauma to inpatient rehabilitation
source agreementNo material mismatch detected
This means the supplied sources agree under the active ruleset. It does not establish clinical correctness.
Joined reference decisions + adjudicationManual entry after viewing the system output is not independent evidence.
3 source documentsIndependent field findingsAdjudicated reference findings — not system labels
Not yet applicableReviewer 1 findingsSelect the packet decision first
Reviewer 2 findingsSelect the packet decision first
Adjudicated findingsSelect the packet decision first
When both reviewers agree on the packet and exact findings, their finding set becomes the reference. Any disagreement requires adjudication. LimbRelay output never breaks the tie.
Predeclared pass / fail criteria
Let the evidence kill the product.
A passing synthetic benchmark unlocks a real study; it never substitutes for one. These gates remain pending until 50 real, consecutive packets have independent reference decisions.
Evidence report
Counts, uncertainty, and reference reliability.
Point estimates without denominators can make a small pilot look certain when it is not. Every proportion below carries its underlying count and a Wilson 95% interval.
0/0 pairs · 95% interval pending
Agreement beyond the reviewers' category marginals.
0/0 packets · 95% interval pending
0/0 actionable · 95% interval pending
0/0 system findings · 95% interval pending
0/0 reference findings · 95% interval pending
Field-level error accounting0 packets with a complete field reference
TP = matched finding · FP = system-only finding · FN = reference-only finding
| Field | TP | FP | FN | Precision | Recall |
|---|---|---|---|---|---|
| Affected limb | 0 | 0 | 0 | Pending | Pending |
| Weightbearing | 0 | 0 | 0 | Pending | Pending |
| Duration / phase | 0 | 0 | 0 | Pending | Pending |
| Restriction rationale | 0 | 0 | 0 | Pending | Pending |
| Brace / orthosis | 0 | 0 | 0 | Pending | Pending |
| Range of motion | 0 | 0 | 0 | Pending | Pending |
| Walking aid | 0 | 0 | 0 | Pending | Pending |
| Review point | 0 | 0 | 0 | Pending | Pending |
pending0/50
Consecutive adjudicated packets
At least 50
pending0/50
Two-reviewer reference
Two independent decisions per packet
pendingPending
Pre-adjudication agreement
At least 80% raw and κ ≥ 0.60; if κ is not estimable, at least 90% raw
pending0/50
Field-level reference
Locked field findings for every adjudicated packet
pendingPending
Actionable packet rate
At least 10%
pass100.0%
Exact-quote support
100%
pendingPending
Material-discrepancy sensitivity
At least 95%
pending0
False releases
Zero
pending0
Laterality false releases
Zero
pendingPending
False-block rate
At most 10%
pendingPending
Median assisted review
Under 120 sec across at least 20 timed packets
pendingPending
Median time saved
At least 180 sec across at least 20 timed packets
Technical loop and study mechanics
- Fifty controlled fixture variations execute reproducibly.
- Every extracted claim is checked against an exact quote.
- Invalid packets fail closed without hiding valid packets.
- Agreement, uncertainty, field errors, and timing intervals are computable.
Clinical usefulness and willingness to pay
- Real discrepancy prevalence in consecutive discharges.
- Performance on local documentation and abbreviations.
- Whether reviewers save meaningful time.
- Whether a service will pilot instead of using EHR rules.