How to evaluate an AI medical record tool on your own file
Answer in brief: test the tool on a closed file you already know, against 10 facts you wrote down before uploading, including 2 negatives, 1 tricky date, 1 duplicate and 1 missing record. Score it with the sheet below. A tool passes at 80 of 100 points with no stop failure; a flipped negative, an invented finding, no BAA or no written no-training term ends the trial.
Limits: Vendor facts come from each vendor's own website, read 2026-11-12, and we haven't tested every tool; prices and features change, so check the vendor before you buy.
Why vendor demos don't tell you enough
A demo file is chosen by the vendor, usually clean, typed and free of duplicates. Your files have fax headers, handwriting, the same visit 3 times and a referral nobody followed up. The only test that predicts how a tool will do on your work is your work. Most vendors offer a trial (the directory lists terms for each), and every trial should start with a signed BAA, because a real file contains PHI.
Pick a closed case where you know the answers. 300 to 1,500 pages is enough to show the failure modes without making the scoring a project. If you can't use a real file, use the synthetic test page further down, which reproduces the most common traps on a fictional patient.
The 5-step test protocol
- BAA first. Get the signed BAA and a written answer on model training before you upload. If the vendor won't sign for a trial, stop here.
- Write down 10 facts from the closed file that you would stake your signature on, each with its page number: 2 negatives ("denied loss of consciousness"), 1 date that is easy to get wrong (a fax date next to a date of service), 1 page that appears twice, 1 record you know is missing, and 5 key findings or events.
- Upload the whole file as it arrived, not a cleaned-up version. Note the time from upload to usable output.
- Score the output with the sheet below. For every fact, open the page the tool cites and confirm the page says it.
- Decide on the score, then repeat with a second file of a different type if the first one passes. 1 file is a test; 2 files are evidence.
Scoring sheet with a pass mark
Enter your results. The sheet adds up the points as you type; nothing is sent anywhere, and it works without a login. Stop items end the trial whatever the score.
Why these weights: facts and citations carry 70 of 100 points because they are the 2 things you sign on. With every other check met, a tool still passes with 2 missed facts and 4 bad citations, and fails with 3 of each. These weights are our judgment, not a standard; change them to fit your work.
Worked example: a synthetic test page with planted traps
If you can't use a real file, paste or print the excerpt below into a PDF of other synthetic pages and upload it. It is synthetic, no real patient: the names, dates and facilities are invented. It packs 6 traps into 5 pages.
SYNTHETIC RECORD, NO REAL PATIENT. Patient: Dana Testcase, DOB 1979-02-14.
p. 14. Fax header: 2025-03-11 09:42. Riverbend ED note, date of service 2025-03-04. Slipped on wet floor at work. Denies loss of consciousness. Denies numbness or tingling in the legs. Straight leg raise negative bilaterally. Lumbar strain. Ibuprofen 600 mg.
p. 15. Same ED note as p. 14, from the insurer's production, different fax header (2025-03-18).
p. 22. Oakview PT note, 2025-04-22. Visit 6 of 12. Reports left leg numbness since last week. Progressing slowly.
p. 30. Billing statement, Lakeside Imaging. MRI lumbar spine without contrast, date of service 2025-04-02. No report attached.
p. 31. Primary care note, 2025-06-10. Back pain persists. Referred to orthopedics.
| Trap | Correct output | Common wrong output |
|---|---|---|
| Negative (p. 14) | No loss of consciousness; no leg numbness at the ED visit | "Leg numbness" listed among ED findings |
| Date (p. 14) | ED visit on 2025-03-04 | ED visit on 2025-03-11 (the fax date) |
| Duplicate (p. 15) | 1 ED visit, p. 15 marked as a duplicate | 2 ED visits a week apart |
| Onset (p. 22) | Leg numbness first reported mid-April 2025, after the ED visit | Numbness "since the injury" |
| Missing record (p. 30) | MRI report for 2025-04-02 missing; request from Lakeside Imaging | MRI "performed", with an invented result |
| Missing records (p. 22, p. 31) | PT visits 1 to 5 and any orthopedic note are not in the file | No flag |
The MRI line is the one to watch. A tool that writes an MRI result when the file holds only a bill has invented a finding, which is a stop failure even if everything else is right.
Questions to ask the vendor after the test
- When the tool couldn't read a page, did it say so, or did it skip it silently?
- Can I see a list of every source document the output used?
- What happens to my file and the output when I delete the case?
- How is the price counted: pages uploaded, pages after removing duplicates, cases or seats?
- If a citation is wrong, how do I report it, and do you fix the cause or just the case?
Write the answers next to your score. Then use the cost calculator to compare the tools that passed on price.