AI Healthcare
Stanford ChatEHR shows benchmarks miss how clinicians really chat
By Arjun
·
10 Aug 8:07 PM
·
Via Nature Medicine
Nature Medicine published lessons from deploying ChatEHR, a large language model system inside Stanford Medicine. Authors including Nigam Shah argue that benchmark-style scores are a poor monitor once clinicians drive open-ended chats, and that hospitals need new methods to watch live performance, including unsupported claims spotted during deployment. Stanford University owns ChatEHR; the authors are Stanford employees. The piece reframes evaluation as an operations problem for medical centers, not a leaderboard race. Caveat: the article is a deployment essay rather than a randomized outcome trial, so it guides monitoring practice more than it proves clinical benefit.
Read the original →
Our summary is original writing; the full story belongs to Nature Medicine.
Sign in or create a free account to keep this story in a bookmark folder.
Bharat Hospital and Institute of Oncology in Mysuru launched an AI-powered MRI and will offer free Sunday scans for smokers and tobacco users for four weeks, The Hindu reported. Radiologists say Deep Resolve and My Assist cut scan time from about 20 to 30 minutes to 10 to 15 minutes while improving image quality, with blood samples collected for cancer biomarker study during the pilot. Chairman B.S. Ajaikumar framed the push around tobacco-linked cancer risk locally. The hospital presents the system as clinician assist, not autonomous diagnosis. It is a single-centre rollout with a short free window, so wider expansion depends on still-unpublished pilot results.
Nature Medicine reports two experiments with 623 lay people and 153 primary care physicians using a fairness-trained dermatology model plus explainable AI, including multimodal LLM explanations. Balanced skin-tone performance improved final accuracy and reduced skin-tone gaps for both groups. Effects of explanations diverged: lay users showed stronger automation bias, gaining when the model was right and losing when it erred, while experienced physicians stayed more resilient and still benefited. Showing the AI diagnosis before a human decision also risked stronger anchoring. The work is task-specific dermatology in controlled trials, so clinics should not assume the same pattern holds for every specialty or explanation style.
Union Health Minister J.P. Nadda released a Praxis-FICCI knowledge paper, AI in MedTech, at India Medical Device 2026, arguing India has digital foundations but not yet an ecosystem for routine clinical AI. The note, covered by ETHealthworld, flags AI diagnostics as the first large-scale use case and lists priorities around better data and evidence, lifecycle rules for adaptive models, and procurement plus reimbursement paths. It cites ABDM, IndiaAI, SAHI, BODH, and the 2023 medical devices policy as starting points. The paper is guidance, not a binding CDSCO rule change, so hospitals still face the same reimbursement and liability gaps.
A Nature Medicine LLM-assisted systematic review identified 4,609 peer-reviewed clinical LLM studies from January 2022 through September 2025, about 3.2 papers a day. Only an estimated 1,048 used real-world patient data, and just 19 were prospective randomized trials. Most work stayed in simulated scenarios or exam-style tasks, OpenAI models dominated evaluations, and LLMs beat human comparators in only about one third of head-to-head results, less often on real clinical data than on quizzes. That matters as hospitals buy copilots faster than evidence matures. The caveat is that the review itself used LLM screening validated on samples, so residual misclassification remains possible.