FindMyKOL
Clinical trialNCT07199231Active, not recruitingObservational

OpenEvidence Safety and Comparative Efficacy of Four LLM's in Clinical Practice

SponsorCambridge Health AllianceEnrollment20AI (Artificial Intelligence)Generative Artificial IntelligenceLarge Language Model
View the registry record on ClinicalTrials.gov

Summary

The lay summary the sponsor registered on ClinicalTrials.gov.

OpenEvidence is an online tool that aggregates and synthesizes data from peer-reviewed medical studies, then producing a response to a user's questions using generative AI. While it is in use by a number of clinicians (including residents) today, there is little to no published data on whether the tool's outputs are accurate and whether this information appropriately informs clinical decision making. Similarly, a number of clinicians are turning to other large language models (LLM's) to assist in decision making when providing clinical care. While there have been a number of studies published on the accuracy of these LLM's responses to medical boards questions or clinical vignettes, there have been few studies to date examining their performance in a real world clinical setting, and even fewer comparing this performance. In this study, investigators have two goals: 1. To determine whether the use of the AI tool "OpenEvidence" leads to clinically appropriate decisions when utilized by family medicine, internal medicine, and psychiatry residents in the course of clinical practice. 2. To determine how the output of the OpenEvidence tool compares with three other commonly-used, publicly-available large language models (OpenAI's ChatGPT, Anthropic's Claude, and Google's Gemini) in answering common questions that residents have in the course of clinical practice. To accomplish study goal #1, investigators have enlisted residents in the above specialties to use the OpenEvidence tool in the course of clinical practice. In order to mitigate any safety risks, the residents will also use a typical reference tool for their question, which is referred to as the "Gold Standard" tool. These tools include PubMed and UpToDate. The residents will: 1. State their clinical question. 2. Query OpenEvidence, capturing their prompt and the OpenEvidence output for data analysis. All residents will undergo training in prompt engineering at the start of the study. 3. State their clinical conclusion based on the OpenEvidence data. 4. Query the Gold Standard Resource. 5. State their final clinical conclusion. 6. Answer a question on whether their clinical conclusion was modified by the Gold Standard reference. 7. Answer a question on whether they had any clinical safety concerns on the output from OpenEvidence. Attending physician Subject Matter Experts (SMEs) matched by specialty with at least 5 years of post-training clinical experience will then evaluate the residents' responses. 5 years was chosen based the book "Outliers" by Malcolm Gladwell, in which he asserts that 10,000 hours of focused practice is needed to achieve expertise in a field. SMEs will be asked to evaluate the residents' initial clinical questions and their conclusions based only on OpenEvidence. They will be asked to rate the clinical appropriateness of those conclusions on a scale of 1-10. For questions where the SME's rate the clinical appropriateness of the residents' conclusions poorly (\< 5/10), they will be asked to review the OpenEvidence output and answer an additional question as to whether the output was incorrect or the resident misinterpreted the output from the tool. To accomplish goal #2, the initial prompt entered by the residents into OpenEvidence will be copied by the research team into ChatGPT, Gemini, and Claude. The outputs from each tool (including OpenEvidence) will be surfaced to SMEs, who will be asked to rate each output based on accuracy, completeness, and bias. Likert scales will be used for these ratings. SMEs will also be asked an open-ended question to identify any patient safety issues from any of the outputs.

Investigators

The overall officials registered on this study. Names that resolve to a FindMyKOL profile link to it; the rest are shown as registered.

A profile link is shown only when the registered name resolves to a person record with a confident, namesake-checked match; ambiguous names stay unlinked.

Sites

1 registered facility, with the per-site recruitment status where the registry publishes one. ClinicalTrials.gov does not publish per-site enrollment counts.

United States
  • Cambridge Health Alliance

    Cambridge, Massachusetts, United States

Site investigator names are published by ClinicalTrials.gov only while a study recruits, so completed studies list sites without them.

Drugs studied

No drug or biological interventions are in our data for this study. It may test a procedure, device or behavioral intervention, or the intervention list isn't in our current snapshot. See the registry record ↗. Why? →

Registration & key dates

Registry attributes as recorded on the ClinicalTrials.gov study record.

Start
1 Oct 2025
Primary completion
30 Jul 2026
Completion
30 Sept 2026
Last update posted
18 Aug 2026
Registered enrollment
20
Registry id
NCT07199231

Source: ClinicalTrials.gov, via the CTTI AACT database, data as of 11 Nov 2026. Registry facts are shown as registered by the study sponsor; FindMyKOL does not interpret them.