Project Summary

Doctors and other healthcare providers make many decisions when they are not sure what choice is best for their patients. For example, when a prescriber chooses between two slightly different diabetes drugs, they may be unsure which drug is best. An example is choosing between two different types of long-acting insulin, where prescribers know that both work well but think one might be slightly better than the other.

For these kinds of questions, the only way to get a reliable answer on what is best for the majority of patients is to make the decision at random and see how patients do after. For example, researchers might conduct a study where they choose at random to give diabetes drug A to some patients and diabetes drug B to others. But, because making choices at random is so different from regular medical care, doing this kind of “randomized trial” is difficult and expensive. It is so difficult that most questions never have a randomized trial done. This keeps patients from getting the best possible care.

The project team is interested in a new kind of randomized trial that works through the computer prescribers use to look up patient information and order medications. Sometimes patients and their doctors (or other prescribers, such as nurse practitioners or physician assistants) have very little reason to prefer one drug over another when the drugs are only slightly different. Small details about the computer system, like which drug is listed first in a menu of options, might nudge the prescriber, making them more likely to select that drug. The project team believes it could be useful for research to change those details of the computer system at random, giving providers nudges toward one drug or another by chance. This would make prescribers slightly more likely to choose one drug or another based on chance alone. The study’s researchers expect this kind of study would lead to the same kind of reliable answers to questions as regular randomized trials do. Because these studies would be easy and safe, they could be conducted to answer many of questions and hopefully improve medical care for more patients.

Before this kind of study can be attempted, the study team must learn what rules patients and others (such as prescribers and regulators) think the team should follow. For example, in regular randomized trials, it is important to tell patients about the study and obtain their permission for that specific study. Often this means reading and signing a form that is more than 10 pages long. This is partly because patients expect that prescribers have some idea on what drug is best for them and are not making any decisions by chance. For this new kind of study, the prescriber and patient may be affected by the nudge from the computer, but they are still making the final choice of what drug to take. Patients and providers might not feel they need to sign a form or even discuss the study at all, because they will still be freely choosing the treatment they prefer.

To find out what patients think about these questions, the project team will conduct surveys and focus groups with about 100 people from the general public to tell them about this new research idea and see what they think the rules should be. Researchers will also talk to prescribers and members of the committees that decide whether specific research studies are ethical. The team will also study the effects past changes in details of the computer system at its hospital have had on prescribers’ treatment choices, to make sure these nudges work as expected.

Past changes were made at once across the study team’s hospital to improve care, not by chance. In its final step, the study team will randomly change the nudge in the computer, so one prescriber is nudged toward one treatment and the other is not. The research team may use this study to help determine which kind of insulin is best to use in the hospital.

*Methods to Support Innovative Research on AI and Large Language Models Supplement
This study received supplemental funding to build on existing PCORI-funded comparative clinical effectiveness research (CER) methods studies to improve understanding of emerging innovations in large language models (LLMs).

Project Information

James Flory, MD, MS
Memorial Sloan Kettering Cancer Center

Key Dates

36 months
November 2022


Award Type
State State The state where the project originates, or where the primary institution or organization is located. View Glossary
Last updated: March 15, 2024