Results Summary

What was the project about?

A rare disease is one that affects fewer than 200,000 people in the United States. Because few people have these diseases, clinical studies on treatments can be hard to conduct. One way to study rare disease treatments is with an snSMART study.

snSMART studies have two stages. In the first stage, researchers assign patients to a treatment by chance. In the second stage, patients may stay with the same treatment or switch treatments. Patients stay on the same treatment if it’s working well. If the treatment isn’t working, researchers assign patients by chance to a new treatment.

snSMARTs can help researchers learn more from a smaller number of patients than a standard clinical study. But most current methods for analyzing snSMARTs use data only from the first stage, which can lead to inefficient results.

In this project, the research team developed and tested new methods that use data from both stages to analyze snSMARTs. The team compared results from the new methods to actual treatment effectiveness to see

  • Bias, or whether results are too high or too low
  • Efficiency, or how big the difference is between the results and actual treatment effectiveness

What did the research team do?

The research team created two Bayesian and two frequentist methods. Bayesian methods use findings from previous studies in the analysis, but frequentist methods don’t.

Using a computer program, the research team created test data sets with 45–300 patients that looked like data from an snSMART study. Using the test data, the team compared the new and current methods to see which worked better to find the best

  • First stage treatment
  • Sequence of treatments

Clinicians that treat rare diseases, a patient with a rare disease, and a family member gave input on the study.

What were the results?

All methods worked well enough to see which first stage treatment or sequence of treatments worked best. To find the best first stage treatment, the Bayesian methods using data from both stages worked better than methods using only first stage data and the frequentist methods. To find the best treatment sequence, the Bayesian method was the most efficient but could result in biased estimates.

The research team also created an online sample size calculator that figures out how many patients should be in an snSMART study.

What were the limits of the project?

The research team compared the methods using test data. Bayesian methods may not get accurate results if assumptions about data aren’t met. The new methods don’t apply to designs using a placebo, or a pill that doesn’t contain medicine. Also, the snSMART design is most appropriate to study diseases with symptoms that are stable over time and don’t change.

Future studies could test the methods when data are missing. Researchers could also develop methods to analyze data from an snSMART with a placebo.

How can people use the results?

Researchers can consider using the new methods and the online calculator when designing and analyzing snSMART studies.

Final Research Report

View this project's final research report.

Peer-Review Summary

Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments. 

Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:

  • The reviewers asked for additional details and expanded discussion on various topics, including on the difference between sequential, multiple assignment, randomized trials (SMARTs) versus small n SMARTs (snSMARTs). The researchers added details to various sections of their report and explained that their snSMART methods were specifically designed for studies involving small numbers of samples or rare diseases.
  • The reviewers asked whether the estimators calculated in this study could be applied to other settings or used with pooled data, even with larger sample sizes. The researchers noted that the utility of their design and methods diminish as sample size increases but can still be used. They said the greatest benefit of their snSMART design and methods are for studies involving fewer than 100 cases.
  • The reviewers asked whether the researchers planned to write nontechnical manuscripts to disseminate their findings to a broader audience. The researchers said they have made efforts to disseminate their findings to broader audiences and regulatory communities. However, they believed the study would be most useful for statisticians and rare disease researchers.

Conflict of Interest Disclosures

Project Information

Kelley M. Kidwell, PhD
University of Michigan School of Public Health
$892,697
10.25302/11.2020.ME.150731108
Design and Methodological Improvements for Patient-Centered Small n Sequential Multiple Assignment Randomized Trials (snSMARTs) in the Setting of Rare Diseases

Key Dates

April 2016
August 2020
2016
2020

Study Registration Information

Tags

Has Results
Award Type
Health Conditions Health Conditions These are the broad terms we use to categorize our funded research studies; specific diseases or conditions are included within the appropriate larger category. Note: not all of our funded projects focus on a single disease or condition; some touch on multiple diseases or conditions, research methods, or broader health system interventions. Such projects won’t be listed by a primary disease/condition and so won’t appear if you use this filter tool to find them. View Glossary
State State The state where the project originates, or where the primary institution or organization is located. View Glossary
Last updated: April 26, 2022