This project has results

Improving Trial Design and Analysis for Treatments for Rare Diseases

Results Summary and Professional Abstract

Results Summary

What was the project about?

A rare disease is one that affects fewer than 200,000 people in the United States. Because few people have these diseases, clinical studies on treatments can be hard to conduct. One way to study rare disease treatments is with an snSMART study.

snSMART studies have two stages. In the first stage, researchers assign patients to a treatment by chance. In the second stage, patients may stay with the same treatment or switch treatments. Patients stay on the same treatment if it’s working well. If the treatment isn’t working, researchers assign patients by chance to a new treatment.

snSMARTs can help researchers learn more from a smaller number of patients than a standard clinical study. But most current methods for analyzing snSMARTs use data only from the first stage, which can lead to inefficient results.

In this project, the research team developed and tested new methods that use data from both stages to analyze snSMARTs. The team compared results from the new methods to actual treatment effectiveness to see

Bias, or whether results are too high or too low
Efficiency, or how big the difference is between the results and actual treatment effectiveness

What did the research team do?

The research team created two Bayesian and two frequentist methods. Bayesian methods use findings from previous studies in the analysis, but frequentist methods don’t.

Using a computer program, the research team created test data sets with 45–300 patients that looked like data from an snSMART study. Using the test data, the team compared the new and current methods to see which worked better to find the best

First stage treatment
Sequence of treatments

Clinicians that treat rare diseases, a patient with a rare disease, and a family member gave input on the study.

What were the results?

All methods worked well enough to see which first stage treatment or sequence of treatments worked best. To find the best first stage treatment, the Bayesian methods using data from both stages worked better than methods using only first stage data and the frequentist methods. To find the best treatment sequence, the Bayesian method was the most efficient but could result in biased estimates.

The research team also created an online sample size calculator that figures out how many patients should be in an snSMART study.

What were the limits of the project?

The research team compared the methods using test data. Bayesian methods may not get accurate results if assumptions about data aren’t met. The new methods don’t apply to designs using a placebo, or a pill that doesn’t contain medicine. Also, the snSMART design is most appropriate to study diseases with symptoms that are stable over time and don’t change.

Future studies could test the methods when data are missing. Researchers could also develop methods to analyze data from an snSMART with a placebo.

How can people use the results?

Researchers can consider using the new methods and the online calculator when designing and analyzing snSMART studies.

Professional Abstract

Background

Clinical studies of rare diseases often have small sample sizes. A small n sequential multiple assignment randomized trial (snSMART) is a two-stage design for small samples. In the first stage, snSMART studies randomize patients to one of two or more treatments. In the second stage, patients who respond to their initial treatment continue with that treatment while patients who do not respond are re-randomized to another treatment. Most existing statistical methods for analyzing snSMART designs only use data from the first stage, which can lead to inefficient estimates.

Objective

To develop new methods for designing and analyzing snSMARTs that increase efficiency and have low bias

Study Design

Design Element	Description
Design	Simulation studies
Data Sources and Data Sets	Simulated data created with sample sizes between 45 and 300 patients
Analytic Approach	Estimating response rates to determine optimal first stage treatment: Compared 2 models that use data from 2 stages to determine the optimal first stage treatment (Bayesian joint stage model, Log Poisson joint state model) with 2 models using only first stage data (Bayesian first stage model, first stage maximum likelihood estimates) Determining optimal treatment sequence: Compared 2 extended models (Bayesian joint stage model with multiple linkage parameters, Log Poisson joint state model with multiple parameters) with an existing method called weighted and replicated regression method Application to snSMART trials: Created an applet for other researchers to calculate sample size and power when designing snSMART trials
Outcomes	Accuracy and efficiency based on bias and root mean squared error, respectively

Methods

Using data from both stages of a snSMART studying three treatments, researchers developed two models:

Bayesian joint stage model (BJSM)
Log Poisson joint stage model (LPJSM)

The models estimate the best first stage treatment based on response rates or the percentage of patients who responded to each of three treatments, from both stages. Researchers compared simulation estimates from the new models with two models that relied on data from only the first stage:

Bayesian first stage model (BFSM)
First stage maximum likelihood estimates (FSMLE)

To estimate response rates for treatment sequences, researchers extended the new models to allow for the second stage treatment effect to depend on the first stage treatment:

BJSM with multiple linkage parameters (BJSMM)
LPJSM with multiple parameters (LPJSMM)

To identify the optimal treatment sequence, researchers compared BJSMM and LPJSMM with an existing method, weighted and replicated regression method (WRRM).

Clinicians who treat rare disease, one patient with a rare disease, and one family member gave input on the study.

Results

Optimal treatment. When statistical assumptions held, BJSM had low bias and was most efficient, as measured by the lowest root mean square error. BJSMM did not result in a substantial reduction in bias or error from the BJSM. The LPJSM, BFSM, and FSMLE all had negligible bias but were less efficient.

Treatment sequence. BJSMM can result in small, non-negligible bias but provides the most efficient estimates. LPJSMM and WRRM result in negligible bias but are less efficient compared with BJSMM.

Applet. Researchers created an applet to calculate required sample sizes and statistical power for snSMART designs. The applet uses assumptions of BJSM to determine the best first stage treatment.

Limitations

Researchers tested the models using simulation analyses. Statistical assumptions for the new models may not hold in clinical trials but can be tested by comparing results from alternative models presented in the study. The models do not apply to snSMARTs with a placebo.

The snSMART design is not ideal for studying diseases that change over time.

Conclusions and Relevance

The new models were more efficient than existing models. The new Bayesian models worked well for identifying the optimal treatment and the best treatment sequence.

Future Research Needs

Future research could assess the robustness of the methods for missing data. Researchers can also develop methods to analyze data from an snSMART with a placebo.

Final Research Report

View this project's final research report.

Journal Articles

Statistics in Medicine

Bayesian methods to compare dose levels with placebo in a small n, sequential, multiple assignment, randomized trial

Statistics in Medicine

Design and analysis considerations for utilizing a mapping function in a small sample, sequential, multiple assignment, randomized trials with continuous outcomes

Journal of Biopharmaceutical Statistics

Sample size determination for Bayesian analysis of small n sequential, multiple assignment, randomized trials (snSMARTs) with three agents

Contemporary Clinical Trials

Dynamic treatment regimens in small n, sequential, multiple assignment, randomized trials: An application in focal segmental glomerulosclerosis

Peer-Review Summary

Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.

Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:

The reviewers asked for additional details and expanded discussion on various topics, including on the difference between sequential, multiple assignment, randomized trials (SMARTs) versus small n SMARTs (snSMARTs). The researchers added details to various sections of their report and explained that their snSMART methods were specifically designed for studies involving small numbers of samples or rare diseases.
The reviewers asked whether the estimators calculated in this study could be applied to other settings or used with pooled data, even with larger sample sizes. The researchers noted that the utility of their design and methods diminish as sample size increases but can still be used. They said the greatest benefit of their snSMART design and methods are for studies involving fewer than 100 cases.
The reviewers asked whether the researchers planned to write nontechnical manuscripts to disseminate their findings to a broader audience. The researchers said they have made efforts to disseminate their findings to broader audiences and regulatory communities. However, they believed the study would be most useful for statisticians and rare disease researchers.

Conflict of Interest Disclosures

View the COI disclosure form.

Project Details

Principal Investigator

Kelley M. Kidwell, PhD

Project Status

Completed; PCORI Public and Professional Abstracts, and Final Research Report Posted

Project Title

Design and Methodological Improvements for Patient-Centered Small n Sequential Multiple Assignment Randomized Trials (snSMARTs) in the Setting of Rare Diseases

Board Approval Date

April 2016

Project End Date

August 2020

Organization

University of Michigan School of Public Health

Year Awarded

2016

State

Michigan

Year Completed

2020

Project Type

Research Project

Health Conditions

Rare Diseases

Rare Diseases - Other

Funding Announcement

Improving Methods for Conducting Patient-Centered Outcomes Research

Project Budget

$892,697

DOI - Digital Object Identifier

10.25302/11.2020.ME.150731108

Study Registration Information

HSRP20163133

Partners

Patient/Caregiver Partners

Other Stakeholder Partners

Page Last Updated:

January 12, 2021