Results Summary
What was the project about?
A patient’s preference for a treatment may affect how well the treatment works. For example, if patients prefer a specific medicine, they may be more likely to take that medicine.
Traditional randomized clinical trials can’t tell how much patient preferences affect how well a treatment works. But a two-stage clinical trial might. In a two-stage trial, researchers assign patients by chance to one of two groups. In the first group, researchers assign patients by chance to get a specific treatment, regardless of their preference. In the second group, patients choose their treatment. In a two-stage trial, researchers can compare health outcomes for patients who choose their treatment with patients who don’t. But few methods exist for researchers to design and analyze this type of trial.
In this project, the research team developed new statistical methods for two-stage trials. The team wanted to find out how many patients are needed for two-stage trials to provide accurate results. They also wanted to learn how to measure whether patient preference for a specific treatment affects patients’ health outcomes.
What did the research team do?
The research team developed statistical methods to figure out how patient treatment preferences affect different types of health outcomes. One method can be used with the types of outcomes that could be true or false, such as having or not having a symptom. Another method was for outcomes that can be counted, such as the number of symptoms a patient had. The team also developed a formula to calculate the number of patients to include in the study.
The research team created test data using a computer to check how the methods worked. The test data looked like data from two-stage trials. The team also tested the methods using data from two example trials.
The research team worked with other researchers and statisticians to develop and test the methods.
What were the results?
The new methods worked well to figure out how many patients to include in a two-stage trial, how well treatments worked, and how patient preferences affected health outcomes. The methods worked in studies with many or few patients. The methods also worked in the example two-stage trials. Studies needed to include more patients when different groups of patients, such as men and women, have different treatment preferences.
What were the limits of the project?
The research team didn’t try all possible ways of testing the methods using the computer program. The methods may not work well if many patients are unsure of which treatment they want.
Future research could develop other methods for analyzing results of two-stage trials.
How can people use the results?
Researchers can use these new methods to design two-stage trials and learn how patient treatment preferences affect how well treatments work.
Professional Abstract
Background
Patient preferences for treatment may affect how treatments work. Conventional randomized clinical trials cannot measure the effect of patient treatment preference on health outcomes, but a two-stage clinical trial study design can. In these trials, researchers first randomly assign patients to either the random or choice group. Second, in the random group, researchers randomly assign patients to a treatment, while patients in the choice group can choose the treatment they prefer.
In two-stage trials, researchers can measure
- Treatment effect, or the average effect of the treatment on health outcomes
- Selection effect, or the comparison of patients who are assigned a treatment with patients who choose a treatment, regardless of the treatment received
- Preference effect, or the comparison of patients who receive their preferred treatment with those who do not
Few methods exist for designing and analyzing two-stage trials. Specific gaps in methodology include calculating the sample size and analyzing noncontinuous treatment outcomes, such as binary or count outcomes.
Objective
To develop statistical methods for estimating sample size and measuring treatment, selection, and preference effects for noncontinuous outcomes in two-stage clinical trials
Study Design
| Design Element | Description |
|---|---|
| Design | Statistical modeling, simulation studies |
| Data Sources and Data Sets | Simulated data |
| Analytic Approach |
|
| Outcomes | Type I error for normal approximation and exact distribution, power |
Methods
The research team developed a binomial statistical model and a Poisson statistical model for analyzing different noncontinuous outcomes. In each model, the team developed estimators of the treatment, selection, and preference effects for both small and large sample sizes and derived sample size formulae.
To test the performance and properties of the statistical methods, the research team created simulated data by varying the rate of preference for a treatment, rate of treatment response, and sample size. The team also applied the new methods to calculate sample sizes and treatment effects using two two-stage trials with noncontinuous treatment outcomes as test examples.
Statisticians and researchers with expertise in clinical trial design, statistical methods, and statistical programs helped develop and test the methods.
Results
The research team applied the new methods to the test data to calculate sample size and measure the preference and selection effects for noncontinuous outcomes in two-stage clinical trials. Applying the new methods with varying sample sizes, the team found that for both binomial and Poisson models
- Normal approximation of selection and preference effects was possible when the sample size was relatively small (n=50 to 200) and response was moderate (30% to 50% response rate; average number of events was 5).
- Exact calculations for selection and preference effects were appropriate when the sample size was small (n<50).
Two-stage trials require greater sample size when they include multiple groups with different treatment preferences.
Limitations
Although the study showed the new methods to be effective for estimating selection and preference effects, they may not be useful when many patients are undecided on a treatment.
Conclusions and Relevance
The new methods can estimate sample size for two-stage clinical trials and measure treatment, selection, and preference effects for noncontinuous outcomes.
Future Research Needs
Future research could develop and test additional solutions for methodological gaps in design and analysis for two-stage clinical trials.
Final Research Report
View this project's final research report.
Journal Citations
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers asked how the presence or absence of patient preferences and self-selection for treatments might interact with treatment effects in different ways. The researchers responded that they could not conduct all possible simulations and relied on their stakeholders to decide which would be most relevant. The researchers added that when they allow many parameters to vary, it would become complicated to describe simulation results and to tease out the import of findings. The researchers noted that this is a limitation of the study.
- The reviewers said that the researchers’ method of adjusting for additional variables was limited in the types of variables it could accommodate whereas a method that they cited did not present such restrictions. The researchers explained that their goal was not to adjust for additional variables or covariates, but to be able to separate data into subgroups via stratification. The researchers said they cited the work the reviewers suggested, but said the method was better suited for analysis than for using when developing a new method.
- The reviewers asked how the study’s two-stage trial design for estimating preference effects would differ from what would be obtained in a fully randomized preference trial where participants are asked their treatment preference but then randomized to treatment regardless of preference. The researchers explained that while a fully randomized preference trial would provide an unbiased estimate of preference effect, such a design would increase the bias in estimating selection effect. Also, treatment effect would only be valid for the groups where researchers randomized participants to their preferred treatment. The researchers also noted that it seems unethical to ask participants their treatment preference but ignore that information when assigning treatment groups.
- The reviewers asked the researchers to address the potential causal effects related to assigning patients to treatment based on randomization versus patient preference. The reviewers noted that causal definitions of selection and preference effects would be important to consider when discussing methods for a two-stage randomized trial design. The researchers responded that addressing the causal-related definitions of selection and preference was beyond the scope of the study but did add to the discussion that consideration of these other effects would be important to generalizability of the approach.
Conflict of Interest Disclosures
Project Information
Patient / Caregiver Partners
No information provided by awardee
Other Stakeholder Partners
Stephen Walter, PhD. McMaster University Erich Greene, PhD. Yale University Frederick Altice, MD. Yale University Maria Ciarleglio,PhD. Yale University Morris Weinberger,PhD. The University of North Carolina at Chapel Hill Fangyong Li, MS. Yale University