Results Summary
PCORI funded the Pilot Projects to explore how to conduct and use patient-centered outcomes research in ways that can better serve patients and the healthcare community. Learn more.
Background
Research has shown that customized email messages can help motivate people to take steps to improve their health. Computer programs can create customized, or tailored, email messages based on information about a person. This can be done in two ways:
- Standard tailoring. The computer program uses a set of rules to select messages for a person based on that person’s characteristics. For example, if someone is a smoker, the computer program might send a message like “Quitting smoking will make your teeth look much nicer.”
- Content tailoring. The computer program looks at what a person does online, what similar types of people do, and uses machine learning to select the messages for the person. For example, if a person visits parenting websites and buys diapers, the program may recommend messages about baby food.
Content tailoring has not been well tested for communicating health messages.
Methods
The research team developed a content tailoring program that used data to learn about users and decide which messages to send them. The research team compared the new program with a standard tailoring program that was already shown to help people stop smoking.
A total of 120 current smokers took part in the study. All participants were age 18 or older, spoke English, and had access to the Internet. When participants joined the study, researchers asked about their age, sex, race and ethnicity, how much they smoke, whether they have tried to quit smoking before, and whether they felt ready to quit smoking.
The research team then randomly split participants into two groups: one group got emails from the standard tailoring program, and the other group got emails from the content tailoring program. Participants in both groups got emails for 30 days. Each email asked participants to rate whether they thought the message would help them quit smoking.
After 30 days, researchers contacted participants and asked whether the messages helped them take steps to quit and whether they had stopped smoking for at least one day during the study. The research team looked at how each group rated the daily messages. To see which computer program worked better, the research team looked at
- How many people quit smoking during the study
- How many people said the messages were motivational
Findings
Compared with participants in the standard tailoring group, participants in the group using the content tailoring program were more likely to say that the messages motivated them to quit smoking.
About the same number of participants in each group said they stopped smoking for at least one day by the end of the study.
Limitations
Researchers asked people only at the end of the study if they had quit smoking for at least one day. The results might be different if the research team had asked again six months after the study started or asked if they had quit for longer than one day.
Conclusions
Smokers who got messages from the content tailoring program were more likely to say that the messages were motivational than smokers who got messages from the standard tailoring program. Both programs helped about the same number of participants quit smoking for at least one day.
Sharing the Results
The research team has published the results of this project in medical journals (see below) and given talks at scientific meetings.
Professional Abstract
PCORI funded the Pilot Projects to explore how to conduct and use patient-centered outcomes research in ways that can better serve patients and the healthcare community. Learn more.
Background
Across health domains, computer-tailored health communication systems are effective in motivating behavior change. In computer tailoring systems, motivational messages such as “Quitting smoking will make your teeth look much nicer” are selected based on patient characteristics. Current implementations of computer tailoring (standard tailoring) combine tailoring variables and if-then rules (i.e., rules for selecting messages for different segments of the targeted population) to select messages for a patient. Outside health care, content tailoring is driven algorithmically using machine learning, in contrast to the rule-based approach used in standard tailoring systems. A special class of machine learning systems called “recommender systems” is used to select messages combining the collective intelligence of their users (i.e., the observed and inferred preferences of users as they interact with the system) and their user profiles. However, this approach has not been adequately tested for health communication.
Project Purpose
In a randomized experiment, researchers compared a standard rule-based computer tailored health communication system (standard tailoring) to a novel machine learning recommender tailoring system (PERSPeCT).
Study Design
Researchers conducted a randomized experiment. As smokers registered for the study online, they were allocated to the two groups based on a pre-specified, block-randomization allocation table (blocks of 10). Study staff were blinded to allocation during initial baseline assessment and follow-up.
Participants, Interventions, Settings, and Outcomes
Current smokers (n = 120) aged 18 years or older, who were English speaking, with Internet access were recruited.
Current smokers were identified using the electronic medical record (EMR) of a large healthcare system and recruited using an opt-out mailing procedure.
The PERSPeCT recommender system uses machine learning to select and send motivational messages using algorithms incorporating multiple data sources, including explicit feedback (message ratings of 846 previous participants), implicit ratings (usage data from 900 smokers that participated in a prior randomized control trial), participant demographics and smoking patterns, and metadata about each of the messages. The comparison was a standard tailoring system that was previously demonstrated in a prior randomized trial of 900 smokers to be effective in motivating smoking cessation (odds ratio 1.69, p = 0.04).
The primary hypothesis was that the PERSPeCT recommender system would outperform (i.e., select messages of higher influence as determined by smokers reporting “agreed/strongly agreed” that the messages influenced them to quit) the rule-based standard tailoring system.
Data Sources
During registration, smokers were asked questions about their demographic information and smoking behavior. Smokers were asked to rate each motivational email on a five-point Likert scale by clicking on a link included with the email using the following question to collect the rating: “This message influences me to QUIT smoking.” At follow-up, the perceived influence of the intervention and 30-day cessation was assessed.
Data Analysis
By randomization, researchers compared daily ratings (mean of each day’s smoker ratings) of the messages. At 30 days, researchers assessed the intervention’s perceived influence, 30-day cessation, and changes in readiness to quit smoking from baseline.
Findings
The proportion of days when smokers agreed/strongly agreed (daily rating ≥ 4) that the messages influenced them to quit was significantly higher in PERSPeCT (74 percent) than the standard tailoring system (45 percent), p = 0.02. Researchers did not find statistically significant differences between the degree to which smokers randomized to PERSPeCT strongly agreed or agreed that the intervention significantly influenced them to quit smoking (p = 0.07) or use nicotine replacement therapy (p = 0.09) compared with standard tailoring. At 30 days, 36 percent of PERSPeCT smokers stopped smoking for one day or longer compared with 32 percent of the participants receiving the standard tailoring system (p = .70).
Limitations
The standard tailoring (comparison) system has demonstrated effectiveness on long-term smoking outcomes; only short-term quit outcomes were assessed in this study. Thus, this study is limited to surrogate outcomes (ratings of influence) that have been demonstrated in prior work to be associated with longer term cessation.
Conclusions
The PERSPeCT recommender system outperformed the standard system in terms of influence ratings but resulted in similar rates of cessation compared with standard tailoring.