Next Article in Journal
Geographic Heterogeneity in Influenza and Pneumonia Mortality in Hispanic Americans
Previous Article in Journal
Impacts of High Environmental Temperatures on Congenital Anomalies: A Systematic Review
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Investigating Patients’ Preferences to Inform Drug Development Decisions: Novel Insights from a Discrete Choice Experiment in Migraine

by
Aleksandra Torbica
1,2,
Carla Rognoni
1,* and
Rosanna Tarricone
1,2
1
CERGAS (Centre for Research on Health and Social Care Management), SDA Bocconi School of Management, Bocconi University, 20136 Milan, Italy
2
Department of Social and Political Sciences, Bocconi University, 20136 Milan, Italy
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2021, 18(9), 4916; https://doi.org/10.3390/ijerph18094916
Submission received: 2 April 2021 / Revised: 29 April 2021 / Accepted: 2 May 2021 / Published: 5 May 2021
(This article belongs to the Section Health Behavior, Chronic Disease and Health Promotion)

Abstract

:
There is limited evidence on the scope and overall benefit of patient-centred drug development decisions. The present study assessed patients’ preferences for the characteristics of an ideal migraine treatment through a discrete choice experiment in order to inform decision-making and drug development processes. We investigated the preferences according to five treatment attributes identified from a systematic literature review and two focus group elicitations. The heterogeneity of preferences was also investigated. Overall, the respondents considered the presence of adverse events, duration of treatment effect, reduction of symptom intensity, speed of effect and cost born by the patient as the most relevant treatment features. As expected, the patients preferred treatments with lower levels of adverse events and costs and treatments with greater speed, duration of treatment effect and effectiveness in reducing symptom intensity. There was significant preference heterogeneity only for the presence of adverse events. Compared to men, women had significantly higher preferences for quicker treatment effect and limited adverse events and reported higher preferences for costly treatments. The results of our survey help address research and development strategies in the pharmaceutical industry and public policy regarding treatments that are clinically effective and responsive to the needs expressed by patients.

1. Introduction

Pharmaceutical companies are continuously investing in the research and development of products that are supposed to meet the needs of patients, physicians and payers while adhering to regulatory requirements. The development of a treatment to help improve patients’ lives should be rooted in a solid understanding of the challenges these patients face in their daily activities, their needs and the compromises they are willing to make to obtain relief. To ensure the creation of valuable treatments, all aspects of the healthcare system and treatment decisions need to be aligned with the needs of patients [1]. At present, there are limited data describing the scope and overall benefit of existing patient-centred drug development activities [2], and the assessment of patients’ preferences for treatment characteristics in this setting would increase the current evidence on the impact of diseases to support decision-making processes in the field of public policies and research and development in pharmaceutical industries.
Investigating patients’ preferences is essential to inform these decisions for all diseases, but it appears to be particularly important for chronic conditions that require patient commitment to their management, and this has important long-term consequences for patients’ personal and social lives. Chronic migraine may be considered a paradigmatic example in this respect.
Available data estimate that approximately 12% of adults worldwide develop some form of migraine throughout their lives [3,4]. Migraine is generally classified into episodic migraine (4 to 14 days of migraine a month) and chronic migraine (≥15 days of migraine per month). Women are most affected by migraine. In Italy, 6 million people suffer from migraines, and 4 million of the affected are women. Considering the plurality of costs associated with the management of the disease, migraine also has a significant socio-economic weight [5]. A study conducted in Italy estimated an annual cost for the management of a patient with migraine of EUR 4352, mainly composed by productivity losses (36%) and informal care (34%) [6]. Moreover, Italian women lose more days of work (16.8 per year versus 13.6 for males) and days of social life (26.4 per year versus 20) and are more likely to work in conditions of malaise (51.6 days per year versus 35.6). The study highlighted also the burden of this disease on caregivers, showing that 63% of patients received informal care from family members, on average for 4 days a month. Migraine has also a significant impact on patient’s quality of life, especially for women in relation to the interruption or limitation of daily activities [7]. Despite these evidences, a cultural gap persists on the migraine pathology that leads the disease to be underestimated, underdiagnosed and consequently undertreated or possibly erroneously managed [6].The literature includes different studies that evaluated patient preferences for migraine treatment characteristics. A few studies focused on specific treatments such as triptans [8,9], while other studies focused on preventive treatments only [10,11]. Most studies used ad hoc surveys to investigate patient preferences, but only two implemented discrete choice experiments [11,12], which allowed the assessment of the relative importance of different attributes. The studies in general were not limited to individuals with migraine but included patients with headache and did not consider gender differences in treatment preferences.
The available evidence highlights that both episodic and chronic migraine present a high social burden and represent an “unmet need” for public health that requires particular attention. Although there is increasing evidence supporting the role of gender in epidemiology and diagnosis, the identification of gender differences in migraine treatment and related efficacy is basically ignored [13]. In the presence of a disease that primarily affects women, gaps remain in gender-specific research at preclinical and clinical levels. The same gap occurs in the drug development process, which does not generally consider patient preferences and differences based on gender [14].
The current study expands the existing literature on patient preferences for an ideal treatment for migraine to overcome the identified gaps by considering the specific population of individuals with migraine, applying a solid methodology and investigating gender differences and heterogeneity in these preferences. We employed a discrete choice experiment (DCE) methodology to investigate the preferences of a heterogeneous group of patients for different characteristics of a hypothetical but realistic therapeutic regime for migraine. The ultimate aim is to improve the existing body of knowledge and provide an evidence base to inform patient-centred policies and the research and development decisions made by pharmaceutical industries.

2. Materials and Methods

To evaluate the preferences of individuals with migraine for treatments characteristics and evaluate how socio-economic characteristics, particularly gender, may influence those preferences, a DCE was implemented [15]. The DCE was performed to identify and estimate the relative importance assigned to different characteristics of a hypothetical treatment for migraine.
DCE is a technique to elicit preferences, which is based on the fact that goods or interventions may be described according to their characteristics (or attributes), and each attribute is represented by a defined number of dimensions, called attribute levels, on which the preferences of individuals for these goods depend [16]. For example, attributes for a migraine treatment may be route of administration (e.g., oral or intravenous), duration of the effect (e.g., 6 h, 8 h, or all day), speed of relief (e.g., 30 min, 1 h, or 2 h), etc.
A DCE consists of the presentation of two or more hypothetical scenarios, created by combining attribute levels, to participants, who choose among a number of alternative options [17]. Respondents trade-off between the attributes during the decision-making process and select the preferred option. The choices of the respondents indicate the preference or utility attached to a good or intervention and its attributes [18].

2.1. Development of DCE Attributes and Associated Levels

A systematic literature research was performed in March 2018 to retrieve the available evidence on the preferences of patients for the main features of migraine treatments. This preliminary activity allowed the identification of a possible list of treatment characteristics for further investigation via a discussion with a sample of individuals with migraine (see Appendix A for details).
The search query led to the selection of 14 studies that contained information about patient preferences [8,9,10,11,12,19,20,21,22,23,24,25,26,27], which were measured on the basis of more or less structured questionnaires to understand which characteristics of a treatment they considered most important. The population considered was mostly composed of patients with migraine, without distinction based on type (chronic or episodic). The treatment characteristics considered in the various published studies were resolution of symptoms (32%), efficacy (speed of effect, no recurrences, delay in the attack and persistence of effect) (30%), adverse events (14%), return to daily activities (11%), formulation (11%) and cost (3%). Although these studies investigated both genders, no differences in preferences emerged.
After collecting the available evidence in the international literature, we organized two focus groups to evaluate the characteristics of a hypothetical treatment that the patients considered most relevant in the national context. The focus group was a group interview led by a moderator who followed a structured outline and proposed stimuli to the participants [28]. The idea behind this method is that the considerations or evaluations expressed by each of the participants elicit reactions, comments and reflections from the others, which activates a spontaneous and intense discussion that leads to the sharing of further and more in-depth considerations. Each focus group lasted two hours and involved 8 adult patients who suffered from migraine with an average number of attacks per month greater than or equal to 4 (group 1: mean age 49 years, 37% males; group 2: mean age 45 years, 50% males). The focus group discussion started with the presentation of the participants with their personal characteristics of migraine and with the sharing of treatment features studied in the literature and derived by the literature search. After interactive discussion, each participant rated the treatment features from the most important to the less important. The following five most important characteristics were considered by the participants in the two focus groups:
  • speed of effect (how quickly the treatment relieves the symptoms);
  • reduction in the intensity of symptoms (efficacy-strength);
  • duration of effect (efficacy-duration);
  • adverse events;
  • cost born by the patient.
Concerning the quantification of the levels of variation of these attributes, the participants reported that the effectiveness, in terms of speed in achieving the effect, should ideally be within 30 min of drug administration. Times from 30 to 90 min were acceptable, while times over 3 h were considered excessive. Regarding the efficacy of the treatment in terms of intensity of symptom reduction, acceptable levels were approximately 50%, with ideal values of symptom reduction of at least 90%. Patients believed that the effectiveness of the treatment should ideally last all day, or at least for 8 h, considering the workday. Furthermore, patients specified the possible side effects (i.e., tachycardia, drowsiness, daze, tingling, and gastrointestinal effects) and the preference for treatments with limited adverse events. Regarding the cost born by patients to purchase an ideal treatment, patients found a cost of EUR 50 per month acceptable, but they would be willing to bear a cost of up to EUR 200 per month to have resolution of all symptoms related to migraine. Table 1 reports the summary of attributes and levels discussed during the focus groups. Adverse events severity was classified according to FDA Adverse Event Reporting System (FAERS) [29].

2.2. Experimental Design

After the elicitation of attributes and associated levels, a DCE with three choice alternatives (i.e., two alternative choices and an opt-out option) was developed. Inclusion of an opt-out was deemed necessary because forcing respondents to make a choice on a treatment can lead to an overestimation of the utility for parameters [30]. The opt-out option, corresponding to “no treatment”, was associated with zero speed of effect, zero percent of symptoms reduction, no duration of effect, no presence of adverse events and no cost born by the patient.
The choice sets were combined using Ngene software [31]. From a full design (factorial), an optimal design (orthogonal in the differences—OOD) [32] was derived by considering the total number of choices between alternatives equal to 100 (D-optimality = 92.5%). The choice sets were grouped into five blocks. Each block comprised 20 choice sets, ordered differently across blocks. A sample size of minimum of 250 respondents was identified for parameter estimations for the DCE, and the blocks were distributed randomly among them [33].

2.3. Data Collection

A questionnaire representing the choice set was administered online in October–November 2018 to a sample of 466 adult patients suffering from migraine with an average number of attacks per month greater than or equal to 4 (see Appendix B). The web survey was implemented by an external company (Pepe Research https://www.peperesearch.it (accessed on 2 April 2021)), which selected respondents from its own database according to the characteristics described above. The database contains individuals that self-reported to be affected by one or more diseases. Individuals classified as migraineurs have been selected for the study. The first two questions of the survey served as a confirmation that the respondents were effectively affected by migraine.
The survey was pretested online on a sample of 27 respondents, who had the chance to report possible comprehension problems and difficulties in the questionnaire completion. This phase did not identify possible comprehension problems or difficulties in executing the tasks.
The following data categories were collected:
  • Socio-demographic characteristics;
  • Education level, professional status, and net annual income;
  • Duration of the single migraine attack;
  • Type of migraine (with or without aura);
  • Average number of attacks per month.
The e-survey presented an explanation of the scope of the interview, a description of the different levels and attributes and 20 choices among the three alternatives. Each attribute was carefully explained using patient-friendly language. In order to simulate a clinically relevant decision context, we clarified in the questionnaire that patients had to choose only on the basis of the available given information and we explicitly specified an opt-out condition in line with good practice recommendations.
The participants’ task was to evaluate two alternatives (A and B) and the opt-out option (alternative C, no treatment) in each choice set and choose the option that reached the preference from their point of view. An example of a scenario with the choice of three alternatives is reported in Figure 1.
At the end of the survey, 5 questions investigated the importance of the considered attributes (see Table 1) through a Likert scale with 5 levels (not at all important, not very important, quite important, very important and extremely important). One question investigated the difficulties experienced during the choices among the proposed alternatives, and the possible responses were no difficulties, few difficulties, moderate difficulties, many difficulties and extreme difficulties.

2.4. Data Analysis

Respondent characteristics and responses were summarized using descriptive statistics. The DCE choices were first reviewed to determine whether respondents showed dominant preferences (i.e., frequently selected the set with the best level of a particular attribute) and in which cases the respondents chose the “opt-out” option. For each respondent, dominance scores were calculated for each attribute; a score of 1 was assigned for each question in case the respondent chose an attribute at its best level (e.g., maximum power of the effect or minimum cost). Therefore, a maximum score equal to 20 for an attribute was assigned to a respondent who chose the best level of this attribute for all 20 questions.
The DCE data were analysed within a random utility maximization framework [34,35], and a mixed logit regression analysis was used to construct a model of choice behaviour. In this framework standard deviations are generated to quantify preference heterogeneity for attributes and attribute levels [36].
The following model of choice behaviour (Model 1) was used to evaluate the impact of different attributes and levels on the preferences for alternative treatment approaches [37]:
Uijs = β1 (speed) + β2 (efficacy) + β3 (duration) + β4 (adverse events) + β5 (cost) + εijs
where Uij is the utility individual i derives from choosing alternative j in choice situation s, β1–β5 are the coefficients of preference weights reflecting the desirability of the attributes and εijs is the error term assumed independent and identically distributed (random component).
The output of a mixed logit model includes the means and standard deviations (SDs) of random coefficients and their respective confidence intervals (CIs). The mean coefficients represent the relative utility of each attribute conditional on other attributes, and the SDs reflect the degree of heterogeneity among the respondents (higher the value, higher the heterogeneity).
The inclusion of the cost born by the patient among the attributes allows the estimation of the willingness to pay (WTP) [38]. WTP is interpreted as an estimate of the relative values assigned to an attribute included in the choice set, and it is expressed in monetary terms. A WTP analysis was performed starting from the results of the regression analyses for Model 1 and males and females separately. We calculated the WTP for nonmonetary attributes as the ratio of the cost coefficient and mean coefficients for the attributes.
We extended the baseline model to examine the drivers of response heterogeneity. More specifically, to investigate the extent to which preferences were driven by respondents’ characteristics, Model 1 was extended to Model 2, which considered interactions between attributes and patients’ gender and age. For this model, continuous (i.e., age) and categorical (gender) interaction variables were considered.
The random parameters for all attribute levels were estimated assuming a normal distribution. The Akaike information criterion (AIC) was applied to compare the goodness of fit and to test the extended model against the model with no interactions.
Data analyses were undertaken using Stata (StataCorp. 2019. Stata Statistical Software: Release 15. College Station, TX: StataCorp LP).

3. Results

All of the 466 participants who enrolled completed the e-survey. The participants had an average age of 43 years (range 18–77), and 66% were women. A statistically significant difference emerged for age between the two genders, with males being older than females (45 years vs. 41 years). Working respondents reported that they had a full-time job in most cases (72%). The annual income ranges were equally distributed over the total number of respondents. Only 7% of respondents preferred to not declare their income. Statistically significant differences emerged between the genders for declared income, with lower incomes for females, who also reported less remunerative working roles.
Considering the characteristics of migraine, females showed a greater and significant duration of attack than males; a duration of at least 4 h was reported by 70% of females (= 45% duration 4–24 h + 25% duration 2–3 days) and by 49% of males (=41% duration 4–24 h + 8% duration 2–3 days). Aura affected more females than males (45% vs. 35%).
The socio-demographic characteristics of the sample, including disease characteristics, are summarized in Table 2.
Regarding DCE, the participants reported 9320 responses (20 questions each), with a significantly high number of cases (38%) in which respondents chose the opt-out option. In addition, 25% of respondents reported having encountered moderate-to-extreme difficulty in choosing among the alternatives.
The analysis of dominant preferences revealed that only eight respondents expressed dominant preferences (score = 20) for the presence of adverse events, seven for speed of effect and seven for cost. The dominant preferences of other attributes were limited. The overall pattern of results suggests that the presence of side effects, speed of effect and the cost born by the patients were the most important factors for respondents in deciding which treatment they would select.
The full results of mixed logit models are presented in Table 3.
In Model 1, all the attributes significantly impacted the probability of choosing an alternative (p-values < 0.05). The negative sign of the coefficient for speed of effect (−0.00328) indicates that as the time to obtain a treatment effect increased, the patients’ likelihood of choosing this scenario decreased. The same results were found for adverse events and cost. In contrast, respondents preferred higher levels for strength of efficacy and duration of effect. Only the presence of adverse events showed a significant preference heterogeneity (standard deviation p-value = 0.003). These results are consistent with the indications received during the focus groups.
In Model 2, all the attributes (β1–β5) significantly impacted the probability of choosing an alternative. The presence of adverse events maintained heterogeneity in the preferences (standard deviation p-value = 0.007).
Gender interactions were statistically significant for speed of effect, presence of adverse events and cost. The results revealed that, compared to men, women had significantly higher preferences for quicker treatment effect and the presence of limited adverse events, and they reported higher preferences for more costly treatments. Age did not seem to influence the preferences of patients.
According to the Akaike information criterion, Model 2 showed a better fit than Model 1.
The WTP analysis performed on the base-case mixed logit model (Model 1) showed that respondents would be willing to pay EUR 0.61 to anticipate the effect of the treatment by one minute, everything else being equal. Patients would be willing to pay EUR 4.52, €16.66 and €141.68 to acquire a one percentage point increase in the strength of symptom reduction, for having an additional hour of effect duration and for de-escalating the adverse events, respectively, with everything else being equal. These results show that the reduction of adverse events is the most important dimension for which patients would be willing to pay the highest amount.
The WTP analysis performed separately on males and females showed a higher willingness to pay for women compared with men for all of the considered attributes (EUR 0.98 vs. EUR 0.18 to anticipate the effect of the treatment by one minute, EUR 5.25 vs. EUR 3.72 to gain a one percentage point increase in the strength of symptom reduction, EUR 19.60 vs. EUR 12.62 for having an additional hour of treatment effect duration and EUR 172.69 vs. EUR 97.10 for de-escalating the adverse events).
The responses on the Likert scale were consistent with the results obtained in the DCE, and they confirmed the presence of adverse events as the most important treatment feature. The following other attributes, in order of importance, were efficacy-strength, speed of effect, duration of the effect and cost born by the patient (see Table 4).

4. Discussion

For many patients suffering from migraine, finding the right combination of clinical treatment and routine is a lifelong challenge. Where different treatment strategies are available, it is of the utmost importance to support the treatment choices of patients to improve their compliance with therapies.
The present study employed a DCE to investigate gender differences in preferences for a set of attributes of treatment for migraine. The key feature of the study model was the use of a mixed logit model that allowed us to account for heterogeneity in preferences driven by observable characteristics (primarily gender) and test for the existence of residual significant heterogeneity in nonobservable characteristics.
The presence of adverse events, duration of the treatment effect, reduction of the intensity of the symptoms, speed of the effect and cost born by the patient were, in that order, the attributes considered most relevant by the respondents. These data are consistent with our international literature review, which reported these five attributes as the most important in 79% of studies. These results are consistent with another DCE involving 510 patients with migraine in the USA [12], which investigated the severity and duration of symptoms in headache and postheadache phases, the limitation in activity and the chance of migraine attack recurrence. The study showed that hypothetical treatments that relieved and shortened symptoms during the postheadache phase offered significant benefits to individuals with migraine. The same results were found in another DCE study [11] that focused on migraine prevention, in which 72% of respondents rated treatment effectiveness as the most important aspect. The present study revealed that all the attributes significantly impacted the probability of choosing an alternative and that, in general, respondents preferred lower levels for speed of effect (quick response), adverse events and cost and higher levels for strength of efficacy and duration of effect.
Only the presence of adverse events was associated with significant heterogeneity with respondents’ preferences. The interaction analysis also highlighted that compared to men, women had significantly higher preferences for quick treatment effect and the presence of limited adverse events and that women reported higher preferences for costly treatments. Women, who in general report a worse quality of life and worse symptoms than men, seemed willing to pay more than males to receive a more effective treatment. In fact, 45% of female migraine patients have aura (a set of sensory, motor or verbal disturbances) compared with 35% of male migraine patients, with a statistically significant difference between the two groups. Based on this, it is reasonable for women to be willing to pay more for migraine treatment, irrespective of preference for treatment methods, in order to alleviate these heavy symptoms. Age did not influence patients’ preferences.
As direct beneficiaries of health services, patients have a widespread awareness of the impact and the effects of a treatment on their condition and on different aspects of their life. The incorporation of patient preferences in the drug development process may be of paramount value. Different initiatives have started to integrate the patient’s voice into therapeutic development and regulatory review. An example is the Patient-Focused Drug Development (PFDD) initiative [39], which includes meetings that aim to help the Food and Drug Administration (FDA) understand the burden of disease from the patient perspective and to gain an appreciation for the factors that are taken into account by patients when a treatment is chosen. Similar developments started also in Europe, with the incorporation of patient preferences into the assessment of oncology treatments by the European Medicines Agency (EMA) [40]. Again, the European Patients Academy on Therapeutic Innovation (EUPATI) [41] focused on education and training to increase the capacity and capability of patients to be valuable contributors to medicines research and development. The PREFER initiative, built upon the experiences and outcomes of previous projects and initiatives, aims at establishing recommendations to support development of guidelines for industry, regulatory authorities and HTA bodies on how and when to include patient perspectives on benefits and risks of medicinal products [42].
This study provides novel insight into this growing body of literature, but some limitations need to be recognized. First, involved patients were recruited via an online survey, and this population may be biased towards the ability to use a computer or a mobile device; moreover, only respondents with internet access were able to complete the questionnaire [43]. Another limitation relates to the selection of migraineurs; although they were selected from a pre-existing database managed by a market research company (Pepe Research), no official documentation on migraine diagnosis was requested in order to include an individual in the database and respondents may either have identified themselves as being affected by migraine or may have received an official diagnosis by a physician. This could have increased the heterogeneity in the sample analysed.
Third, in the range of levels presented, a significant portion of the sample did not appear to use a trade-off between the attributes and chose the opt-out option in 38% of cases. Fourth, as a cross-sectional hypothetical experiment among respondents, the elicited preferences described in this study may change over time, especially once respondents experience different treatment strategies for migraine. Therefore, while the results are internally valid, generalizability beyond the study context cannot be explicitly guaranteed [44]. However, these concerns are not peculiar to our own study but represent general concerns pertaining to the application of stated preferences techniques [44].

5. Conclusions

To date, there is no resolutive treatment for migraine. Nausea, visual disturbances and hypersensitivity to sounds, smells and light make the disorder more complex, and patients may not even be able to get out of bed on the worst days. Although the range of pharmacological opportunities in use today is being enriched, the involvement of patients in drug development activities and in the collection of their preferences seems not routinely used.
Our study showed that patients prefer treatments with lower levels of adverse events and costs and treatments with greater speed, duration of treatment effect and effectiveness in reducing symptom intensity. Female migraine patients are more inclined to choose faster treatments, fewer side effects, but also more expensive treatment options. The results of our survey can help address the research and development strategies of the pharmaceutical industry towards treatments that are clinically effective and responsive to the needs expressed by the patients.

Author Contributions

Conceptualization, R.T. and A.T.; methodology, R.T. and A.T.; software, C.R. and A.T.; validation, A.T.; formal analysis, C.R. and A.T.; investigation, C.R. and A.T.; data curation, C.R.; writing—original draft preparation, C.R.; writing—review and editing, A.T., R.T. and C.R.; visualization, C.R.; supervision, A.T.; project administration, R.T.; funding acquisition, R.T. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by Novartis Farma S.P.A. The APC was funded by Bocconi University.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of Bocconi University (ref. n. 0068538, 8 May 2018) (protocol code 0068538, 8 May 2018).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study. Written informed consent has been obtained from the patient(s) to publish this paper.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. Only aggregated data can be shared due to ethical restrictions.

Conflicts of Interest

The authors declare no conflict of interest. The funder had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Appendix A

Appendix A.1. Search Process

Appendix A.1.1. Search Query

Migraine AND (“patient* preference*” OR “discrete choice experiment*” OR “willingness to pay” OR “stated preference*” OR “conjoint analysis”).
Figure A1. PRISMA flow diagram.
Figure A1. PRISMA flow diagram.
Ijerph 18 04916 g0a1
Table A1. Characteristics of the retrieved studies.
Table A1. Characteristics of the retrieved studies.
StudyFocus of StudyType MigraineList of Attributes Considered
Dahlöf, 2001 [19]ReviewMigraineRelief of migraine pain, fastest relief of pain, quickest return to normal function, few or no side effects, longer duration of effect, lower cost.
Dowson, 2005 [8]Oral triptansMigraineEffective pain relief, restored ability to function, requirement for fewer doses, relief of migraine-associated symptoms, rapid onset of efficacy, no tired feelings, fewer side effects.
Gonzalez, 2012 [9]TriptansMigrainePain and sensitivity to light or sound 1 h after taking the medicine.
Pain and sensitivity to light or sound 2 h after taking the medicine.
Nausea and/or vomiting (24 h after taking the medicine).
Chance that the migraine returns within 24 h.
Nervous system side effects (24 h after taking the medicine).
Chest-related side effects (5 min after taking the medicine).
Limitations on ability to do daily activities (24 h after taking the medicine).
Chance of a heart attack within the next year because of the medicine.
Gonzalez, 2013 [12]DCE, headache and post-headache phase, migraine type not specifiedMigraineHeadache phase: Migraine headache phase symptoms.
Postheadache phase: limitations on ability to work and participate in social activities.
Symptom-free phase: chance headache returns within 24 h.
Hamelsky, 2005 [20]WTP, migraine type not specifiedMigraineSpeed of relief, consistency of relief, side-effects, recurrence.
Katsarava, 2011 [21]WTP bidding game, migraine type not specifiedCM or EMEffective headache treatment, headache type, frequency, MIDAS score, SF-36 score.
Leinisch-Dahlke, 2004 [22]Questionnaire, expectancies of an ideal drug for acute therapyIdiopathic relapsing or chronic headache disorderSpeed of onset, pain free, side effects, restoring working ability, pain relief, maintaining working ability, efficacy against concomitant symptoms.
Lenert, 2003 [23]WTP bidding game, migraine type not specifiedMigraineReductions for different attributes:
50% chance of rebound headache;
unable to work afterward;
2 h delay in onset;
incomplete relief of pain;
no relief of photophobia;
no relief of nausea.
Lipton, 2002 [24]WTP survey, migraine type not specifiedMigraine(acute migraine therapy) complete relief of head pain, no recurrence, rapid onset of action.
Manandhar, 2016 [25]HeadacheHeadache disorders, including migraineEffective headache care.
Matías-Guiu, 2012 [26]Kano, migraine type not specifiedMigrainePain relief achievement, fast pain relief achievement, complete pain disappearance, complete pain relief during 24 h, symptom relief achievement, complete symptom disappearance, fast symptom relief, lasting symptom relief achievement, possibility of resuming social and family activities, possibility of resuming occupational or academic activities, not an injection, it dissolves in mouth and does not require water administration, no discomfort upon administration, no long term adverse effects, media advertisement, it is a new drug.
Mitsikostas, 2017 [10]different types of migraine (groups)CM or EMSafety, efficacy, route of administration.
Peres, 2007 [11]DCE, Preventive treatmentCM or EMEfficacy, speed of onset, out-of-pocket expenses, adverse events, formulation of therapy, type of treatment, and frequency of dosing.
Smelt, 2014 [27]Delphi, migraine type not specifiedMigraineTake away the headache, prevent the attack from carrying through, make sure no other attack follows within a few hours or within a day, let me function properly again, clear my head, take away the pressing or thumping feeling, take away the nausea, take away the problems with vision (light flashes, hazy vision, double vision), take away the sense of illness during a headache attack, take away the neck pain, take away the tiredness, take way the loss of function (problems with speech, tingling or loss of power in arms/legs), take away the persistent headache after the headache attack.
CM = chronic migraine; EM = episodic migraine; DCE = discrete choice experiment; WTP = willingness to pay.

Appendix B

Appendix B.1. E-Survey

(Age, gender and educational level were already available for the responders). Several diseases are listed below. Please kindly indicate whether you are affected by one or more of these diseases (check all that apply), or none.
-
Celiac Disease
-
Diabetes
-
Dermatitis
-
Migraine (if yes, the questionnaire continues, if not it ends)
-
Hypertension
-
Low back pain
-
Osteoporosis
-
Ulcer
-
Vitiligo
-
None of the above (screen out)
(If the respondent is affected by migraine). You indicated that you suffer from migraine. Migraine is a recurrent disease characterized by predominantly unilateral (but can also be bilateral) headache, of moderate or severe intensity, described as pulsating with a duration of 4 to 72 h, which tends to worsen with movement and physical effort and is usually associated with nausea, visual disturbances, sensitivity to light and/or sound.
1. Can you confirm that you suffer from migraines?
-
Yes
-
No
(If yes) The aim of the present questionnaire is to investigate the social and economic burden of migraine, evaluating the differences between male and female and the preferences of people with migraine towards the characteristics of a possible treatment.
2. How long does a single migraine attack typically last?
-
Few minutes
-
Up to 3 h
-
From 4 to 24 h
-
2–3 days
3. What type of migraine do you suffer from? (1 answer only)
-
Migraine, without aura
-
Migraine, with aura
4. How many migraine attacks do you have on average in a month? ……… attacks
5. What is your current employment status? (1 answer only)
Employee:
-
blue-collar worker
-
white-collar worker
-
executive, manager
Self-employed worker, freelance:
-
entrepreneur, freelancer
-
other autonomous
Not employed:
-
retired
-
student
-
homemaker
-
other, unemployed
6. (If the respondent is employed) What is your work schedule? (1 answer only)
-
Full time
-
Part time: 20 h/week
-
Part time: 24 h/week
-
Part time: 30 h/week
-
Part time: 36 h/week
7. (If the respondent is employed) What is your net annual income? (1 answer only—OPTIONAL)
-
less than €15,000
-
€15,000–19,999
-
€20,000–29,999
-
€30,000 or more
The second part of the questionnaire, consisting of 20 questions, analyses the preferences of people suffering from migraine towards the characteristics of a possible treatment. Some possible treatments will be presented below, with different characteristics. The characteristics concern the following:
-
The speed of the effect, i.e., within how long the symptoms are reduced (e.g., 30 min, 180 min)
-
The power of the effect, i.e., in what percentage the overall symptoms are reduced (e.g., 60%, 90%). Please refer to the migraine specific symptoms like pain, nausea, vomiting, visual disturbances, hypersensitivity to lights and sounds, etc.
-
Duration of the effect, i.e., how long the relief from symptoms lasts (e.g., for 4 h, for 10 h)
-
Any side effects (i.e., tachycardia, drowsiness, daze, tingling, gastrointestinal effects, etc.). Adverse events are classified in mild (no antidote or treatment is required), moderate (a change in treatment like dosage modification or addition of a drug may be performed, but not necessarily discontinuation of the drug is required; a specific treatment may be required), severe (the event is potentially life threatening and requires discontinuation of the drug and specific treatment), very severe (the event may directly or indirectly contribute to the patient’s death).
-
The cost of the therapy.
For each proposed therapy, please read the characteristics and choose the alternative you prefer, on the basis of the available given information, between alternative A, alternative B or, in case of indifference, the third alternative proposed.
Finally, a couple of questions will be presented to directly evaluate the importance of the characteristics of a treatment and the complexity of the questionnaire.
Questions from 1 to 20 are similar to the following (with different combinations of characteristics values):
1–20. Which one would you choose from the following options?
Treatment CharacteristicsAlternative AAlternative BAlternative C
(No Treatment)
Speed of effect (minutes)30800
Efficacy-strength (% of symptoms reduction)90600
Duration of the effect (hours)460
Presence of adverse events very severemildno adverse events
Monthly cost born by the patient (€)1001500
21. How many difficulties did you experiment in the choice among the proposed alternatives? (1 answer only)
-
no difficulties
-
few difficulties
-
moderate difficulties
-
many difficulties
-
extreme difficulties
22. What importance do you attach to the treatment characteristic “Speed of effect”? (1 answer only)
-
not at all important
-
not very important
-
quite important
-
very important
-
extremely important
23. What importance do you attach to the treatment characteristic “Efficacy-strength (% of symptoms reduction)”? (1 answer only)
-
not at all important
-
not very important
-
quite important
-
very important
-
extremely important
24. What importance do you attach to the treatment characteristic “Duration of the effect”? (1 answer only)
-
not at all important
-
not very important
-
quite important
-
very important
-
extremely important
25. What importance do you attach to the treatment characteristic “Presence of adverse events”? (1 answer only)
-
not at all important
-
not very important
-
quite important
-
very important
-
extremely important
26. What importance do you attach to the treatment characteristic “Monthly cost born by the patient”? (1 answer only)
-
not at all important
-
not very important
-
quite important
-
very important
-
extremely important

References

  1. Lowe, M.M.; Blaser, D.A.; Cone, L.; Arcona, S.; Ko, J.; Sasane, R.; Wicks, P. Increasing Patient Involvement in Drug Development. Value Health 2016, 19, 869–878. [Google Scholar] [CrossRef] [Green Version]
  2. Getz, K.A. Establishing Return-on-Investment Expectations for Patient-Centric Initiatives. Ther. Innov. Regul. Sci. 2015, 49, 745–749. [Google Scholar] [CrossRef]
  3. Steiner, T.J.; Stovner, L.J.; Katsarava, Z.; Lainez, J.M.; Lampl, C.; Lantéri-Minet, M.; Rastenyte, D.; Ruiz de la Torre, E.; Tassorelli, C.; Barré, J.; et al. The Impact of Headache in Europe: Principal Results of the Eurolight Project. J. Headache Pain 2014, 15, 31. [Google Scholar] [CrossRef] [Green Version]
  4. GBD 2015 Disease and Injury Incidence and Prevalence Collaborators. Global, Regional, and National Incidence, Prevalence, and Years Lived with Disability for 310 Diseases and Injuries, 1990-2015: A Systematic Analysis for the Global Burden of Disease Study 2015. Lancet 2016, 388, 1545–1602. [Google Scholar] [CrossRef] [Green Version]
  5. Tarricone, R. Cost-of-Illness Analysis. What Room in Health Economics? Health Policy 2006, 77, 51–63. [Google Scholar] [CrossRef] [PubMed]
  6. Emicrania: Impatto Sociale ed Economico in Italia [Migraine: Social and Economic Impact in Italy]; Springer Healthcare: Milan, Italy, 2021.
  7. Nica, M.; Colombo, D.; Ravasio, R.; Rognoni, C. Socioeconomic Impact of Migraine in Italy: Results of a National Survey. AboutOpen 2020, 7, 80–88. [Google Scholar] [CrossRef]
  8. Dowson, A.J.; Tepper, S.J.; Dahlöf, C. Patients’ Preference for Triptans and Other Medications as a Tool for Assessing the Efficacy of Acute Treatments for Migraine. J. Headache Pain 2005, 6, 112–120. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  9. Gonzalez, J.M.; Hauber, A.B.; Levitan, B.; Coplan, P. PND36 Structured Benefit-Risk Assessment of Triptan Treatments Using Patient-Preference Data. Value Health 2012, 15, A147. [Google Scholar] [CrossRef] [Green Version]
  10. Mitsikostas, D.D.; Belesioti, I.; Arvaniti, C.; Mitropoulou, E.; Deligianni, C.; Kasioti, E.; Constantinidis, T.; Dermitzakis, M.; Vikelis, M. Patients’ Preferences for Headache Acute and Preventive Treatment. J. Headache Pain 2017, 18. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  11. Peres, M.F.P.; Silberstein, S.; Moreira, F.; Corchs, F.; Vieira, D.S.; Abraham, N.; Gebeline-Myers, C. Patients’ Preference for Migraine Preventive Therapy. Headache 2007, 47, 540–545. [Google Scholar] [CrossRef]
  12. Gonzalez, J.M.; Johnson, F.R.; Runken, M.C.; Poulos, C.M. Evaluating Migraineurs’ Preferences for Migraine Treatment Outcomes Using a Choice Experiment. Headache 2013, 53, 1635–1650. [Google Scholar] [CrossRef] [PubMed]
  13. Schroeder, R.A.; Brandes, J.; Buse, D.C.; Calhoun, A.; Eikermann-Haerter, K.; Golden, K.; Halker, R.; Kempner, J.; Maleki, N.; Moriarty, M.; et al. Sex and Gender Differences in Migraine-Evaluating Knowledge Gaps. J. Women’s Health 2018, 27, 965–973. [Google Scholar] [CrossRef] [PubMed]
  14. Cook, N.S.; Cave, J.; Holtorf, A.-P. Patient Preference Studies During Early Drug Development: Aligning Stakeholders to Ensure Development Plans Meet Patient Needs. Front. Med. 2019, 6, 82. [Google Scholar] [CrossRef] [PubMed]
  15. Reed Johnson, F.; Lancsar, E.; Marshall, D.; Kilambi, V.; Mühlbacher, A.; Regier, D.A.; Bresnahan, B.W.; Kanninen, B.; Bridges, J.F.P. Constructing Experimental Designs for Discrete-Choice Experiments: Report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force. Value Health 2013, 16, 3–13. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  16. Louviere, J.J.; Pihlens, D.; Carson, R. Design of Discrete Choice Experiments: A Discussion of Issues That Matter in Future Applied Research. J. Choice Model. 2011, 4, 1–8. [Google Scholar] [CrossRef] [Green Version]
  17. de Bekker-Grob, E.W.; Ryan, M.; Gerard, K. Discrete Choice Experiments in Health Economics: A Review of the Literature. Health Econ. 2012, 21, 145–172. [Google Scholar] [CrossRef]
  18. Lancsar, E.; Louviere, J. Conducting Discrete Choice Experiments to Inform Healthcare Decision Making: A User’s Guide. Pharmacoeconomics 2008, 26, 661–677. [Google Scholar] [CrossRef]
  19. Dahlöf, C. Assessing Patient Preference in Migraine Treatment. Cephalalgia 2001, 21, 791–795. [Google Scholar] [CrossRef] [Green Version]
  20. Hamelsky, S.W.; Lipton, R.B.; Stewart, W.F. An Assessment of the Burden of Migraine Using the Willingness to Pay Model. Cephalalgia 2005, 25, 87–100. [Google Scholar] [CrossRef]
  21. 15th Congress of the International Headache Society 23-26 June 2011, Berlin, Germany. Cephalalgia 2011, 31, 1–216. [CrossRef]
  22. Leinisch-Dahlke, E.; Akova-Öztürk, E.; Bertheau, U.; Isberner, I.; Evers, S.; May, A. Patient Preference in Clinical Trials for Headache Medication: The Patient’s View. Cephalalgia 2004, 24, 347–355. [Google Scholar] [CrossRef] [PubMed]
  23. Lenert, L.A. Use of Willingness to Pay to Study Values for Pharmacotherapies for Migraine Headache. Med. Care 2003, 41, 299–308. [Google Scholar] [CrossRef] [PubMed]
  24. Lipton, R.B.; Hamelsky, S.W.; Dayno, J.M. What Do Patients with Migraine Want from Acute Migraine Treatment? Headache 2002, 42 (Suppl. 1), 3–9. [Google Scholar] [CrossRef] [PubMed]
  25. Manandhar, K.; Risal, A.; Linde, M.; Steiner, T.J. The Burden of Headache Disorders in Nepal: Estimates from a Population-Based Survey. J. Headache Pain 2016, 17, 3. [Google Scholar] [CrossRef]
  26. Matías-Guiu, J.; Caloto, M.T.; Nocea, G. Comparison of Expected Outcomes between Patients and Neurologists Using Kano’s Methodology in Symptomatic Migraine Treatment. Patient 2012, 5, 147–162. [Google Scholar] [CrossRef]
  27. Smelt, A.F.H.; Louter, M.A.; Kies, D.A.; Blom, J.W.; Terwindt, G.M.; van der Heijden, G.J.M.G.; Gucht, V.D.; Ferrari, M.D.; Assendelft, W.J.J. What Do Patients Consider to Be the Most Important Outcomes for Effectiveness Studies on Migraine Treatment? Results of a Delphi Study. PLoS ONE 2014, 9, e98933. [Google Scholar] [CrossRef]
  28. Focus Groups: A Practical Guide for Applied Research/Richard A. Krueger & Mary Anne Casey—Trove. Available online: https://trove.nla.gov.au/work/11230018 (accessed on 2 April 2021).
  29. Center for Drug Evaluation and Research. FDA Adverse Event Reporting System (FAERS) Public Dashboard; FDA: Silver Spring, MD, USA, 2019.
  30. Campbell, D.; Erdem, S. Including Opt-Out Options in Discrete Choice Experiments: Issues to Consider. Patient 2019, 12, 1–14. [Google Scholar] [CrossRef] [Green Version]
  31. ChoiceMetrics. Available online: http://www.choice-metrics.com/ (accessed on 2 April 2021).
  32. Ishizaka, A. The Construction of Optimal Stated Choice Experiments, Theory and Methods, (1st Ed., 312 Pp. Hardback). By D. J. Street, & L. Burgess Hoboken: Wiley (2007). ISBN: 978-0-470-05332-4. J. Behav. Decis. Mak. 2009, 22, 362–363. [Google Scholar] [CrossRef]
  33. de Bekker-Grob, E.W.; Donkers, B.; Jonker, M.F.; Stolk, E.A. Sample Size Requirements for Discrete-Choice Experiments in Healthcare: A Practical Guide. Patient 2015, 8, 373–384. [Google Scholar] [CrossRef] [Green Version]
  34. Lancaster, K.J. A New Approach to Consumer Theory. J. Political Econ. 1966, 74, 132–157. [Google Scholar] [CrossRef]
  35. Hall, J.; Viney, R.; Haas, M.; Louviere, J. Using Stated Preference Discrete Choice Modeling to Evaluate Health Care Programs. J. Bus. Res. 2004, 57, 1026–1032. [Google Scholar] [CrossRef]
  36. Hole, A.R. Modelling Heterogeneity in Patients’ Preferences for the Attributes of a General Practitioner Appointment. J. Health Econ. 2008, 27, 1078–1094. [Google Scholar] [CrossRef] [Green Version]
  37. Hanemann, W.M. Welfare Evaluations in Contingent Valuation Experiments with Discrete Responses. Am. J. Agric. Econ. 1984, 66, 332–341. [Google Scholar] [CrossRef]
  38. Lancsar, E.; Fiebig, D.G.; Hole, A.R. Discrete Choice Experiments: A Guide to Model Specification, Estimation and Software. Pharmacoeconomics 2017, 35, 697–716. [Google Scholar] [CrossRef] [Green Version]
  39. Center for Drug Evaluation and Research. CDER Patient-Focused Drug Development; FDA: Silver Spring, MD, USA, 2020.
  40. Postmus, D.; Mavris, M.; Hillege, H.L.; Salmonson, T.; Ryll, B.; Plate, A.; Moulon, I.; Eichler, H.-G.; Bere, N.; Pignatti, F. Incorporating Patient Preferences into Drug Development and Regulatory Decision Making: Results from a Quantitative Pilot Study with Cancer Patients, Carers, and Regulators. Clin. Pharmacol. Ther. 2016, 99, 548–554. [Google Scholar] [CrossRef] [PubMed]
  41. What Is EUPATI? Available online: https://www.eupati.eu (accessed on 2 April 2021).
  42. Holm, A. About PREFER—PREFER. Available online: https://www.imi-prefer.eu/about/ (accessed on 2 April 2021).
  43. Bethlehem, J. Selection Bias in Web Surveys. Int. Stat. Rev. 2010, 78, 161–188. [Google Scholar] [CrossRef]
  44. Louviere, J.J.; Hensher, D.A.; Swait, J.D.; Adamowicz, W. Stated Choice Methods: Analysis and Applications, 1st ed.; Cambridge University Press: Cambridge, UK, 2000. [Google Scholar]
Figure 1. Example of choice set.
Figure 1. Example of choice set.
Ijerph 18 04916 g001
Table 1. Description of attributes and levels.
Table 1. Description of attributes and levels.
Attribute/LevelsLevel 1Level 2Level 3Level 4
Speed of effect (minutes)3080130180
Efficacy-strength60%70%80%90%
Efficacy-duration (hours)46810
Adverse eventsmildmoderateseverevery severe
Monthly cost born by the patientEUR 50EUR 100EUR 150EUR 200
Table 2. Socio-demographic characteristics of the sample of respondents with the characteristics of migraine.
Table 2. Socio-demographic characteristics of the sample of respondents with the characteristics of migraine.
ParameterTotal PopulationMalesFemalesp-Value
Gender466159 (34%)307 (66%)
Mean age (years)43 (18–77)4541<0.0001
Education level
Primary school 8%6%9%0.62
High school diploma52%54%51%
Bachelor’s degree14%12%15%
Master’s degree21%22%20%
Doctorate6%7%5%
Professional activity
White collar62.00%78.60%53.50%<0.0001
Blue collar8.80%12.60%6.80%
Retiree2.60%2.50%2.60%
Homemaker13.50%1.90%19.50%
Student5.40%0.60%7.80%
Unemployed7.70%3.80%9.80%
Income ranges declared by the workers (annual net)
Less than EUR 15,00021%10%29%<0.0001
EUR 15,000–19,99921%14%26%
EUR 20,000–29,99930%36%26%
EUR 30,000 or more28%40%19%
Duration of migraine attack, frequency and symptoms
Few minutes2%4%2%<0.0001
Up to 3 h35%47%28%
From 4 to 24 h43%41%45%
2–3 days20%8%25%
Average number of attacks per month7.27.57.00.85
Number of attacks per month from 4 to 876%72%78%0.339
Number of attacks per month from 9 to 1519%23%18%
Number of attacks per month higher than 155%6%5%
Presence of aura42%35%45%0.037
Table 3. Results of the mixed logit Models 1 and 2.
Table 3. Results of the mixed logit Models 1 and 2.
AttributesModel 1Model 2
Mean Coefficient ValuesStandard
Deviations
Mean Coefficient ValuesStandard
Deviations
βp-Valueβp-Valueβp-Valueβp-Value
Speed of effect (minutes)−0.0033<0.000010.00050.8500−0.00180.18300.00170.4440
Efficacy-strength (% of symptoms reduction)0.0242<0.000010.00020.90300.0244<0.000010.00020.9040
Duration of the effect (hours)0.0891<0.00001−0.00080.96800.09030.0070−0.00030.9880
Presence of adverse events−0.7580<0.00001−0.20120.0030−0.4484<0.00001−0.2404<0.00001
Monthly cost born by the patient (€)−0.0054<0.000010.00090.5220−0.00460.00100.00200.1670
Female*speed of effect −0.0037<0.00001
Female*efficacy-strength 0.00370.0670
Female*duration of the effect (hours) 0.02370.1280
Female*presence of adverse events −0.3346<0.00001
Female*monthly cost born by the patient 0.00120.0690
Age*speed of effect 0.00000.5960
Age*efficacy-strength 0.00000.7310
Age*duration of the effect (hours) −0.00030.6850
Age*presence of adverse events −0.00310.0660
Age*monthly cost born by the patient 0.00000.1600
N. observations27,68127,681
N. of respondents466466
Log-likelihood−8664−8550
Akaike information criterion (AIC)17,33817,114
Table 4. Assessment of the importance of the considered attributes on the Likert scale.
Table 4. Assessment of the importance of the considered attributes on the Likert scale.
GradingSpeed of EffectEfficacy-StrengthDuration of the EffectPresence of
Adverse Events
Monthly Cost
Extremely important33.9%30.0%26.6%58.2%26.0%
Very important33.0%40.6%37.8%22.3%30.5%
Quite important28.5%27.5%31.1%14.6%32.6%
Not very important4.1%1.7%3.9%4.3%8.6%
Not at all important0.4%0.2%0.6%0.6%2.4%
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Torbica, A.; Rognoni, C.; Tarricone, R. Investigating Patients’ Preferences to Inform Drug Development Decisions: Novel Insights from a Discrete Choice Experiment in Migraine. Int. J. Environ. Res. Public Health 2021, 18, 4916. https://doi.org/10.3390/ijerph18094916

AMA Style

Torbica A, Rognoni C, Tarricone R. Investigating Patients’ Preferences to Inform Drug Development Decisions: Novel Insights from a Discrete Choice Experiment in Migraine. International Journal of Environmental Research and Public Health. 2021; 18(9):4916. https://doi.org/10.3390/ijerph18094916

Chicago/Turabian Style

Torbica, Aleksandra, Carla Rognoni, and Rosanna Tarricone. 2021. "Investigating Patients’ Preferences to Inform Drug Development Decisions: Novel Insights from a Discrete Choice Experiment in Migraine" International Journal of Environmental Research and Public Health 18, no. 9: 4916. https://doi.org/10.3390/ijerph18094916

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop