Stakeholders’ views on the value of outcomes from clinical genetic and genomic interventions

The sequencing of the human genome and other related endeavors have led to remarkable advances in increasing our understanding of the genetic basis of disease. As a result, there has been exponential growth in the development of diagnostic genetic/genomic tests and targeted clinical interventions that are the foundation of precision medicine. Yet, adoption of genetic/genomic testing and clinical genetic evaluation into clinical care has been slower than anticipated, in part due to the lack of evidence describing their value.¹

The clinical utility of a genetic/genomic test is typically defined as the balance of health benefits and harms resulting from the test. As such, clinical utility is an important measure contributing to its value.² However, with respect to genetic/genomic tests, value also depends on the clinical actionability of the test results, which includes evidence for both clinical utility and clinical validity (i.e., the ability of the test result to predict the presence, absence, or risk of a disease).³ As clinical actionability increases, clinicians report a corresponding increase in the value of outcomes resulting from genetic/genomic tests.⁴ Moreover, the value of clinical genetic/genomic test results for patients and their family members can extend beyond clinical validity and clinical utility, and is referred to as personal utility, such as enhancing perceptions of personal control.⁵

As described by Porter, achieving high value for patients should be the overarching goal for health-care delivery, with value defined as patient outcomes achieved relative to cost.⁶ Value in health care encompasses and integrates other objectives, such as health-care quality, safety, patient-centeredness, and cost containment.⁷ Value is also central to improving equity and expanding access at reasonable cost.⁷ In the context of limited or conflicting evidence of value for a clinical intervention, as is the case for many clinical genomic interventions (CGIs), the perspective of stakeholders can be key in determining value and guiding implementation decisions.^8,9 The goal of this study was to understand where there is consensus among diverse stakeholder groups about the value of patient outcomes resulting from CGIs. The results could help prioritize topics for future outcomes research, as well as policy decisions for precision medicine implementation and evaluation.

We used an adaptation of the Delphi panel process to assess the opinions of stakeholder groups regarding the value of outcomes resulting from CGIs, such as diagnostic genetic tests, genetic evaluation, and therapies targeting genetic variation.¹⁰ The Delphi method involves convening a group of experts who anonymously reply to a survey and subsequently receive feedback in the form of a statistical representation of the “group response,” after which the process repeats itself. The goal is to reduce the range of responses and arrive at something closer to expert consensus.¹¹ The project was considered nonresearch by the Department of Veterans Affairs (VA) Greater Los Angeles Healthcare System Institutional Review Board.

Panelist recruitment

We used purposive recruitment to achieve a diverse participant panel that would provide a comprehensive view of precision medicine. The panelists were invited to participate in a conference sponsored by the Department of Veterans Affairs (VA). The goal of the conference was to foster the creation of partnerships among stakeholders to advance a research agenda that impacts policy, research, and delivery of precision medicine, with a focus on clinical genetics and genomics. The conference invitation also included a request to complete a survey, the results of which would inform the discussions at the conference.

Survey development

The survey asked about 44 outcomes resulting from CGIs. We identified the outcomes by searching for articles in PubMed published prior to April 2016 using the search terms of “gene* OR genom* AND outcome.” We selected articles relevant to the clinical setting.^{12,13,14,15,16,17,18,19} We excluded those limited to research or direct-to-consumer settings. The conference planning committee (M.T.S., M.M.R., C.C.-C., J.P., B.L., D.P., S.J.K., C.I.V.) reviewed the articles, developed the survey items, and grouped the outcomes into seven categories: health-related, medical management, reproductive, diagnostic/prognostic, patient behavioral, patient psychosocial, and family-related. Subject matter experts, including three clinical geneticists and three genetic counselors, reviewed the items and provided input to ensure the list of outcomes was comprehensive and relevant to clinical practice.

For each CGI outcome, the survey asked panelists to rate their agreement with two statements—whether the CGI outcome is a valuable outcome of precision medicine and whether a third party (e.g., Medicare/Medicaid, the VA, or commercial health-care insurance) should pay for the outcome—using a response scale ranging from 1 (strongly disagree) to 9 (strongly agree). The approach of asking for separate ratings of value and payment by a third party was based on our assumption that it would help distinguish the CGI outcomes that are medically reasonable from those that are beyond the realm of medical care. In framing the questions, we asked the panelists to consider each outcome as the sole outcome of a CGI when responding to each item, even if this scenario would be unlikely in real-world practice. We conducted usability testing of the survey with three individuals (a researcher, a clinician, and a consumer) and adjusted the survey items and format accordingly.

The survey asked the panelists to select the stakeholder group that best represented their primary role, as it relates to precision medicine. Options matched the five stakeholder groups invited to participate: patients/consumers (including family members and representatives of patient advocacy organizations); clinicians (e.g., physician, genetic counselor, nurse); researchers (e.g., basic scientists, health services researchers, and information technology experts); administrators/managers from health-care organizations; and policy-makers/payers from integrated health-care systems, governmental agencies, and commercial insurers. Another question asked panelists to choose any other roles from this list that they may have relating to precision medicine. We also asked panelists about their experience with precision medicine (e.g., evaluation for a genetic diagnosis or genetic testing) either as a patient or a family member of a patient.

Survey administration and panel discussion

Following the Delphi method, we conducted two rounds of anonymous survey ratings by the panelists, with discussion of the aggregate findings from the first survey in between rounds. The first round occurred prior to the conference; a web-based survey was distributed by email to 90 invited panelists. The second survey round occurred about a week after the conference; the same web-based survey was emailed to the 80 panelists who attended the conference. For each survey, we sent reminder emails twice over a two-week period, and panelists had about two weeks after the second email to complete the survey.

At the beginning of the conference, the first-round survey findings were reviewed (M.T.S.) during a plenary session, with a focus on items of disagreement or uncertainty. Discussion about these items among all panelists ensued with opinions expressed from multiple stakeholder groups. Seven panel discussions followed, one after the other, with presentations made to all participants. Panel topics included clinical genetics/genomics practice, population-based precision medicine, patient/advocacy, policy, ethics and equity, and two research panels. Each panel was comprised of a moderator (B.L., D.P., S.J.K., or C.I.V.) and five to six panelists who were invited to speak based on their expertise and experience with a panel topic. Prior to the conference, the moderators asked the speakers to address benefits of and opportunities for precision medicine, challenges and barriers to precision medicine, and strategies that can promote precision medicine adoption and implementation from the perspective of their panel topic. Time was available for audience discussion during each panel as well. The theme of CGI outcomes and their value to individuals and the population permeated the discussions.

Survey analysis

We analyzed the survey data as described by the RAND/UCLA Appropriateness Method that assesses the respondents’ level of agreement for individual items using the median rating and interpercentile range adjusted for symmetry (IPRAS) minus the IPR.²⁰ Any median rating with a negative value for IPRAS-IPR indicated disagreement. Standard operational definitions of agreement were used: a median rating of 6.5 to 9.0 without disagreement = valuable/third party should pay; a median rating of 4 to 6 or any median rating with disagreement = uncertain if valuable/third party should pay; a median rating of 1 to 3.5 without disagreement = agree not valuable/third party should not pay. For each outcome, we calculated the level of agreement overall and by stakeholder group.

Table 1 reviews the percentage of panelists responding to the surveys by stakeholder group, and shows a similar overall response rate of 73% (n = 66/90) and 75% (n = 60/80) for the first and second rounds, respectively. We know from the conference roster that about half of the panelists were women and about half were affiliated with the VA. There were slightly more administrators/managers and fewer researchers responding to the second-round survey after the conference. About half of the survey respondents had experience with precision medicine either as a patient or a relative of a patient. However, the participants who identified their primarily role as patients/consumers included four individuals with a confirmed or suspected genetic diagnosis (i.e., hemophilia, Lynch syndrome, Niemann–Pick type C, and infertility) and two individuals from two patient advocacy organizations (one was the patient with hemophilia and the other a genetic counselor). The patient with Niemann–Pick type C was accompanied by his mother, who spoke on his behalf and completed the survey.

Table 2 shows the levels of agreement about value of and third-party payment for CGI outcomes among all survey respondents. There was convergence toward more positive opinions with higher ratings from the first to second round, particularly for value ratings. The findings from the second-round survey are presented in the results that follow.

Value ratings and whether a third party should pay: all stakeholders

The overall agreement about value often matched with the overall agreement about third-party payment. (A table showing ratings of agreement about value and whether a third party should pay for CGI outcomes by stakeholder group is available as supplementary material.) Across all stakeholder groups, we observed similar trends regarding value and third-party payment for 35 of the 44 CGI outcomes, including all health-related, reproductive, and diagnosis/prognosis outcomes. For six CGI outcomes, there was agreement that the outcomes were valuable. However, there was uncertainty about third-party payment for these six outcomes, including two patient behavioral outcomes (discouraging uptake of a clinical intervention with unproven health benefits, and improving a patient’s understanding of the effectiveness of an intervention); one patient psychosocial outcome (informing a patient’s life planning decisions); two family-related outcomes (helping family members cope with having a relative with a genetic disorder, and promoting uptake of lifestyle choices by family members that can result in health benefits); and one medical management outcome (establishing eligibility for a clinical trial). For four CGI outcomes, there was uncertainty about the value and agreement that a third party should not pay for these outcomes, including two patient psychosocial outcomes (determining ancestry and establishing consanguinity), one family-related outcome (establishing paternity), and one medical management outcome (prevention through surgery that has unproven health benefits). Based on the overall ratings, we did not observe any CGI outcome rated as not valuable.

Differences in perceptions of value and third-party payment by stakeholder group

As a group, researchers gave the highest ratings for value on average (8.0) across all CGI outcomes, followed by administrators/managers (7.5), patients/consumers (7.5), clinicians (7.4), and policy-makers/payers (7.4). The average ratings across all CGI outcomes regarding third-party payment were slightly lower compared with the average ratings for value, with administrators/managers giving the highest ratings regarding third-party payment (7.4), followed by patients/consumers (7.2), researchers (6.7), clinicians (6.7), and policy-makers/payers (6.2).

The level of agreement about value and whether a third party should pay for CGI outcomes by stakeholder group is shown in Table 3. The stakeholder groups unanimously agreed that 21 outcomes were valuable, and for 14 of these outcomes they were unanimous that a third party should pay. For 7 of the 21 CGI outcomes deemed valuable, at least one stakeholder group had uncertainty about third-party payment.

The patient/consumer group had the greatest number of CGI outcomes with uncertainty about value and uncertainty about payment by a third party (13 and 18 outcomes, respectively), followed by policy-makers/payers (11 and 16 outcomes), clinicians (10 and 15 outcomes), researchers (10 and 15 outcomes), and administrators/managers (10 and 10 outcomes). Typically, the outcomes with uncertainty were in the categories of medical management or patient behavior, or were related to recommending a clinical intervention or lifestyle that has unproven health benefits, or prenatal diagnosis of a low-risk fetus.

A few stakeholder groups agreed that some CGI outcomes did not have value and a third party should not pay for them, including the policy-maker/payer group (3 medical management outcomes), clinicians (1 medical management and 2 patient psychosocial outcomes), and administrators/managers (2 patient psychosocial and 1 family-related outcome). For example, clinicians and administrators/managers agreed that determining a patient’s ancestry without a hereditary disorder in mind is not valuable and a third party should not pay for this outcome, and clinicians agreed that establishing consanguinity for a patient without a hereditary disorder in mind is not valuable and a third party should not have to pay for this outcome.

There were many more CGI outcomes for which there was uncertainty about value and agreement that a third party should not pay, and policy-makers/payers had the greatest number of these outcomes (7), followed by clinicians (5), administrators/managers (3), and researchers (2). For example, clinicians had uncertainty about the value of and agreed a third party should not pay for prenatal diagnosis of a fetus at low risk for a hereditary disorder through a procedure like amniocentesis, and policy-makers/payers had uncertainty about the value of and agreed a third party should not pay for promoting uptake of lifestyle choices by family members that can result in health benefits for those family members. None of the CGI outcomes were perceived by the patients/consumers as not valuable or that they should not be paid for by a third party.

The concept of health-care value is complex and multifaceted, integrating economic considerations and aspects of health-care quality.⁷ We found stakeholders gave high ratings for value and third-party payment for outcomes resulting from GCIs having direct health benefit (e.g., physical health, daily functioning, or reducing the need for urgent or emergent care). Stakeholders also gave high ratings for outcomes supporting an indirect chain of evidence for health benefits. For example, medical management decisions that use interventions shown to improve health outcomes had high ratings for value and third-party payment, as did patient behaviors such as uptake of a clinical intervention or lifestyle associated with improved health, or decrease in unhealthy behaviors. Overall, stakeholders also agreed about the value of and that a third party should pay for diagnostic and prognostic outcomes resulting from CGIs that might not necessarily lead to improved health outcomes, but would end uncertainty about a genetic diagnosis or genetic predisposition for a patient or their family members, and, thus, could reduce unnecessary health-care utilization and avoid harms.

For many of the behavioral and psychosocial outcomes from CGIs, there was uncertainty about value or third-party payment, such as improving a patient’s understanding of the effectiveness of treatment, management, or prevention options; informing a patient’s life planning decisions; and helping family members cope with grief, stress, and the emotional challenges of having a relative with a genetic disorder. Such topics are typically addressed in a genetic counseling session,²¹ and all are important to an informed consent process that reviews the potential benefits, harms, and limitations of a CGI.^22,23 Professional societies promote genetic counseling as integral to genetic services,^24,25 yet third-party coverage and reimbursement is limited.²⁶ Thus, there is a need to study the value of genetic counseling outcomes, either directly by measuring health-care quality improvement, or indirectly as links in the chain of evidence for health-related outcomes. Coverage and reimbursement might improve for genetic counseling outcomes with evidence of value, as value-based health-care initiatives take hold that shift the care delivery focus from volume to patient outcomes and reward health-care providers with incentive payments for the quality of care delivered.²⁷

We found trends in ratings that suggest inherent differences in perceived value of CGI outcomes by stakeholder group. We speculate that this might be explained by experience with CGIs and familiarity with the available evidence about CGI outcomes. Researchers consistently gave the highest value ratings for CGI outcomes, which supports the idea that they are more optimistic about the benefits of CGIs or less concerned about or aware of potential harms than the other stakeholder groups. This is supported by a study that found researchers who did not have clinical experience or prior experience returning secondary findings from research sequencing were more inclined to offer return of secondary findings compared with researchers who did have clinical or prior experience.²⁸

Patients/consumers consistently had the greatest uncertainty about the value of CGI outcomes and third-party payment relative to the other stakeholder groups, which suggests this group has unmet information needs about the implications of CGI outcomes and how best to balance the benefits and harms of CGIs. This has been shown previously among individuals seeking cancer genetic counseling, where the unpredictability of the future and lack of knowledge were identified as the main sources of uncertainty for decision-making.²⁹

Policy-makers/payers often diverged from the other groups on their ratings for many CGI outcomes with consistently lower ratings of agreement about value. This suggests policy-makers/payers may require more robust evidence or may have a narrower definition of value compared with the other groups. This is apparent in the Blue Cross Blue Shield Association’s medical review process of medical technologies and therapies, which “collects and analyzes available peer-reviewed evidence…then synthesizes that data and ascertains if the evidence is sufficient or insufficient to determine the effect on health outcomes.”³⁰ Generally, other patient outcomes (e.g., diagnosis, reproductive decisions) are not considered. However, we did find policy-makers/payers were the only group to agree that a third party should pay for noninvasive prenatal screening in low-risk pregnancies. Thus, when evidence exists showing superior or noninferior performance compared with usual care, as in the case of noninvasive screening in low-risk pregnancies,³¹ policy-makers/payers do recognize the value proposition of a CGI.

We also found certain stakeholder groups differed from the rest in their value ratings for specific outcomes of CGIs. We speculate that this might be explained by their role in adoption and implementation of CGIs. For example, administrators/managers were the only group that did not agree about the value of or third-party payment for treatment with a drug or medical procedure that has proven health benefits. This group—responsible for resource allocation in health-care systems—may be considering the high cost for biologics and other therapies that are the targets of CGIs and the availability of alternative, less costly treatments.^32,33 Policy-makers/payers were the only group that did not agree about the value of or third-party payment for confirming a predisposition to a hereditary disorder in an asymptomatic individual with a family history. This may be due to their focus on health outcomes³⁰ and concern about potential lack of participation in effective surveillance or preventive interventions when an inherited predisposition is confirmed.^34,35 Policy-makers/payers and clinicians did not agree with other stakeholders that a third party should pay for improvements made in a patient’s diet or nutrition, possibly because of their awareness of the limited evidence that genetics can guide nutrition choices, or that such changes would be sustained.³⁶ Finally, patients/consumers were the only group that did not agree that a third party should pay for prenatal diagnosis of a high-risk fetus through a procedure like amniocentesis. This may be due to lack of awareness regarding the high-quality evidence to support amniocentesis as the procedure of choice for prenatal diagnosis.³⁷ Additionally, attitudes about intellectual disability, congenital anomalies or pediatric genetic disorders and pregnancy termination may have influenced their opinions regarding prenatal diagnosis.^38,39

A systematic review of studies exploring stakeholder views on secondary findings resulting from exome or genome sequencing supports our suppositions that experience and role influence value ratings for CGI outcomes.⁴⁰ The review found that views of potential recipients (e.g., patients or research participants) of secondary findings were largely influenced by a sense of entitlement to the information, whereas the views of genetics professionals were informed by a sense of professional responsibility, and experience with genetic illness and testing resulted in greater caution about receipt of secondary findings. Of note, studies included in the review investigating impacts of secondary findings from exome or genome sequencing were “largely hypothetical.” Our findings expand upon this work.

The diversity of the ratings we found among the various stakeholder groups speaks to the need for further engagement of stakeholders to address the breadth of CGI outcomes. Discussions should incorporate the available evidence for CGI outcomes, the influence of experience with CGIs, the stakeholder role in implementation of CGIs, and personal preferences relating to CGI outcomes. Reaching consensus on the CGI outcomes that matter to stakeholders is a key first step in creating outcome measures for CGIs. Outcome measures can be used to generate the evidence needed to demonstrate value (i.e., patient outcomes relative to cost) resulting from CGIs; to assess the quality of genetic health care provided by clinicians; to evaluate the stewardship of CGIs within a health-care organization; and to guide policy development regarding coverage and reimbursement of these interventions.

Important limitations to this work deserve mentioning. It was somewhat contrived to ask survey respondents to consider each CGI outcome in isolation, because this is rarely the case in real-life scenarios. Therefore, the results may not be generalizable to more complex situations when more than one outcome exists. However, looking at each individual outcome allowed us to tease out value ascribed to specific CGI outcomes. This revealed a hierarchy for CGI outcomes that can be explained by the degree of actionability associated with these outcomes. When there are multiple CGI outcomes to consider for an individual, applying this hierarchy could be useful to decision-making. Additionally, parsing CGI outcomes and considering the specific impact of each should help with communication about the value of GCI outcomes, both within and between stakeholder groups.

We did not ask about specific costs for each outcome, but rather assessed cost indirectly by asking whether a third party should pay for the outcome. This was because our focus was on interventions most relevant to clinical practice. We believe asking about third-party payment (e.g., Medicare/Medicaid, VA, and commercial insurers) provided context for respondents to consider CGI outcomes that are medically reasonable and relevant to clinical practice. Our results would likely be different if we considered CGI outcomes in different settings, such as direct-to-consumer testing paid by the consumer.

We provided feedback on the anonymous survey responses in between survey rounds in person, in keeping with the original Delphi method developed at RAND in the 1950s.¹¹ Because the survey responses were anonymous, we could not provide or discuss individual judgments, which is characteristic of a modified Delphi process that is considered more robust.²⁰ While we provided feedback about the group responses, the discussion focused on items of disagreement and uncertainty, and was not exhaustive. Additional discussion or survey rounds could have resulted in greater consensus. Nonetheless, there was convergence of responses with more positive second-round ratings, especially regarding the value of CGI outcomes.

Lastly, we convened multiple stakeholder groups to weigh in on the value of a comprehensive list of outcomes resulting from GCIs, and we believe this is the first effort of this kind. Stakeholders self-assigned their primary role and some had multiple roles; however, we only considered their primary role in our analyses. Although we assembled diverse stakeholder groups, we did not include all possible groups. For example, representatives from the biotechnology sector were not included as a stakeholder group; though many of the participants had ties to this industry, especially among the researchers that included information technology experts.

In conclusion, we have characterized the value of outcomes resulting from CGIs informed by key stakeholder groups. We believe stakeholder experience and role in implementation of GCIs influence perceptions about the value of CGI outcomes. Future research is needed to better understand factors contributing to disagreement and uncertainty about outcomes from CGIs within groups and differences between groups. Consensus on the value of outcomes from CGIs will accelerate the creation of metrics to generate the evidence needed to inform value and guide practices and policies that promote availability, uptake, coverage, and reimbursement of these interventions.