Login to MyKarger

New to MyKarger? Click here to sign up.



Login with Facebook

Forgot your password?

Authors, Editors, Reviewers

For Manuscript Submission, Check or Review Login please go to Submission Websites List.

Submission Websites List

Institutional Login
(Shibboleth or Open Athens)

For the academic login, please select your country in the dropdown list. You will be redirected to verify your credentials.

Original Paper

Free Access

Researchers’ Perceptions of the Ethical Implications of Pharmacogenomics Research with Children

Avard D. · Silverstein T. · Sillon G. · Joly Y.

Author affiliations

Centre de recherche en droit public, Law Faculty, Université de Montréal, Montréal, Qué., Canada

Corresponding Author

Denise Avard

Centre de recherche en droit public, Faculté de Droit

C.P. 6128, succ. Centre-ville, Montréal, Qué., H3C 3J7 (Canada)

Tel. +1 514 343 7702, Fax +1 514 481 6144

E-Mail denise.avard@umontreal.ca

Related Articles for ""

Public Health Genomics 2009;12:191–201

Abstract

Background: This paper presents the results of an exploratory qualitative study that assesses Canadian pediatric researchers’ perceptions of a pre-selected group of ethical issues raised by pharmacogenomics research with children. Methods: As a pilot study, we conducted semi-structured telephone interviews with Canadian pediatric pharmacogenomic researchers. The interviews were guided by the following themes: (1) benefits and risks of inclusion, (2) the consent/assent process, and (3) the return of research results. Results: Issues about assent, consent, risks and benefits, as well as the communication of results were addressed by the respondents. Some issues, such as the unique vulnerability of children, the long term privacy concerns associated with biobanking, additional core elements that need to be discussed and included in the consent/assent forms, as well as the challenges of communicating research results in a pediatric research were not explicitly identified by the respondents. Conclusion: Further consideration should be given to address the ethical challenges of including children in pharmacogenomics research. This exploratory study indicates that further guidance is needed if children are to be protected and yet benefit from such research.

© 2009 S. Karger AG, Basel


Introduction

Pharmacogenomic research studies the relationship between genetics and drug responses. This research has the potential to optimize drug development and drug therapy and to reduce the risks of adverse drug reactions [1,2,3]. In addition, the application of pharmacogenomics promises to reduce the costs associated with health care services [4,5,6]. Although children might potentially benefit from such research, it raises a number of ethical issues.

Children have traditionally been underrepresented in research because they are more vulnerable than adults and therefore are more protected [7]. Furthermore, many developmental factors in early years of life, such as organ development and protein levels, influence the way that medication affects children [8]. These developmental differences between children and adults make it difficult to extrapolate meaningful drug metabolism information from adult trial data [9]. More than 75% of the drugs used to treat children have never been tested in the pediatric population. Studies have shown that children are therefore at a greater risk for severe adverse drug reactions (ADRs) than adults [9]. Including such a vulnerable population should enable pharmacogenomic researchers to assess genetic variation and gene expression, making medications potentially safer for children [4, 10].

The inclusion of children in medical research is contentious. Historically, the Nuremberg Code of Ethics protected vulnerable persons and restricted participation by making informed consent a pre-requisite. As a result, minors were generally excluded. This exclusion of children from research can be seen as unjust and a breach of their rights as they could be deprived of therapies tailored to their personal needs [11]. In 1964 the Declaration of Helsinki addressed the issue of pediatric inclusion, permitting child participation in research so long as permission from their guardians was obtained, comparable research cannot be done on adults to address the same questions, the research aims to improve the scientific understanding of the condition or of those in the same category, there is only minimal risk for the individual, the person concerned does not object, and the consent is in writing and can be withdrawn at any time [7, 12]. Several scandals involving child research illustrated their vulnerability. For example, in the Willowbrook case, mentally retarded children were infected with a hepatitis virus to better understand the natural history of hepatitis [13].

Thus, in contrast to medical research in general, pediatric research progressed very little in the 2nd half of the 20th century. As a result, pediatric researchers lacked data and often resorted to off-label prescription. In the last decade, pediatric research has become increasingly accepted [14, 15] and even deemed necessary [10]. Several guidelines make it an explicit obligation to include children, based on the principle that it would be unjust to exclude them from the benefits of research, an application of the principle of justice [16, 17].

However, it is important to realize that the inclusion of children in research is significantly different from that of adults participating, if only because children are still in need of special protection. Unlike in adults, the cognitive, psychological, and physiological characteristics of children limit their ability to express their needs and to look after their own interests. Additionally, children differ from adults because they lack the legal status to make the decision to participate in the research. While excluding them from research is unjust, concerns over their vulnerability drives the ethical evaluation of pediatric research.

In recent years, there has been a change of outlook in the ethics of pediatric pharmacological research. Ethicists, researchers, and policy makers realized that by excluding children from research they were also depriving them of the opportunity to benefit from potentially life saving medication adapted to their unique physiognomy. Increasingly, countries like Canada, the United Kingdom, and the United States favor the inclusion of children. However, there is also a concern that the pendulum has swung too far towards inclusion over protection [18]. This raises interesting issues regarding the researcher’s duty to safeguard children from harm while simultaneously encouraging their inclusion in research.

Ethical reflection in the field of pediatric pharmacogenomic research is still underdeveloped. A limited number of articles have focused on a broad range of ethical considerations that may be more pronounced in the context of pediatric pharmacogenomics research [1, 6, 10, 19, 20]. These issues include protecting vulnerable populations, consent and assent, considerations of risks and benefits, the communication of research results, public and health professional knowledge, unintended consequences of creating therapeutic orphans, and the effect of research on the pediatric drug approval process and recruitment.

Among the mechanisms that protect children against possible negative effects of participation in research is the requirement of consent and assent. Consent and assent are based on 3 key elements: information, voluntariness, and the capacity to understand. In a pediatric research context, the decision-making process involves the views of the parent, the child (when appropriate), and the researcher [21]. While parental input is considered fundamental and legally required, it has been suggested that the researcher should play a greater role in determining the benefits and risks of the child’s research participation [22]. This is because parents have the potential to make decisions that may be counter to the interest of the child. Moreover, there is a growing recognition that children should have a choice, known as assent, in their own participation [23]. A number of policy statements recognize that maturity and competence of children are fluid concepts, that children should participate in important decisions about their lives when developmentally appropriate, and that if they are too young to consent, their assent should be sought [24, 25].

To minimize the possibility of harming children, their inclusion in research is subject to an evaluation of risks. Questions related to risk in the pharmacogenomic context include: will children experience pain or fear? Will the revelation of genetic information invade children’s privacy? Will the use of biobanks to store, for example, samples and genetic information lead to discrimination later in life [26]?

In pediatrics there is a need to address privacy issues specific to biobanks, because when children reach the age of majority, they may be unaware or have forgotten that samples were collected and banked and perhaps still used in research. These pediatric research participants may come to disagree with their parents’ decision to allow their samples and information to be stored and/or used for research. Furthermore, the family focused nature of genetic information is another issue at the heart of confidentiality and risk assessment, because the genetic results may have implications for family members as well.

The issue of the return of results is increasingly discussed in the literature [27, 28] and has recently been the subject of recommendations at the international level [19]. This issue also raises many difficult questions such as: what are the results that need to be shared (general/specific), and when? What is the duration of this obligation: a few years, the life of the participant, or the duration of the project? How does one respect the right not to know?

While many of these ethical issues have been raised in the bioethics literature [10,29,30,31], little is known about the views of parents, children, family members, investigators, and ethics review board members regarding pediatric pharmacogenomic research. Although gathering the views of a wide cross-section of stakeholders is needed, we first concentrated on the ethical challenges experienced by clinician researchers who conduct pediatric pharmacogenomic research. This pilot research project consisted of interviews with leading pediatric pharmacogenomics researchers in Canada to obtain their views on some of the social and ethical implications of including children in their research. Owing to the exploratory nature of the study and the limited amount of researchers working in this specialized area in Canada, a small number of pharmacogenomic researchers were recruited. The list of potential contacts was developed by identifying researchers carrying out pediatric pharmacogenomic projects (2003–2007) listed on websites of governmental and non-governmental organizations and from searching PubMed. Potential names were also obtained by asking key informants to name colleagues. Individuals were eligible for inclusion if they worked in a clinical environment. A total of 13 researchers were identified, and 8 accepted our invitation.

The respondents’ ethical concerns about pediatric pharmacogenomics will be analyzed for their congruence with current bioethical literature and guidelines. These preliminary results will be used as an aid to design a future questionnaire that will be administered to a larger group of researchers in another phase of the research.

Methods

Because little is known about the experiences of pediatric pharmacogenomic researchers regarding the ethical and regulatory questions they encounter during their research, we adopted an exploratory qualitative research design. A questionnaire was based on the results of a previous review of the key ethical issues emerging from the literature on pediatric pharmacogenomic research [10]. A topic guide, informed by key themes emerging in the pharmacogenomics bioethics literature, focused on risks and benefits, consent, and the return of study results in the context of pediatric pharmacogenomics (table 1). Based on the inventory of pediatric pharmacogenomic research in Canada, we consider that our sample of 8 is to some extent representative of research practices in this country. For the purpose of this study, the term pharmacogenomics is used in the broadest sense and includes pharmacogenetics.

Table 1

Questionnaire

http://www.karger.com/WebMaterial/ShowPic/248935

Semi-structured interviews in the language of the interviewee’s choice (French or English) were carried out between November 2006 and April 2007. Most interviews were carried out in English by the same researcher (D.A.) and research assistant (G.S.), and one respondent elected to respond in writing. Telephone interviews lasted approximately one hour, were taped, and transcribed verbatim. The research protocol was reviewed and approved by the University of Montreal research ethics committee. Data was thematically coded by the same research assistant (G.S.) and principal investigator (D.A.) to correspond with the study questions and to address themes that arose from answers to open-ended questions. Secondary codes were assigned to reflect the many nuances of the results.

Results

Although the word vulnerability was not used in the questionnaire and the researchers were not specifically asked about the duty to protect potentially vulnerable persons, several of our questions alluded to the vulnerability or the inclusion of children as research participants. In particular, the questions about recruitment or consent triggered responses that dealt specifically with these issues (table 1).

Results are organized into the following 4 themes: (1) vulnerability and protection, (2) benefits and risks, (3) consent/assent, and (4) return of research results (table 2). Each theme is further subdivided in several categories, and for each category we have included excerpts from the interview transcripts.

Table 2

Themes and issues raised by respondents

http://www.karger.com/WebMaterial/ShowPic/248934

Theme 1: Vulnerability and Protection

There was an understanding amongst respondents that the protection of children is what drives the ethical evaluation of consent, assent, and the weighing of the risks and benefits. Most respondents expressed confidence in their own ability to handle the ethical implications of research with children and that their medical establishments were ‘specialized’ in this domain. These explanations were often cited to demonstrate that research with pediatric patients did not raise particular difficulties for their research.

‘Obviously there are special concerns about children. Obviously they can’t consent; you can’t look for informed consent from a newborn. But I think those of us who do pediatric studies have learned to live with that over the years. [I] think, to people who work in adult medicine, they look unique, but those of us who have worked with children over the years have developed a consistent ethical framework that we operate in, and as far as we know it’s in keeping with standards of society.’

In general, the researcher respondents were concerned that children were being left behind in much of the modern therapeutic era and therefore focused on the need to promote the inclusion of children in research. In essence, the researchers felt the imbalance between the quantity of adult research to pediatric research was unethical, and thus sought to equilibrate the difference.

‘I just think there is a major ethical imperative to do this work. I mean when we know that so many children, everyday, are being impacted by the effect of the drugs through the failure of these drugs to be adequately tested in children, we have to catch up and do the appropriate research to minimize these effects. I think we are judged by the way we treat our most vulnerable in society and I would say it was woefully inadequate.’

‘There are ways of doing research that are ethical, but I also think that not doing certain types of research is probably unethical. [W]hen we talk about [ethical] issues, it is probably to demystify all this and to make sure that this population is studied.’

One respondent expressed concern about treating children differently from the rest of the population in genomics research. It was argued that this would be a disservice to children particularly because children stand to gain more from better genomic research than anybody else.

‘So I hope that we’re not moving towards some misguided ideas that children need to be protected from genomic research and that therefore they shouldn’t be allowed to participate until they turn 16 or something.’

In general, pediatric pharmacogenetics was considered an understudied area that lacks pertinent information. Moreover, although the respondents supported the enrolment of children in their research study, they were cognizant of the fact that children should not be forced to participate against their will, that child assent should be sought when possible, and that not all drugs were appropriate to be tested on children.

Theme 2: Benefits and Risks

Benefits

Three main benefits of carrying out pharmacogenomics research in pediatric populations were identified by interviews. There is great potential for pharmacogenomics research because it: (1) enables the discovery of reliable information about drug response, (2) contributes to improved clinical use of medications in pediatric populations, and (3) contributes to safer pharmaceutical products for children by reducing the risk of adverse reactions.

(1) Reliable information about drug response. Respondents felt strongly that historically children have been underrepresented for various reasons in drug related research and are considered therapeutic orphans. As a result, most participants endorsed pediatric pharmacogenomic research. Many respondents also indicated that advances in pharmacogenomic research will allow a better understanding of pharmacokinetics, the relationship between drug response and genotype interactions, and of the genetic and epigenetic effects of genes involved in pediatric pharmacology.

‘We’re saying very simply, if you know which patients are going to have an altered metabolism, both by virtue of their age as well as by virtue of their genetics and you’re able to monitor and change that, it’s profoundly important to do this.’

‘I think we all have understood that intuitively for a very long time, it is about the interaction between your genetic make-up and the environment that you are in and obviously that changes as you go from being a newborn to being an adolescent.’

Some respondents pointed to the uniqueness of children because of their rapid development. Children are a group whose physiological and cognitive development is at a different stage than adults. Children’s responses to drugs can only in few limited cases be extrapolated from adult trial data. Because drugs are metabolized differently in children than adults, little is known about children’s reactions to medication.

‘... there is very little known from a pharmacokinetic principle, pediatrics patients are often misconstrued as being small adults but mechanistically their gene paths are expressed at different times ... So drugs are metabolized in very different ways in children, and very little is understood about that because pharmaceutical trials don’t really take into account these things.’

The lack of research regarding pediatric drug response was attributed to the practical difficulties in conducting pediatric research. For example, assessing drug responses in very young children is difficult because the symptoms or endpoints are not easy to appraise or difficult to recognize. A respondent commented:

‘How do you evaluate drug responses in a 2 or in a 5 year old? It’s very hard to ask them whether or not they’re having any joint pain or nausea even. Those concepts are adult concepts, and so it isn’t easy to detect trouble that is brewing, and it is growing until it’s a full scale serious reaction.’

Some respondents recognized that the capacity to carry out this type of research requires special expertise on the part of the health professionals. Without this expertise, moving forward in this area will be difficult and will provide a barrier to adequately communicating research findings within the health care community. Some respondents were concerned that pediatric pharmacogenomics research is under-funded and will therefore constrict the development of this needed expertise.

(2) Improved clinical application. Clinical research establishes how to efficiently administer medication. Pharmacogenomics can be used as a tool to improve the clinical application of drugs which are currently used to treat children, but have not been tested on children.

‘Some 80% of prescription drugs used in Canada have not been tested in children and they’re all used because what else do you do when you have pharmaceutical needs of somebody under 18? You can get a product on the market in pediatrics without actually testing it in children, ever.’

(3) Safer products. Several respondents explicitly mentioned that pharmacogenomic information could be of relevance to study adverse drug reactions (ADRs). Indeed, 15% of hospitalized children experience ADRs, with 28% of those leading to life-threatening or disabling reactions [25, 26]. ADRs are poorly understood, especially in children. Pharmacogenomics represents an opportunity to identify biomarkers that would predict a child’s susceptibility to drug toxicity, to assess concentration dependant reactions, and to improve the quality of health services. More generally, respondents stated that ADR research is needed to promote a safer environment.

‘Adverse reactions are like auto accidents, they are so routine and so much part of everyday life that we really don’t worry when we get into a car and we drive about. We should probably be much more attentive to this.’

One respondent noted that a serious difficulty in studying, understanding, and reducing ADRs is that a great majority of cases are unreported.

‘95% of ADRs aren’t reported, there is no understanding of risk and then we use these drugs and we end up with these tragic consequences without any collective data to help us understand how better to solve this problem in the future.’

Risks

Most investigators raised issues about the acceptable levels of risk to which children may be exposed in the context of pediatric pharmacogenomic research. The respondents recognized that there could never be ‘no risk’ for children and described a variety of possible concerns: (1) physical risk, (2) psychological impact, (3) privacy concerns, and (4) potential stigmatization.

(1) Physical risk. Respondents felt that the physical risks for participants posed by pharmacogenomics research were generally low. This is because pharmacogenomic studies tend to be performed in conjunction with clinical studies, where the patient is already ill, medications are already being administered, and samples are already being taken.

‘Ordinarily I see a pharmacogenomic study piggy-back on another protocol where the administration of the drug would be justified for patient-care purposes, with the possibility of a benefit to the subject. In that case, the risks are those of the original study.’

Several respondents explained that genotyping can be accomplished without hurting the child because of non-invasive techniques such as buccal swabs, spitting, or the ability to withdraw blood at the time of a treatment procedure.

‘So, we use 2 other methods, which are blood and saliva. Kids love to spit. It’s interesting, you know, they ask ‘what do I have to do?’ And I say ‘you spit’, they say ‘that’s it?’ and I say ‘Yeah, that’s it.’ They’re like ‘great’. So that helps. And then blood sometimes, depending on who they are, they don’t mind because they get blood drawn a lot. If we can meet them at the lab and get a tube at the end of the regular blood draw, then we do that, so we can avoid stress.’

(2) Psychological impact. One respondent expressed concern that parents who have been seeking answers about their child’s treatment may feel anger at professionals if, as a result of a pharmacogenetic test, they are advised that their child should not receive the drug because of a specific variant in one or more genes; this may mean the child will be unresponsive to the treatment. This refusal to prescribe a drug may cause shock, despair, or loss of hope.

‘I think the principal risk is if, for example, you test a genotype of an individual and you discover that his genetic profile prevents him from receiving the planned medication due to a high reaction risk factor ... So you don’t prescribe the medication, but at the same time, you have no alternative medication to offer. At times, some people argue that we like better not knowing. I have the impression that when the clinical staff tells the parents, it must be extremely hard on them, because they will have the impression that due to the gene ‘XYZ’, their child is not able to receive the best treatment. I have the impression that at that level, the level at which we communicate information to the patient and their family ... that is what needs attention ... that is where the risk is.’

(3) Privacy. The risk of breach of privacy was mentioned by some respondents, specifically the possibility of the pharmacogenomic data being used by 3rd parties. One respondent remarked that this problem can be more acute with genetic information and in areas like mental health.

‘Well, there is always a risk of breach of privacy. Somehow, the 3rd party getting information and using it in a way that was not intended. So, I think that is an issue with any research, really, especially in mental health and especially in genetics.’

Nevertheless, another participant argued that, while this risk is possible, it is unfair to children if they cannot profit from research information because of privacy concerns.

‘This is a problem if someone cannot profit from your research information because a certain insurance company can profit from this. For me, there is no sense in this, because this is the problem of the insurance company or the problem of society who is letting others have access to inside information and not really the problem of the researcher. So, would you stop providing this information to the patient ... I mean the clinician should have this information in order to attend to treatment.’

(4) Stigmatization. Another risk identified by the respondents was the fact that some genes relevant to drug metabolism vary in their frequency between different ethnic groups. If a drug reaction is more common in certain populations, this may lead to racial profiling and reinforce the risk of cultural stigmatization. Stigmatization can potentially change the way society looks at members of a family or cultural group.

‘There is a fundamental possibility that there may be a basis for racial differentiation, and some people find that threatening. I see more of a theoretical risk than a real one, I won’t call it a risk in the normal sense but it is a reality of the times. Let’s say we found a kind of drug reaction that occurred only in Aboriginal people that would be a lot harder to study than finding a reaction that occurs in Anglo-Saxons or Caucasians.’

Theme 3: Consent and Assent

In order to obtain consent in pediatrics, both the needs of the child’s legal representative and the child’s capacity to give assent must be considered. The process becomes increasingly complex in the case of pharmacogenomic pediatric research because of having to describe concepts like ‘genotyping’, the probabilistic nature of the information, the potential future uses of the data or tissue, and biobanking. The respondents raised at least 3 issues: (1) the duration of the consent process, (2) their approach to the inclusion of children, and (3) the need for further guidance.

(1) Duration of the consent process. Informed consent in pediatric research takes time and requires that a research nurse or a research coordinator plays a critical role. The researcher should not only give explanations to the parents and child, but also link the different actors of the research project – researchers, ethics committees, parents, and children – to the consent process.

‘You need to have someone on site at the hospital that can spend the time, explain the thing and go through the consent form.’

‘[...] I think that you need to talk to my research nurse. She does all of our ethical stuff and she is constantly in touch with the Ethics Committee.’

(2) Approach to the inclusion of children. Many respondents felt that children 12 years and older should be consulted about their willingness to participate in research. They also recognized that children younger than 12 years should be provided with information and involved in the decision process as much as possible, according to their capabilities. One researcher indicated a key question that children often ask when deciding whether to participate: ‘What are you going to take from me?’

More generally, respondents highlighted the need to communicate with children in a clear and concise manner that makes them feel respected. Some researchers talked about the consent and assent forms they use for parents and children and stressed the need for intelligible consent forms that are specifically designed for children. While parents should of course be involved at this stage, the process of communication could be further enhanced if children were made to feel part of the decision process.

‘I always think that children in early adolescence enjoy being communicated to by adults. I sit down with them and explain what we are doing and tell them that I’m going to leave them so that they can read it. And, if they have questions when I come back that we can talk about it and with those kinds of issues, and then I always stress that they don’t have to do this. This is something that they choose to do. If you are talking to 12 year olds and you are not known to the family then I always have the parents there. But, I am really talking to the child and then if the child says okay then we go to the parents and talk to them, but I don’t start it the other way around. I’ll say I want to talk to your child about something that I am working on.’

(3) Need for further guidance. Some respondents commented on the discrepancies between the ethical review boards across the country that have different standards regarding the ethics approval of consent forms. One respondent advocated the development of a framework outlining consent-related obligations and the best practices for pharmacogenomic researchers in order to unify the consent process and ethical evaluation.

‘What would be really helpful for me is if you collected consent forms from everybody and came up with kind of the ideal framework or the issues that we should be working on with our individual Ethical Review Board.’

Theme 4: Communicating Research Results

All respondents supported the notion of only reporting clinically relevant and informative research results to participants (parents and children). However, they also agreed that (1) participants should be informed that results are generally ‘inconclusive’ at this stage, (2) who, to whom, and how should research results be communicated, and (3) what to communicate to families?

(1) Which results will be returned and how? Respondents recognized the importance of including a statement in the consent form addressing whether any individual results will be returned to the participant. However, it was also generally felt that most information would be inconclusive, owing to the fact that the scientific evidence, the robustness of the knowledge, and the validated clinical information are still inadequate at this stage. For this reason, communicating individual results to participants was discouraged.

‘At this stage of our knowledge, the results of such studies are irrelevant to the individual participant. When our knowledge will be complete enough for individual results to be meaningful, we will need to think of how to do it.’

Nonetheless, researchers recognized that a research study may unexpectedly give rise to clinically significant results. They also recognized that pharmacogenomic research may reveal incidental information that may be relevant to the participant’s medical care. In such cases, with ethics committee approval and respecting the participant’s desire to know or not know, researchers may have an obligation to disclose the information if it potentially could prevent future harm.

‘The results more often than not are going to be inconclusive. But, you know, increasingly there will be positive findings and when that happens, I think that information must be communicated back to the patient, the children, and their families.’

(2) Who, to whom and how should research results be communicated? The respondents put forward many ideas on how to address the return of results, particularly when clinically significant results need to be revealed. They suggested that because this is highly complex information, it would be helpful to have a nurse or genetic counselor involved. The nurse was seen as especially helpful, because of her involvement in the initial collection of consent forms.

Many commented on ‘to whom’ and ‘how’ to communicate this information. The participants emphasized that generally research results are not geared for the participant but rather for the pharmacogenomics researcher. Sharing results thus tends to be through scientific publication. However, researchers try to ensure that the general findings in these publications are also available to the participants.

‘We are sharing with each center all the publications and everything else, which we encourage them to share with individuals.’

‘We are downloading our published articles to our website so that patients can have a look and see.’

It is also hoped by these researchers that as the field develops, the information will be shared with a broader health professional audience who will use the information to inform and treat patients.

‘I am hoping that doctors, as they become more educated, will be able to distil this information to their patients in a more straightforward manner. But what’s going to happen as I see it, is that there is going to be information that’s going to come back to the patient or to the doctor or the pharmacist for that matter.’

(3) What to communicate to families? Pharmacogenomic research has implications not only for the child enrolled in a study, but also for the family. Issues were raised regarding the communication of research results with family members. One respondent described the family perspective in this way:

‘What they are worried about is if these reactions occurred this time, is his brother susceptible? What about his little sister who is there, or what about him the next time he needs to be treated? What are we going to do and what is going to be safe and what’s not safe? So, there is this very pragmatic sense of the need for this information to help them better understand and manage the risk within the family.’

Discussion

A number of important themes and issues were identified in this exploratory study. These themes will provide a further framework to facilitate a future in-depth analysis of the socio-ethical issues in pediatric pharmacogenomic research. While a number of important issues were raised by the respondents, they also failed to address other issues. Both warrant consideration and have implications for the future of pharmacogenomic research. In this discussion, we compare the study observations with the current bioethics literature and international guidelines and highlight 4 ethical issues not explicitly raised by the researchers: (1) vulnerability, (2) privacy concerns associated with biobanking, (3) core elements that need to be discussed and included in the consent/assent forms, and (4) the return of research results.

Vulnerability

Vulnerability of children as a group is central. Generally, the respondents addressed the need to include children in research particularly to correct the imbalance between pharmacogenomic child research compared to adult research, an imbalance that they consider unethical. However, the respondents did not specifically address the issue of vulnerability. Although one obvious explanation is that our questionnaire did not specifically ask about vulnerability. It is important to note that considerations about vulnerability flowed implicitly and underpinned many of the responses to the ethical issues discussed during the interview including consent, assent, and the evaluation of risk.

The importance of including children in research should not obscure the fact that they remain a vulnerable population who need specific safeguards. Several international guidelines promote the protection of children: children can only participate when they are subjected to minimal risks, the best interests of the child should be taken into consideration, children should only be involved when it is absolutely necessary, the consent of the legal representative should address their role as to the confidentiality of the child’s data or the communication of the child’s research result, and the assent of the child should be obtained prior to participation in a research [32]. The core elements that emerge from this exploratory study include the need for the consent to be both voluntary and obtained from the parents/legal representative as well as the need to obtain assent from the child. Also it was recognized that the potential risks and benefits should be explained in an understandable language, adapted to the age, maturity, and socio-cultural context of the child. However, many of these elements were described as challenges by the respondents.

Related to the protection of children as particularly vulnerable is the assessment of the risks and benefits of the study they participate in. Respondents stressed the need to minimize the physical risks to children when in a pharmacogenomics study. Because these studies are often part of wider clinical trials, or added on to routine clinical care, respondents suggested that the invasiveness of taking a sample is less intimidating or painful for children. Although this might be true, Friedman-Ross argues that any burden which is imposed on an already sick child is unfair and should be taken into consideration [33].

The fact that the respondents did not stress vulnerability in a singular way appears at odds with the literature where including children in research always raises issues around balancing inclusion with protection [18, 34, 35]. In addition to the fact that the questionnaire did raise this as an issue, this lack perhaps can be attributed to several factors, including but not restricted to (1) limited funding for this type of research [36], (2) the difficulty in recruiting pediatric patients to participate in this research [24], (3) the difficulty in extrapolating health information about children to other groups of people (such as children of different ages or adults) due to differences in their physiological development [37], and (4) the difficulty in assessing symptoms in pediatric patients as a result of their cognitive development [38].

Future studies will be needed to specifically address the issue of whether the vulnerability of children is an ethical issue per se or of whether it is in fact the source of the special protections offered.

Privacy

Collecting sensitive information from children brings up the issues of privacy, since medical and genetic information can be highly sensitive. Several respondents raised concern about privacy, for example the potential for access to a child’s genetic information by insurers, employers, or schools. This concern is reflected in the literature, where access to sensitive information like genomic information by 3rd parties has the potential for discrimination or stigmatization [10, 39, 40], especially in children where the research results could follow them for the rest of their lives [26].

While the respondents raised concerns about access to information by 3rd parties, the issue of the long term protection of privacy of the child’s sensitive information was not mentioned. This in itself is interesting, since biobanks are a focal point of much pharmacogenomic research and the inclusion of information on children in DNA databanks raises specific ethical and legal issues [41]. For example, these biobanks require broad consent, whereby parents on behalf of their child agree to future unspecified uses of data. Is it ethical to have a broad consent on behalf of children? Current guidelines recommend that legal representatives should be aware of the potential privacy issues associated with sensitive information and biobanking, which could have an impact on the child’s life choices [42].

Emerging Core Elements Regarding Consent and Assent

Other emerging issues which are not mentioned by the respondents0 but are a source of ethical debate in the literature and the guidelines include compensation of the participants [16, 17], storage and destruction of data after completion of the study [43], future research use of the collected samples, policy on the communication of research results [19, 44], identification of sources of funding, and the possibility of commercialization of the results [16, 17]. From an ethical point of view, there is a need to think about these issues when addressing the informed consent [45].

Interestingly, the pivotal role of the nurse coordinator in handling the consent forms was frequently mentioned and for this reason the role of the nurse researcher in the consent process warrants consideration in future research.

Returning Research Results

Determining how to communicate the research results to the children and their parents, what type of results and when, is an important responsibility of the researcher [28]. Respondents remarked that at this time, pharmacogenomic research is not yet yielding valid or conclusive health information. It was generally recognized though that research results should be communicated if and when the data is clinically significant and validated.

The respondents stated that they are open to sharing their published research data with children and their parents. However, scientifically valid and clinically significant data is normally only available years after the original study. Furthermore, children and parents likely have no easy access to the published, highly specialized literature. It seems that there is no clear mechanism in place for conveying important medical findings, especially when these findings are discovered years after the original study. While general principles, e.g. that the process for returning results should be covered in the consent form, seem a practical solution, how to do this is another question.

The issues on how to return results and to whom also remain unresolved. Few documents address the return of results in the pharmacogenomic context, and to our knowledge none deal specifically with returning results in the pediatric pharmacogenomic context. A Canadian survey addressing the issue of returning research results noted that favorable or neutral results may be communicated by letter or e-mail, whereas ‘bad’ results should be provided in a personal contact [27].

Moreover, research on the return of study results in the context of research involving children raises particular ethical questions. For example, test results may have unintended predictive information for diseases. Should such incidental findings be returned to parents? Do parents have an ethical duty to disclose the results to their child? Do researchers have an ethical duty to communicate research results? Because of potential psychological effect on the participants’ siblings and other family members, it is essential that the information and its implications be properly explained to the research participants and their parents. This raises similar issues in the consent process: how can complex information be simplified without distorting their meaning, or how can it be presented without sounding too optimistic?

Limitations

The small scale of this exploratory study gives rise to several limitations. It was carried out with a small sample and in one country. Based on the inventory of pediatric pharmacogenomic research in Canada, we consider that our sample of 8, although small, is to some extent representative of research practices in this country. However, our data do not allow us to fully address the possible heterogeneity within the group of pediatric researchers. In addition, we recognize that the views of parents, children, and other health care professionals, such as the research nurse and members of ethical committees, are of equal value. The views expressed by the latter groups will be the subject of later studies.

Conclusions

The above mentioned limitations should not detract from our conclusion that more consideration should be given to the ethical challenges of including children in pharmacogenomic pediatric research. This exploratory study indicates that further guidance is needed if children are to be protected and yet benefit from such research. A broader study addressing similar themes and including the families, children, and other health professionals, such as nurses, will expand our understanding of pharmacogenomics research.

In this study, we sought to present an overview of the ethical challenges of pediatric researchers involved in pharmacogenomic research. Children stand to benefit greatly from research in pharmacogenomics, thus tackling the emerging ethical issues associated with this research should be a priority. Most of the statements of the respondents reflect the issues as discussed in the literature. However, some issues such as the unique vulnerability of children, the long term privacy concerns associated with biobanking, additional core elements that need to be discussed and included in the consent/assent forms, and the challenges of communicating research results in a pediatric research were not explicitly identified by the respondents, but are key.

The ethical concerns raised by pediatric pharmacogenomics are not unique as they overlap with issues raised in other disciplines, such as standard pediatric research, pediatric genetic research, and adult pharmacogenomic research. The key to fostering the ethical development of this research field is the development of proper tools and for counseling children, their families, researchers, health professionals, and the public about the implications of research. In addition, in order to maintain a dynamic approach to this research, it is important to keep open the lines of communication between all these key stakeholders so that children may fully benefit from this new field of research.

Acknowledgement

The authors would like to thank all of the participants for their insightful contributions and reviewer Gillian Nycum and Isabelle Paradis from the Centre de recherche en droit public (CRDP) at the University of Montreal for their help. We are most grateful to the anonymous reviewers. We acknowledge the support from the Canadian Institutes of Health Research (CIHR) under the CIHR grant # FFO 72349.


References

  1. Haga SB, Burke W: Pharmacogenetic testing: not as simple as it seems. Genet Med 2008;10:391–395.
  2. The Royal Society: Personalised medicines: hopes and realities. London, The Royal Society, 2005.
  3. Ross CJ, Carleton B, Warn DG, Stenton SB, Rassekh SR, Hayden MR: Genotypic approaches to therapy in children: a national active surveillance network (GATC) to study the pharmacogenomics of severe adverse drug reactions in children. Ann N Y Acad Sci 2007;1110:177–192.
  4. Leeder JS, Kearns GL: The challenges of delivering pharmacogenomics into clinical pediatrics. Pharmacogenomics J 2002;2:141–143.
  5. Ross CJ, Katzov H, Carleton B, Hayden MR: Pharmacogenomics and its implications for autoimmune disease. J Autoimmun 2007;28:122–128.
  6. Freund CL, Clayton EW: Pharmacogenomics and children: meeting the ethical challenges. Am J Pharmacogenomics 2003;3:399–404.
  7. Diekema DS: Conducting ethical research in pediatrics: a brief historical overview and review of pediatric regulations. J Pediatr 2006;149:S3–S11.
  8. Impicciatore P, Choonara I, Clarkson A, Provasi D, Pandolfini C, Bonati M: Incidence of adverse drug reactions in paediatric in/out-patients: a systematic review and meta-analysis of prospective studies. Br J Clin Pharmacol 2001;52:77–83.
  9. Leeder JS: Developmental and pediatric pharmacogenomics. Pharmacogenomics 2003;4:331–341.
  10. Joly Y, Sillon G, Silverstein T, Krajinovic M, Avard D: Pharmacogenomics: don’t forget the children. Current Pharmacogenomics and Personalized Medicine 2008;6:77–84.
  11. Ernest TB, Elder DP, Martini LG, Roberts M, Ford JL: Developing paediatric medicines: identifying the needs and recognizing the challenges. J Pharm Pharmacol 2007;59:1043–1055.
  12. Knoppers BM, Avard D, Cardinal G, Glass KC: Science and society: children and incompetent adults in genetic research: consent and safeguards. Nat Rev Genet 2002;3:221–225.
  13. Beecher HK: Ethics and clinical research. N Engl J Med 1966;274:1354–1360.
  14. McIntosh N, Bates P, Brykczynska G, Dunstan G, Goldman A, Harvey D, Larcher V, McCrae D, McKinnon A, Patton M, Saunders J, Shelley P: Guidelines for the ethical conduct of medical research involving children. Royal College of Paediatrics, Child Health: Ethics Advisory Committee. Arch Dis Child 2000;82:177–182.
  15. American Society of Human Genetics, American College of Medical Genetics Board of Directors: Points to consider: ethical, legal, and psychosocial implications of genetic testing in children and adolescents. J Hum Genet 1995;57:1233–1241.
  16. Medical Research Council of Canada, Natural Sciences and Engineering Research Council of Canada, and Social Sciences and Humanities Research Council of Canada: Tri-Council Policy Statement – Ethical conduct for research involving humans. Canadian Institute of Health Research, 1998.
  17. Council for International Organization of Medical Sciences: International Ethical Guidelines for Biomedical Research Involving Human Subjects. Geneva, CIOMS, 2002.
  18. Friedman-Ross L: Children in Medical Research. Oxford, Clarendon Press, 2006.
  19. Knoppers BM, Joly Y, Simard J, Durocher F: The emergence of an ethical duty to disclose genetic research results: international perspectives. Eur J Hum Genet 2006;14:1170–1178.
  20. Avard D, Joly Y: Improved understanding of genetic and genomic influences on drug disposition and action: implications for children. Paediatr Drugs 2008;10:275–278.
  21. Ross LF: Genetic testing of adolescents: is it in their best interest? Arch Pediatr Adolesc Med 2000;154:850–852.
  22. Wilfond BS, Carpenter KJ: Incidental findings in pediatric research. J Law Med Ethics 2008;36:332–340.
  23. United Nations: The Convention on the Rights of the Child. Geneva, 1989.
  24. Patenaude AF, Senecal K, Avard D: Whither paediatric research and predisposition genetic testing? GenEdit 2006;4:1–9.
  25. Avard D, Knoppers BM: Ethical dimensions in paediatric neurology: a look into the future. Semin Pediatr Neurol 2002;9:53–61.
  26. Kaufman D, Geller G, Leroy L, Murphy J, Scott J, Hudson K: Ethical implications of including children in a large biobank for genetic-epidemiologic research: a qualitative study of public opinion. Am J Med Genet Part C Semin Med Genet 2008;148C:31–39.
  27. Fernandez CV, Santor D, Weijer C, Strahlendorf C, Moghrabi A, Pentz R, Gao J, Kodish E: The return of research results to participants: pilot questionnaire of adolescents and parents of children with cancer. Pediatr Blood Cancer 2006;332:188–189.
  28. Fernandez CV, Kodish E, Weijer C: Informing study participants of research results: an ethical imperative. IRB 2003;25:12–19.
  29. Joly Y, Knoppers BM: Pharmacogenomic data sample collection and storage: ethical issues and policy approaches. Pharmacogenomics 2006;7:219–226.
  30. Phillips MS, Joly Y, Silverstein T, Avard D: Consent in pharmacogenomic research. GenEdit 2007;5:1–9.
  31. Sillon G, Joly Y, Feldman S, Avard D: An ethical and legal overview of pharmacogenomics: perspective and issues. Medicine and Law (in press).
  32. Canadian Paediatric Society: Guidelines for genetic testing of healthy children. Paediatric Child Health 2003;8:42–45.
  33. Friedman-Ross L: Should we provide healthy children with greater protection in medical research? in: Children in Medical Research. Oxford, Clarendon Press, 2006, pp 76–86.
  34. Meaux JB, Bell PL: Balancing recruitment and protection: children as research subjects. Issues Compr Pediatr Nurs 2001;24:241–251.
  35. Knox CA, Burkhart PV: Issues related to children participating in clinical research. J Pediatr Nursing 2008;22:310–318.
    External Resources
  36. Budetti PP: Ensuring safe and effective medications for children. JAMA 2003;290:950–951.
  37. Steinbrook R: Testing medications in children. N Engl J Med 2002;347:1462–1470.
  38. Matsui D, Kwan C, Steer E, Rieder MJ: The trials and tribulations of doing drug research in children. CMAJ 2003;169:1033–1034.
  39. Burke W, Diekema DS: Ethical issues arising from the participation of children in genetic research. J Pediatr 2006;149:S34–S38.
  40. Hirtle M: Children and genetics: comparative study of international policy positions. Health Law J 1998;6:43–81.
    External Resources
  41. Caulfield T, Upshur RE, Daar A: DNA databanks and consent: a suggested policy option involving an authorization model. BMC Med Ethics 2003;4:E1.
  42. Moore CA, Khoury MJ, Bradley LA: From genetics to genomics: using gene-based medicine to prevent disease and promote health in children. Semin Perinatol 2005;29:135–143.
  43. Holm S: Informed consent and the biobanking of material for children. Genom Soc Policy 2005;1:16–26.
  44. Henrikson NB, Burke W, Veenstra DL: Ancillary risk information and pharmacogenetic tests: social and policy implications. Pharmacogenomics J 2008;8:85–89.
  45. Knoppers BM: Consent revisited: points to consider. Health Law Review 2005;13:33–38.
    External Resources

Author Contacts

Denise Avard

Centre de recherche en droit public, Faculté de Droit

C.P. 6128, succ. Centre-ville, Montréal, Qué., H3C 3J7 (Canada)

Tel. +1 514 343 7702, Fax +1 514 481 6144

E-Mail denise.avard@umontreal.ca


Article / Publication Details

First-Page Preview
Abstract of Original Paper

Published online: February 10, 2009
Issue release date: February 2009

Number of Print Pages: 11
Number of Figures: 0
Number of Tables: 2

ISSN: 1662-4246 (Print)
eISSN: 1662-8063 (Online)

For additional information: https://www.karger.com/PHG


Copyright / Drug Dosage / Disclaimer

Copyright: All rights reserved. No part of this publication may be translated into other languages, reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, microcopying, or by any information storage and retrieval system, without permission in writing from the publisher.
Drug Dosage: The authors and the publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accord with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any changes in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new and/or infrequently employed drug.
Disclaimer: The statements, opinions and data contained in this publication are solely those of the individual authors and contributors and not of the publishers and the editor(s). The appearance of advertisements or/and product references in the publication is not a warranty, endorsement, or approval of the products or services advertised or of their effectiveness, quality or safety. The publisher and the editor(s) disclaim responsibility for any injury to persons or property resulting from any ideas, methods, instructions or products referred to in the content or advertisements.

References

  1. Haga SB, Burke W: Pharmacogenetic testing: not as simple as it seems. Genet Med 2008;10:391–395.
  2. The Royal Society: Personalised medicines: hopes and realities. London, The Royal Society, 2005.
  3. Ross CJ, Carleton B, Warn DG, Stenton SB, Rassekh SR, Hayden MR: Genotypic approaches to therapy in children: a national active surveillance network (GATC) to study the pharmacogenomics of severe adverse drug reactions in children. Ann N Y Acad Sci 2007;1110:177–192.
  4. Leeder JS, Kearns GL: The challenges of delivering pharmacogenomics into clinical pediatrics. Pharmacogenomics J 2002;2:141–143.
  5. Ross CJ, Katzov H, Carleton B, Hayden MR: Pharmacogenomics and its implications for autoimmune disease. J Autoimmun 2007;28:122–128.
  6. Freund CL, Clayton EW: Pharmacogenomics and children: meeting the ethical challenges. Am J Pharmacogenomics 2003;3:399–404.
  7. Diekema DS: Conducting ethical research in pediatrics: a brief historical overview and review of pediatric regulations. J Pediatr 2006;149:S3–S11.
  8. Impicciatore P, Choonara I, Clarkson A, Provasi D, Pandolfini C, Bonati M: Incidence of adverse drug reactions in paediatric in/out-patients: a systematic review and meta-analysis of prospective studies. Br J Clin Pharmacol 2001;52:77–83.
  9. Leeder JS: Developmental and pediatric pharmacogenomics. Pharmacogenomics 2003;4:331–341.
  10. Joly Y, Sillon G, Silverstein T, Krajinovic M, Avard D: Pharmacogenomics: don’t forget the children. Current Pharmacogenomics and Personalized Medicine 2008;6:77–84.
  11. Ernest TB, Elder DP, Martini LG, Roberts M, Ford JL: Developing paediatric medicines: identifying the needs and recognizing the challenges. J Pharm Pharmacol 2007;59:1043–1055.
  12. Knoppers BM, Avard D, Cardinal G, Glass KC: Science and society: children and incompetent adults in genetic research: consent and safeguards. Nat Rev Genet 2002;3:221–225.
  13. Beecher HK: Ethics and clinical research. N Engl J Med 1966;274:1354–1360.
  14. McIntosh N, Bates P, Brykczynska G, Dunstan G, Goldman A, Harvey D, Larcher V, McCrae D, McKinnon A, Patton M, Saunders J, Shelley P: Guidelines for the ethical conduct of medical research involving children. Royal College of Paediatrics, Child Health: Ethics Advisory Committee. Arch Dis Child 2000;82:177–182.
  15. American Society of Human Genetics, American College of Medical Genetics Board of Directors: Points to consider: ethical, legal, and psychosocial implications of genetic testing in children and adolescents. J Hum Genet 1995;57:1233–1241.
  16. Medical Research Council of Canada, Natural Sciences and Engineering Research Council of Canada, and Social Sciences and Humanities Research Council of Canada: Tri-Council Policy Statement – Ethical conduct for research involving humans. Canadian Institute of Health Research, 1998.
  17. Council for International Organization of Medical Sciences: International Ethical Guidelines for Biomedical Research Involving Human Subjects. Geneva, CIOMS, 2002.
  18. Friedman-Ross L: Children in Medical Research. Oxford, Clarendon Press, 2006.
  19. Knoppers BM, Joly Y, Simard J, Durocher F: The emergence of an ethical duty to disclose genetic research results: international perspectives. Eur J Hum Genet 2006;14:1170–1178.
  20. Avard D, Joly Y: Improved understanding of genetic and genomic influences on drug disposition and action: implications for children. Paediatr Drugs 2008;10:275–278.
  21. Ross LF: Genetic testing of adolescents: is it in their best interest? Arch Pediatr Adolesc Med 2000;154:850–852.
  22. Wilfond BS, Carpenter KJ: Incidental findings in pediatric research. J Law Med Ethics 2008;36:332–340.
  23. United Nations: The Convention on the Rights of the Child. Geneva, 1989.
  24. Patenaude AF, Senecal K, Avard D: Whither paediatric research and predisposition genetic testing? GenEdit 2006;4:1–9.
  25. Avard D, Knoppers BM: Ethical dimensions in paediatric neurology: a look into the future. Semin Pediatr Neurol 2002;9:53–61.
  26. Kaufman D, Geller G, Leroy L, Murphy J, Scott J, Hudson K: Ethical implications of including children in a large biobank for genetic-epidemiologic research: a qualitative study of public opinion. Am J Med Genet Part C Semin Med Genet 2008;148C:31–39.
  27. Fernandez CV, Santor D, Weijer C, Strahlendorf C, Moghrabi A, Pentz R, Gao J, Kodish E: The return of research results to participants: pilot questionnaire of adolescents and parents of children with cancer. Pediatr Blood Cancer 2006;332:188–189.
  28. Fernandez CV, Kodish E, Weijer C: Informing study participants of research results: an ethical imperative. IRB 2003;25:12–19.
  29. Joly Y, Knoppers BM: Pharmacogenomic data sample collection and storage: ethical issues and policy approaches. Pharmacogenomics 2006;7:219–226.
  30. Phillips MS, Joly Y, Silverstein T, Avard D: Consent in pharmacogenomic research. GenEdit 2007;5:1–9.
  31. Sillon G, Joly Y, Feldman S, Avard D: An ethical and legal overview of pharmacogenomics: perspective and issues. Medicine and Law (in press).
  32. Canadian Paediatric Society: Guidelines for genetic testing of healthy children. Paediatric Child Health 2003;8:42–45.
  33. Friedman-Ross L: Should we provide healthy children with greater protection in medical research? in: Children in Medical Research. Oxford, Clarendon Press, 2006, pp 76–86.
  34. Meaux JB, Bell PL: Balancing recruitment and protection: children as research subjects. Issues Compr Pediatr Nurs 2001;24:241–251.
  35. Knox CA, Burkhart PV: Issues related to children participating in clinical research. J Pediatr Nursing 2008;22:310–318.
    External Resources
  36. Budetti PP: Ensuring safe and effective medications for children. JAMA 2003;290:950–951.
  37. Steinbrook R: Testing medications in children. N Engl J Med 2002;347:1462–1470.
  38. Matsui D, Kwan C, Steer E, Rieder MJ: The trials and tribulations of doing drug research in children. CMAJ 2003;169:1033–1034.
  39. Burke W, Diekema DS: Ethical issues arising from the participation of children in genetic research. J Pediatr 2006;149:S34–S38.
  40. Hirtle M: Children and genetics: comparative study of international policy positions. Health Law J 1998;6:43–81.
    External Resources
  41. Caulfield T, Upshur RE, Daar A: DNA databanks and consent: a suggested policy option involving an authorization model. BMC Med Ethics 2003;4:E1.
  42. Moore CA, Khoury MJ, Bradley LA: From genetics to genomics: using gene-based medicine to prevent disease and promote health in children. Semin Perinatol 2005;29:135–143.
  43. Holm S: Informed consent and the biobanking of material for children. Genom Soc Policy 2005;1:16–26.
  44. Henrikson NB, Burke W, Veenstra DL: Ancillary risk information and pharmacogenetic tests: social and policy implications. Pharmacogenomics J 2008;8:85–89.
  45. Knoppers BM: Consent revisited: points to consider. Health Law Review 2005;13:33–38.
    External Resources
Figures

Tables
Thumbnail
Thumbnail