|Year : 2020 | Volume
| Issue : 1 | Page : 22-26
Vignette element analysis for automated generation of vignettes in pharmacy education
Carolyne Ma1, Robert Hubal1, Jacqueline E McLaughlin2
1 Division of Practice Advancement and Clinical Education, UNC Eshelman School of Pharmacy, Chapel Hill, NC, USA
2 Division of Practice Advancement and Clinical Education; Center for Innovative Pharmacy Education and Research, UNC Eshelman School of Pharmacy, Chapel Hill, NC, USA
|Date of Submission||13-Feb-2020|
|Date of Acceptance||17-Feb-2020|
|Date of Web Publication||13-Mar-2020|
Jacqueline E McLaughlin
Division of Practice Advancement and Clinical Education, UNC Eshelman School of Pharmacy, Chapel Hill, NC 27599; Center for Innovative Pharmacy Education and Research, UNC Eshelman School of Pharmacy, Chapel Hill, NC 27599
Source of Support: None, Conflict of Interest: None
Objectives: The aim of this study is to analyze and describe vignette elements and structure, and resulting difficulty, as a foundation for vignette construction in medical education, and ultimately automated vignette generation. Methods: Sixty-three vignettes representing a variety of disease states were sourced from the Objective Structured Clinical Experiences, published practice literature focused on pharmacy, and other training and assessment environments within the school. Three coders independently coded each vignette to identify underlying elements and structure. A consensus-building process was used by the coders to discuss and reconcile coding differences. Results: The coding process resulted in 36 vignette elements. The most common elements were age (n = 59, 93.6%), gender (n = 57, 90.5%), and medications (n = 54, 85.7%); others included race, medications, and chief complaint. Vignette structures and wording were found to be highly variable, with elements present in different magnitudes (range: 4–18 elements), being used with different descriptors, and given in different sequences. Conclusions: Vignette construction could benefit from further understanding of vignette structures and wording and their influence on the level of difficulty. This undertaking will allow educators to construct better vignettes for teaching and assessment to ensure that student performance accurately represents student knowledge and skills, rather than construct irrelevant variance due to vignette-level inconsistencies in content or structure. The defined elements and structure will also enable a systematic generation of vignettes for further consistency in teaching and assessment.
Keywords: Assessment, case analysis, case-based learning, pharmacy, vignette analysis
|How to cite this article:|
Ma C, Hubal R, McLaughlin JE. Vignette element analysis for automated generation of vignettes in pharmacy education. Educ Health Prof 2020;3:22-6
|How to cite this URL:|
Ma C, Hubal R, McLaughlin JE. Vignette element analysis for automated generation of vignettes in pharmacy education. Educ Health Prof [serial online] 2020 [cited 2020 May 27];3:22-6. Available from: http://www.ehpjournal.com/text.asp?2020/3/1/22/280541
| Introduction|| |
In healthcare education, vignettes are often used in teaching and learning to prepare students for clinical practice through realistic clinical situations. Characteristically, vignettes are realistic, engaging, able to stimulate the integration of knowledge across disciplines, challenging, set in a context representing the students' future careers, able to address preset learning objectives, logical in flow, and student-centered in design. Further, with the shifting roles of healthcare providers across a variety of settings, educator guidance on constructing clinical vignettes has become increasingly critical to students learning. This is particularly true in pharmacy, where the clinical role of the pharmacist has expanded significantly in recent years, but also reflects the increasingly interprofessional nature of healthcare.
To meet such extensive requirements, preparing clinical vignettes for instruction is often labor- and time-intensive and is reliant on the purpose for which the vignette is used and the course content to which it is linked. Yet, while these vignettes have become an essential component to developing clinical reasoning and decision-making skills in the health professions curricula, and there is a need for vignette construction to follow specific guidelines, educators are not trained on how to construct them. A comprehensive literature search revealed a significant underrepresentation of research regarding the construction of clinical vignettes versus other forms of assessment, such as multiple-choice questions in pharmacy education. In medical education, the National Board of Medical Examiners (NBMEs) has created a handbook for constructing vignettes, Constructing Written Test Questions for the Basic and Clinical Sciences, yet such a framework does not exist in nursing or pharmacy education, and even the NBME guide recommends but does not enforce content or structural integrity. The objective of this study is to utilize a systematic approach for dissecting vignette structure, that is, the presence and frequency of elements and element ordering and how they may vary across subject matter, to serve as a framework for providing standardized guidance on clinical vignette infrastructure in medical education.
| Methods|| |
This study consisted of a descriptive analysis examining elements as they appear in vignettes in pharmacy education and the association between elements and vignette typology. Sixty-three vignettes were sourced from the Objective Structured Clinical Experiences, standardized patient experiences, and other training and assessment environments within the University of North Carolina Eshelman School of Pharmacy (School) and from published pharmacy practice literature.
| Qualitative Methods|| |
Thirty-six vignettes representing a variety of disease states were selected for initial coding. Using qualitative methods, vignettes were manually coded and analyzed for common element patterns. To enhance the reliability and validity of the data, three team members independently reviewed and coded the vignettes. Disagreement in codes was addressed through a consensus-building process, wherein the three coders convened to discuss and reconcile vignette elements. The remaining 27 vignettes were then coded using the framework. After all 63 vignettes were coded and results were again reviewed for consistency to ensure interrater reliability of 80%.
Following vignette coding, vignettes were categorized into mutually exclusive pharmacy topic disciplines (Mechanism of Action, Pharmacology, and Pharmacotherapy) that were identified from NBME's guide. Pharmacotherapy vignettes were identified as situations that assessed students' knowledge of the use of pharmaceuticals to treat disease, whereas Pharmacology vignettes were identified as situations that were designed to assess students' knowledge of the drug action in the human body (i.e., absorption, distribution, metabolism, and excretion). Mechanism of action refers to vignettes that describe biochemical interaction at a molecular level between a drug and target. Vignettes were categorized first by two individuals independently. Vignettes that did not fall into any of the three categories were excluded from further analyses.
Further analyses were intended to determine: (i) the most common elements appearing in the vignettes, (ii) the most common elements appearing in vignettes by category, (iii) differences in the proportion of elements present in each category, and (iv) the degree to which these vignettes complied with the writing stem proposed by the NBME. For part (iv), compliance to the NBME standard was assessed by taking the sum of the absolute distance of the position of an element of a given vignette from the position of an element in a standard vignette. For example, vignette elements that follow the “standard” order – age | sex | chief complaint | past medical history | family history | physical examinations | laboratories – were assigned positions 1 through 7. If given a vignette contained elements in a different order – for example, age | sex | chief complaint | physical examinations | past medical history | family history | laboratories) – then distance in this instance was calculated as 0 + 0 + 0+ |−2| + |1| + |1| +0.
A Chi-square analysis using the Fisher's test, where appropriate, was used to determine differences in the proportion of vignette elements between Mechanism of Action and Pharmacotherapy, Pharmacotherapy and Pharmacology, and Mechanism of Action and Pharmacology. Any results with P < 0.05 were considered statistically significant.
| Results|| |
Vignettes were categorized by gender, age group, and ethnicity. Most commonly, vignettes described 36–65 years old (n = 29, 46.0%), Caucasian (n = 13, 26.0%), females (n = 29, 46.0%) on common disease state topics that students must learn to be prepared to provide collaborative patient-centered care upon graduation and licensure. Most vignettes were sourced from courses taught within the school and from trade journals (e.g., Pharmacy Times) [Table 1].
As a proxy for difficulty, vignettes were coded into disease state tiers, values ranging from 1 to 3 that are assigned by the 2016 American College of Clinical Pharmacy Educational Affairs Committee that reflect levels of training pharmacy students typically receive in their didactic curriculum.
Vignette element frequency
The coding process resulted in 36 vignette elements, including race, medications, and chief complaint. The most common elements were age (n = 58, 92.1%), gender (n = 56, 88.9%), and patient identifiers (n = 48, 76.2%) [Table 2]. Vignette structures and wording were highly variable, with elements presented in different magnitudes (range 4–18 elements), being used with different descriptors, and given in different sequences.
Comparison of vignette element frequency by category
Differences were found between categories for the following vignette elements: setting, patient-reported symptoms, provider reasoning for diagnosis, laboratories/vitals/imaging/calculations, and vignette question hint [Table 3]. Between Mechanism of Action and Pharmacotherapy vignettes, patient-reported symptoms were more frequent in Pharmacotherapy than Mechanism of Action vignettes (P = 0.02), and vignette question hints were found to be more frequent in Mechanism of Action vignettes as opposed to Pharmacotherapy vignettes (P = 0.003) [Table 3]. Between Pharmacotherapy and Pharmacology vignettes, setting and patient-reported symptoms were found to be more frequent in Pharmacotherapy vignettes (psetting= 0.037 and psymptoms= 0.002), and laboratories/vitals/imaging/calculations were found to be more frequent in Pharmacology vignettes (P < 0.001) [Table 3]. Between Mechanism of Action and Pharmacology vignettes, name and laboratories/vitals/imaging/calculations were found to be more frequent in Pharmacology vignettes (pname= 0.03 and plab< 0.001), and vignette question hint were found to be more frequent in Mechanism of Action vignettes (P = 0.04) [Table 3].
Comparison of vignette element ordering to the National Board of Medical Examiner standard
Distance calculations ranged from 0 to 15. Most vignettes had distances ranging from 0 to 5 (n = 59, 93.7%). Fewer vignettes had distances ranging from 6 to 10 (n = 2, 3.2%) and 11–15 (n = 2, 3.2%). On an average, age and gender elements had the lowest order variation (average distance = 0.034 and 0.036, respectively), indicating that these elements most commonly followed the element sequencing proposed by NBME. Family history and history, when present, were found to have the highest order variation (average distance = 1 and 0.915, respectively), although in effect these are relatively short distances from the “standard.”
| Discussion|| |
Clinical vignettes have long been utilized as a key teaching and assessment tool in medical education. However, instructors are often not provided formal guidance on how to construct these vignettes and research is notably lacking in this area compared to other assessment methods, such as multiple-choice tests. Even in other healthcare disciplines, such as medicine, guidance is available yet sparse.
The first step toward informing vignette construction in medical education was to deconstruct the vignette structure and determine whether systemic patterns exist in vignettes. An exploratory analysis of vignettes using qualitative and quantitative methods resulted in a codebook of patterns and a characterization of the most frequent elements (age, gender, chief complaint, etc.) of a typical vignette used in pharmacy education assessments. The next step was to analyze the ordering of these elements against the only existing published guidance (an NBME document focused on constructing vignettes for medical examination) providing a standard template for vignettes. Although NBME's proposed template is designed for assessing medical students, a majority of pharmacy vignettes matched closely, suggesting the applicability of a structured approach to vignette construction in pharmacy education. Yet, there are notable differences. For instance, history was a vignette element found to have one of the highest order variations. This finding may be due to the broadness of the “past history” definition for pharmacy education purposes – the interpretation used in this study included past medical history, medications, as well as the history of present illness. Separating “past history” into three categories and placing them in different positions within the template might influence match outcomes.
In addition to suggesting a standard template for vignette construction, NBME also proposes categories for the vignettes, several of which (in particular, Mechanism of Action, Pharmacology, and Pharmacotherapy) are appropriate for pharmacy. This study found statistical differences with four elements among the three categories: name, race/ethnicity, patient-reported symptoms, laboratories/vitals/imaging/calculations, and extra hint/information given in the vignette to help the student answer the vignette question. Possible factors that contribute to these differences include the stage in the pharmacy student's career that the discipline is taught and the relevance of the vignette element to the discipline. For example, the extra hint/information was found to be significantly higher in the Mechanism of Action vignettes as compared to the Pharmacotherapy vignettes. Vignette content that involves basic sciences and molecular sciences, which cover drugs' Mechanisms of Actions, is often taught early on in a pharmacy students' career, and students at this level in their career may require more of such extra hints to complete the assignment associated with the vignette. Similarly, results such as patient-reported symptoms appearing more frequently in Pharmacotherapy, relative to Mechanism of Action and Pharmacology, and laboratories/imaging/calculations appearing more frequently in Pharmacology, relative to Mechanism of Action and Pharmacotherapy, may be due to the elements' relevance to the vignette content.
Future steps that relate to vignette generation are of particular interest. A motivator for this work was to identify common and needed patterns that underlie medical, educational vignettes. Vignettes' content and structure are important but only part of the story. For instance, there are rules for vignette generation, such as how certain procedures (e.g., laboratory draws) almost never occur in the community pharmacy setting, or how certain descriptors (e.g., pregnancy) cannot apply to some individuals (males, children, and the elderly). One direction of follow-on work will be to apply learning techniques to a larger set of vignettes to establish further rules. Similarly, the level of difficulty of a vignette can play a role in when or how it is used within the coursework. A second direction for further work is to derive measurements for the difficulty or complexity of vignettes and determine how they fit with educational practices. Furthermore, the current categorization of vignettes – into Mechanism of Action, Pharmacology, and Pharmacotherapy – is meaningful to the pharmacy but not all-encompassing of the topics that pharmacy students encounter. A third direction, then, for future work is to further investigate categories for the vignettes. Patient care, and its associated behavioral skills, is one potential expansion to the current categorization.
This study was exploratory in nature and has some limitations. First, a relatively small subset of vignettes was analyzed. However, the formalization of processes used in the qualitative and quantitative analysis suggests findings are valid and will apply to any additional vignettes. Second, it is possible that some elements and structures were not present in the coded vignettes. To accommodate any currently unobserved elements, the framework is readily adaptable. Third, the NBME guidance for categorization and sequencing may be limited due to its specific focus on medical vignettes. The absence of a similar framework for pharmacy highlights a need for this study and may also suggests it might require modifications to the framework with new patterns found in additional pharmacy vignettes, as well as those for other domains such as nursing.
| Conclusions|| |
This study introduces mechanisms for both quantitatively and qualitatively analyzing vignettes used in medical education and utilizes these methods to deconstruct vignette structure and describe essential vignette characteristics. The use of these systematic and rigorous approaches will serve as the foundation for further research into vignette construction and analysis (e.g., analyzing vignettes that assess behavioral skills, comparing vignettes at different levels of difficulty) and is currently being used to inform the development of other applications, such as a software program that auto-generates vignettes on demand.
The authors would like to thank Tazim Kabir and Laura Bobbitt for their input in building the codebook and coding the cases. Authors would like to thank the Eshelman Institute for Innovation for providing financial support for this project.
Financial support and sponsorship
This work was funded, in part, by the Eshelman Institute for Innovation.
Conflicts of interest
There are no conflicts of interest.
| References|| |
McLean SF. Case-based learning and its application in medical and healthcare fields: A review of worldwide literature. J Med Educ Curric Dev 2016;3. pii: JMECD.S20377.
Mostert MP. Challenges of case-based teaching. Behav Anal Today 2007;8:434-42.
Blouin RA, Adams ML. The role of the pharmacist in health care: Expanding and evolving. N
C Med J 2017;78:165-7.
Brown KP, Salerno G, Poindexter L, Trotta K. The Evolving role of the pharmacist in interprofessional practice. N
C Med J 2019;80:178-81.
Eisenhardt A, Ninassi SB, Bruno S. The use of simulation and cases to teach real world decision making: Applied example for health care management graduate programs. Learn High Educ 2016;12:71-5.
Waxman KT. The development of evidence-based clinical simulation scenarios: Guidelines for nurse educators. J Nurs Educ 2010;49:29-35.
Dell KA, Wantuch GA. How-to-guide for writing multiple choice questions for the pharmacy instructor. Curr Pharm Teach Learn 2017;9:137-44.
Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences. 3rd
ed. Philadelphia, PA: National Board of Medical Examiners; 2003.
Schwinghammer TL, Crannage AJ, Boyce EG, Bradley B, Christensen A, Dunnenberger HM, et al
. 2016 ACCP Pharmacotherapy Didactic Curriculum Toolkit. Lenexa, KS: American College of Clinical Pharmacy; 2016. Available from: www.accp.com/docs/positions/misc/Toolkit_final.pdf. [Last accessed on 2019 May 11].
[Table 1], [Table 2], [Table 3]