THE FEASIBILITY AND EFFECTIVENESS OF A PILOT
RESIDENT-ORGANIZED & -LED KNOWLEDGE-BASED REVIEW
Victor J. Vautrot, M.D.
Department of Psychiatry Boston VA Healthcare
Department of Psychiatry, Harvard Medical School
Objective: The ACGME (Accreditation Council for Graduate Medical Education) requires a sufficient medical knowledge base as one of the six core competencies in residency training. We judged that an annual “short-course” review of medical knowledge would be a useful adjunct to standard seminar and rotation teaching, and that a resident-designed course might more closely meet resident-identified needs and learning styles.
Methods: Our residency training program designed a formal summer short-course, called the Knowledge Base Review (KBR), with the hypothesis that the course would improve general knowledge as measured by the Psychiatry Resident In Training Examination (PRITE®) and with the objective that this program would enrich the residents’ general knowledge base in psychiatry and neurology. We designed the KBR as a weekly curriculum composed of nine, two-hour sessions: each equally didactic- and active learning-based. The first hour spotlighted one of the subscales of the PRITE®. The second hour used a question and answer test where four teams of residents competed to answer in a game show style manner. Qualitative and quantitative assessment of the KBR was conducted.
Results: Attendance among the PGY-II, III, and IV classes was 82%, 73%, and 94%, respectively. Participating residents completed a survey midway through the KBR. 95% responded that the course was meeting their own expectations “All” and “Most of the Time.” 100% responded that the didactic component was useful, and 94% felt that the game show component was useful. Among 23 residents who took the PRITE® in the year prior to and the year after the KBR, there was a 2.6% increase in PRITE® global Psychiatry scores (p=0.15) in contrast to a 9.0% decline in global Neurology scores (p=0.001), which was not addressed in the KBR.
Conclusion: The broad participation and acceptability of the course, and the performance difference in PRITE® scores between the psychiatry topics, the majority of which were reviewed, and neurology, which was not reviewed, suggests the potential for such a resident-organized and –led intervention to impact acquisition of medical knowledge through an enjoyable and effective approach.
Key Words: Teaching by Psychiatric Residents, Curriculum Development, Boards: ABPN, Residents General, Teaching Methods
Introduction:
One of the goals of residency education is to prepare physicians for the independent practice of clinical medicine. This schooling is delivered through the combination of didactics and clinical experience. Psychiatry residency education requires its residents to obtain competence in acquiring Medical Knowledge, which is one of the six core competencies outlined by the ACGME (1). The most effective methods of providing resident instruction are not well established. Our residency training program is interested in developing a formal curriculum that enriches residents’ general knowledge in psychiatry, guided by the annual PRITE® examination. This standardized test was found to be a “moderate predictor” of performance on the American Board of Psychiatry and Neurology (ABPN) Part I examination in psychiatry (2). Webb et al. reported that the national correlation between PRITE® scores and subsequent performance on the American Board of Psychiatry and Neurology Part 1 written examination is 0.67 (p<0.01)
Methods:
We designed a once-a-week course, called the Knowledge Base Review (KBR), composed of nine sessions that led-up to the fall PRITE®; participation was mandatory. The American College of Psychiatrists designed PRITE® as an educational resource for psychiatric residents and training programs. Each section of the exam focuses on a particular component of psychiatry, offering references to support and explain correct answers. Residents receive a detailed computer analysis of their test performance in comparison with other residents at a similar level of training. Training directors receive results for their individual residents as well as statistical summary data comparing their training program with other groups of participants. Created in 1978, the PRITE® consists of 300 questions and is administered in two parts over two days; all post-graduate year (PGY) residents take the exact same examination together. To protect confidentiality, scores are returned to the program director, who then gives the test results individually to and privately back to each resident. The content areas covered in the PRITE® are: Neurology and Neurosciences, Growth and Development , Adult Psychopathology, Emergency Psychiatry, Behavioral Science and Social Psychiatry, Psychosocial Therapies, Patient Evaluation and Treatment Selection, Consultation-Liaison Psychiatry, Somatic Treatment Methods, Child Psychiatry, Alcoholism and Substance Abuse, Geriatric Psychiatry, Forensic Psychiatry, and Miscellaneous. There are 14 areas that are scored on the PRITE®. One area is for Neurology, and 13 areas are for Psychiatry (12 of these are topic-specific and one of them is a “Miscellaneous” section). Scores are reported for all 14 areas, but a Global Psychiatry score is also included.
Each session was two hours-long and was equally didactic- and active learning-based. The first hour spotlighted one of the 12 subscales of the PRITE with a PowerPoint lecture that was both prepared and presented by a senior PGY-IV psychiatry resident from a variety of sources—psychiatry textbooks, psychiatry board review books, and even past PRITE® exam explanations to questions. Lecture materials were also provided as hard copies to each of the residents. Interactive residency group discussion was encouraged during this first, one hour peer-presented didactic. Since the time frame available for the course was limited to 9 weeks, the KBR focused on 9 of the 12 PRITE® subscales that could show room for improvement based upon results from the previous year in 2007. The three subscales of psychiatry that were not covered in the curriculum were Somatic Treatment Methods, Child Psychiatry, and Alcoholism and Substance Abuse because these scores were shown to be areas of strength for our training program. Moreover, Neurology did not have a one-hour lecture devoted to it for coverage in the 1st hour. The innovativeness of this first hour is that unlike other published residency in-training exam review course initiatives, ours was not based upon using quizzes or relying on residents to study from textbooks.
The second hour, on the other hand, used a video-projected question and answer test that paralleled the same format, balance, and subject content of the Psychiatry written board exams, and it included Neurology (7). This second element of innovation is that the game show format was video-projected upon a very large canvas screen that encompassed one wall of the room using a laptop computer connected to a video projector. Residents were divided into four teams--comprised of balanced, mixed levels of training. Each team, of about 7 residents apiece, chose a team name and actively competed in a game show style manner which resembled Family Feud & Jeopardy. Nevertheless, during the second hour of the KBR, there were some questions that randomly appeared and were video-projected that were neurology-based (i.e. a question about depression in Multiple Sclerosis or Alzheimer’s Dementia). After a question was video projected, each team convened and selected their answer choice collectively. A third element of novelty is that an explanation of the correct response appeared upon the large canvas screen, and then there was a group residency discussion about the topic, which also expanded more knowledge in that topic. Points accumulated for correct answers and were deducted for incorrect ones. Scores were kept from one session to the next by the residents as the teams competed against one another over the course of the 9 weeks.
Mid-cycle into the KBR course, we collected evaluation forms from all of the residents who participated. We were interested to know if the course was meeting its own objectives, meeting the residents’ expectations, if it was well presented, and if the didactic and game show hours were useful components.
Moreover, data from the year prior to the KBR (2007) were compared to data post-KBR (2008) for 23 residents who took the PRITE® in both years. Because of the possibility of year-to-year variability in changes in scores due to non-specific factors, we analyzed both the Global Psychiatry scores (which were addressed by the KBR curriculum) and Global Neurology scores (which were not addressed in the KBR). Our primary hypothesis was that the Psychiatry Global scores would increase significantly from 2007 to 2008, while we also secondarily analyzed Neurology Global scores to identify any possible secular drift in the scores between 2007 and 2008.
Quantitative data analysis utilizing the residents who took the PRITE both the year before and the year after the KBR (n=23 pairs of scores) began by examining the distributions of raw scores on PRITE global subtotals. Global Subtotals are the Global Psychiatry and Global Neurology scores. Analyses were conducted using SPSS. Since data distributions appeared normal, parametric statistics were used including a one-tailed paired t-test for Psychiatry Global Scores (predicted change was upward since the KBR addressed some but not all subscales) and a two-tailed t-test for Neurology Global Scores (no prior directional hypothesis since the KBR did not address this content). Data were de-identified by the Program Director prior to analysis and are reported on percentage change rather than raw scores to avoid the raw scores being taken out of context. This project was undertaken as a quality improvement, rather than a research effort, and so informed consent from individual residents was not obtained. This project was approved by the Institutional Review Board.
Results:
Participation and Acceptability:
Attendance for the course was required, and participation was high. Participation could not be 100%, however, due to scheduled vacations, sick leave, paternity leave, maternity leave, and post-call next day required time-off (an ACGME requirement for work duty hours). Our resident survey indicated good to excellent participation and acceptability of the KBR course. Attendance during the nine weeks among the PGY-II, III, and IV classes was 82%, 73%, and 94%, respectively. Eighteen out of twenty-eight residents (64%) completed a survey midway through the course. 61% of the residents felt that the course met its own objectives “All of the Time”, while 95% responded that the KBR course was meeting their own expectations “All” and “Most of the Time.” 72% of the residents “Strongly Agreed” that the course was presented in an easy-to-follow and understand manner. 100% responded that the didactic component was useful, and 94% felt that the game show component was useful (Table 1).
Impact on Medical Knowledge:
Our primary hypothesis is that global Psychiatry scores would increase significantly between 2007 (pre-KBR) and 2008 (KBR) for residents who took the PRITE® at both time points. We analyzed the Global Neurology scores to account for year-to-year variability due to non-specific (i.e. non-KBR) factors, since Neurology was not included in the KBR curriculum. Statistical analysis revealed that there was a non-significant 2.6% increase in Global Psychiatry scores (paired t(22)=1.5; p=0.15) in contrast to a 9.0% decline in global Neurology scores (paired t(22)=3.9; p=0.001). In residents tested in both years, individual Psychiatry subscales, some of which were covered by the KBR and others not, did not consistently change in one direction or the other (data available upon request).
Discussion:
This type of focused, resident-developed program was well accepted by the residents. An educational intervention with a team-oriented gaming format that emphasized resident performance in front of peers may be useful to residency programs with a strong desire to impact resident study and learning of the basic and clinical sciences. For example, a game show format was as effective as standard lectures in teaching medical students about ectopic pregnancy, but rated higher in stimulating faculty/student interaction, helping retain information, and in overall enjoyment (6).
While the KBR course was not designed specifically to address PRITE® items, and it did not cover all Psychiatry subscales, we considered that this standardized, national test of medical knowledge would be a reasonable tool with which to gauge impact of the course. Quantitative data suggest some impact of the KBR, since there appeared to be an improvement in Global Psychiatry despite a concomitant decline in the Global Neurology scores which were not addressed by the KBR didactic sessions. It is possible, though unlikely, that the KBR off-set a deteriorating program-wide trend in teaching between 2007-2008, or that the KBR had no significant effect while the neurology teaching deteriorated. It is also possible that the decline in neurology scores were a function of year-to-year variation that may have also impacted psychiatry scores. These possibilities cannot be differentiated without following trends over multiple years. Interesting, in 2008 the program placed in the top-ten programs nationally in the American Psychiatry Association’s Mind Games competition, which tests a similar breadth of knowledge. Whether the KBR course contributed to this achievement is also unclear.
Nonetheless, the KBR format was feasible to implement with minimal resources and was well accepted by the residents. This pilot experience could be developed further to achieve more impact on the outcome variable of interest, PRITE scores. For instance, a more intensive or broader course may have resulted in more robust improvement in scores, and this is a consideration for future iterations of the course. For example, if the KBR curriculum was longer in duration to encompass coverage of all 12subscales, then there may have been more robust and significant global score results. The Schuh intervention did demonstrate a significant change in their neurology residency program’s scores; it is noteworthy that their course was “20-22 weeks long,” while ours was 9 weeks. It would also be possible to try to improve PRITE® scores by linking the KBR course content more closely to the PRITE® items. One possibility would be solely to review and teach from previous PRITE® exams for the full two hours--i.e. teach for the test. However, this was not the intent of the course, which was to enrich residents’ general knowledge in psychiatry, as guided by the annual PRITE® examination.
We asked for feedback and suggestions from the residents for areas for future improvements in the KBR course. The responses included using old PRITE® exams as a pre- and post- curriculum measure to gauge areas that improved and still need improvement; and, it was requested to incorporate clinical case vignette discussions into the didactic lecture.
Conclusion:
In summary, the broad participation and acceptability of the KBR course demonstrates the potential for such a resident-organized and –led intervention to impact acquisition of medical knowledge through an enjoyable and effective approach.
