In order to uphold our mission to elevate the quality of orthodontics through certification, education and professional collaboration, the American Board of Orthodontics (ABO) has a responsibility to the specialty and general public to certify orthodontists in a fair, reliable and valid manner. This requires continual evaluation and review of our certification process to ensure we offer an examination that best evaluates orthodontists to ensure they are capable of providing the highest level of patient care.
Ultimately, the decision to move to a scenario-based examination was based on extensive research, evaluations, board reviews and input from educators.
How The Decision Was Made
Process and Key Findings
The ABO collaborated with leaders in the industry, surveyed our own constituents and worked closely with a certification and licensure testing company with expertise in this area. We also elected to research and observe other dental and medical specialty boards to ensure we are utilizing best practices.
In an effort to collaborate with educators, the ABO hosts regular Educator's Symposiums. Based on discussions in 2016, it was clear there were challenges with the current exam structure based on the changing environment of our specialty.
The ABO agreed to explore ways to address four specific needs:
- Strengthening the specialty in the current environment
- Creating a more fair, reliable and valid examination
- Eliminating barriers to certification
- Leveling the playing field for residents and current orthodontists
This initiated more than a year of extensive research, evaluations, surveys and board reviews.
The ABO conducted a certification survey for AAO members to obtain input regarding the effectiveness of the current certification process and requirements. Respondents included both certified and non-certified orthodontists.
Key findings include:
- Over 60% believe ABO Certification is a designation all orthodontists graduating from a CODA-accredited specialty program should strive to obtain
- Respondents identified unwarranted barriers in the current certification process including length of residency programs, an increase of corporate and DSO work environments and difficulty in meeting case requirements in timely manner
- Biggest obstacles in the certification process are the time commitment and patient case requirements
- Many believe more board certified orthodontists will help strengthen and unify the specialty.
- Overwhelming majority of respondents indicated they are looking for a change in the certification process
The ABO spent two days observing the ABE Clinical Examination, which consists of three components:
- Written Examination
- Oral Examination with 3 case-based scenarios
- Case History Examination requiring submission of 10 cases
The ABO benefited from observing the Oral Examination where examinees are asked questions from nine identified areas for assessment, however their participation level in the certification process also provided valuable insights:
- Over 90% of eligible endodontists complete the Written Examination and Oral Examination, but do not submit case requirements.
- Only 24% of the American Association of Endodontists members are certified and the primary reason for the significant difference is the case requirements.
The ABO spent two days observing the RCDC Orthodontic Clinical Examination, which utilizes a two-part exam design. The first part is similar to the ABO written examination and the second is a scenario-based oral clinical examination. The RCDC has been utilizing a scenario-based exam design for the last 10 years. Their scripted exam is given over a two-day period. The ABO clearly identified key benefits to this examination process:
- Vast subject area covered for a comprehensive examination
- Highly organized and professional process
- Ability to successfully test for clinical proficiency through scenario-based testing, without case submissions
The ABO has had several meetings with The American Board of Pediatric Dentistry (ABPD) to understand their challenges and advantages of transitioning to a scenario-based exam. Their previous exam consisted of:
- Written exam
- 1 hour oral examination
- Case submission
- Site/office visits
Ultimately, their exploration and research during this time of change resulted in the following:
- They were amazed at what other boards were able to test through scenario-based testing
- They concluded they were restricting their profession and needed to eliminate artificial stumbling blocks
- They had an opportunity to improve their testing process and therefore chose to proceed with the change
The ABPD strongly supports their decision to change to a scenario-based examination as they feel that the new structure is a non-restrictive objective testing process. Since moving to a scenario-based test design, the percentage of board certified pediatric dentists has increased from less than 20% to 75% of the parent association – in less than 10 years.
The ABO also met with other leaders in the industry at the Dental Specialties meeting, including the American Board of Oral and Maxillofacial Surgery (ABOMS). The ABOMS moved from a case-based examination to a scenario-based examination over a decade ago and they continue to review their scenario-based exam to better test for critical thinking skills through less fact-based questioning.
The consistent learning from various specialty boards was the similar challenges faced with case-based examinations and the need to adapt to better validate their certification processes. Many boards now have an increased reliance on scenario-based examinations to accurately test clinical proficiency and critical thinking skills.
The ABO partnered with Castle Worldwide, a certification and licensure testing company with 30 years of experience in the science of psychometrics and testing development.
Castle facilitated a Practice Analysis Study to provide the basis for demonstrating the content validity of a certification process and ensure there is close alignment between the content and scoring of the clinical examination and the practice of newly certified orthodontists. This was then followed by a Validation survey to confirm the domains and skills identified in the Practice Analysis.
- Began by defining core proficiencies required of a successful orthodontist
- Utilized the CODA Curriculum Standards and the AAO Clinical Practice Guidelines for Orthodontics and Dentofacial Orthopedics as a foundation for defining key skills and abilities
- Identified the content domains (or categories) and proficiencies that should be covered in the ABO Clinical Examination
- The domains and proficiencies identified were then validated by key stakeholders through a survey to board certified orthodontists, members of AAO, residents and educators
- The Practice Analysis and Validation study resulted in the recommendation to move towards a scenario-based clinical examination – in accordance with the psychometric and test development principles as an effective assessment of the cognitive skills required by orthodontists
In 2017, the ABO Board presented all of the research, survey findings, board reviews, Practice Analysis results and key insights gathered directing the board to make a change to the Clinical Examination design.
Educators provided their feedback, recommendations and questions regarding a possible change. The majority of meeting attendees responded in support of a change to a Scenario-based Oral Clinical Examination. They also participated in a collaborative workshop to develop possible scenario-based questions.