1. Why has the ABO decided that NOW is the time to change to a Scenario-based Oral Clinical Examination?
The decision to change the examination design did not come lightly and is based on years of rigorous evaluation and extensive research that stemmed from discussions and challenges faced within our specialty.
Since its inception in 1929, the ABO has always stayed abreast of different testing methodologies based on the trends of other dental and medical specialties. As the practice of orthodontics continues to develop, we believe the time to change is now in order to do what is best for the future of our specialty. The changes being made are necessary in order to produce an objective examination that tests the knowledge, skills, and critical thinking abilities that are required of an orthodontist in a fair and valid manner ensuring that each certified orthodontist has developed the expertise and the ability to provide the highest level of patient care.
2. Case submission has always been a core part of the Clinical Examination. Why have case submission requirements been eliminated?
The ABO considered a more transitional approach, with fewer case requirements combined with scenario-based examinations. However, based on our research and feedback from educators, we believe a full scenario-based format is the best approach to ensure we adhere to our mission to elevate the quality of orthodontics through certification, education, and professional collaboration.
We recognize the significance of this change and realize that case submission has always been part of the ABO process. However, we feel that we now have the opportunity to improve this process. This change eliminates unnecessary barriers for residents and orthodontists that are interested in participating in the certification process while allowing the ABO to accurately test all areas of the four domains established in the Practice Analysis Study as well as critical thinking skills.
We believe the scenario-based examination will allow us to more objectively test for proficiency and clinical expertise. The tools previously created by the ABO to assess case outcomes will be utilized within the scenario-based examination process, including the discrepancy index (DI), cast radiograph evaluation (CRE), case management form (CMF) and cephalometric superimposition technique and interpretation. We believe this examination will more effectively and reliably test critical thinking skills in all aspects of orthodontics.
3. How will the Clinical Examination be able to assess clinical competency without presenting cases?
Under the new scenario-based testing format, the ABO will be able to make a direct assessment of candidates’ ability to formulate diagnoses, develop treatment plans, assess implementation of treatment plans, and evaluate outcomes. The test focuses on the cognitive skills and decision-making process that are critical for the proficient practice of orthodontics. Hands-on skills assessment will remain within the residency programs, where it can be done most effectively.
4. Will scenario-based testing make the Clinical Examination easier?
No, scenario-based testing is not intended to be an easier examination design. The scenario-based examination will require hours of preparation and continued focus on outcomes. The new examination design will require an orthodontist to be proficient in four key domain areas: data gathering and diagnosis, treatment objectives and planning, treatment implementation and management, and critical analysis and outcomes assessment. This allows the examination to fully evaluate an orthodontist on all aspects of treatment in a standardized manner.
Orthodontists must demonstrate knowledge and skills as applied to specific cases selected for the scenario-based format. The examination will still be testing the same skills and outcomes required in individual case submission, but will be testing that knowledge in a more reliable, objective manner – as well as very specific critical thinking skills.
5. What are the biggest benefits of the new Scenario-based Examination design?
There are numerous benefits to the new examination design. One of the most important is that all orthodontists now have the ability to go through the certification process regardless of their practice environment and where they are within their career. The challenge of finding the cases with specific criteria and submission requirements is no longer an issue. Additionally, all orthodontists taking the examination will be objectively tested on their knowledge, skills, and critical thinking abilities. Ultimately, this will benefit the general public to ensure the ABO has certified orthodontists who are capable of providing the highest level of patient care.
6. How will these changes have an effect on orthodontic residency programs?
The ABO will continue to collaborate and work with educators to ensure residents are prepared for the certification examination. We encourage program leaders to continue to focus on clinical self-evaluation and outcomes assessments, as the scenario-based examination will test for knowledge and critical thinking skills that apply to these areas.
The ABO will continue to use assessment instruments including discrepancy index (DI), cast radiograph evaluation (CRE), case management form (CMF) within the scenario-based examination. If a resident is not proficient in outcomes assessments utilizing these instruments, or is not able to interpret the results of superimpositions, he or she will likely not perform well on this portion of the examination.
The ABO plans to continue to meet with educators regularly to gain feedback on how we can best support the orthodontic programs so that the scenario-based Clinical Examination is complementing the curriculum the educators are teaching in their programs. Many programs have expressed that they will continue to hold mock board exams and advocacy visits and will begin putting a larger focus on critical thinking skills. The ABO will also provide detailed examination results to the educators to let them know where their resident’s strengths and weaknesses rank on the scenario-based Oral Clinical Examination.
7. Does the scenario-based examination open up an increased opportunity for unethical behavior?
No, the ABO believes that the opportunity for unethical behavior will decrease as a result of the new examination design. Case submission requirements involved a higher risk of unethical behavior with misrepresentation of cases and the possibility of technological alterations. The scenario-based examination will eliminate these issues, as we will ensure that the cases and problems remain secure and are rotated or replaced for all examinations.
8. What are the options for those who still have “banked” cases?
At the conclusion of the February 2019 examination, Examinees who have fewer than 3 cases banked will be required to sign up for the new Scenario-based Oral Examination.
Examinees who have 3 or more cases “banked” have two options to complete the board certification process. Option one requires the examinee to mail-in their remaining cases for review until they have completed a total of 6 (original required categories) cases complete. Option two requires the examinee to complete the number of online Board Case Exams (BCEs) to match the remaining number of Incomplete/Unsubmitted cases until a total of 6 is met.
In addition, examinees with Incomplete BCOE scores are required to take the BCOE Web-based Retake Exam until this portion of the examination is passed. The web-based exam will be conducted live with two examiners during a regularly scheduled ABO exam.
Each examinee is given a 10-year time period form the time they initiated the Clinical Examination to complete all necessary components in order to successfully become board certified.
9. Who will be involved in the development of the new examination?
The case scenarios are created by the ABO Board of Directors, Examiners and CODA-accredited Program Directors/Program Chairs. This process began at the November 2017 Educator’s Symposium where we worked closely with Castle Worldwide, a certification and licensure testing company with 30+ years of experience. Castle will continue to assist the ABO with the testing design and development of the scenario-based examination using the expertise and the knowledge they have gained as a result of the Practice Analysis Study conducted for the ABO.
10. What does the scenario-based examination look like, how does an examinee best prepare for the new examination format, and what kind of preparation resources will be provided by the ABO?
The examinee will be exposed to several pairs of examiners, each of which will present scenario cases and asking specific questions that will be answered orally by the examinee.
When first presented with a set of case documents (a scenario case), which may include different components of patient records and information, candidates may take time to familiarize themselves before the examiners begin asking the questions that have been developed for the scenario. Responses will all be oral, and the examiners will assess the responses as they are given.
The new examination design will require an orthodontist to be proficient in four key domain areas: data gathering and diagnosis, treatment objectives and planning, treatment implementation and management, and critical analysis and outcomes assessment. Self-evaluation of clinical patient cases will play a large part in the preparation process. Participation in “mock boards” and continuing to use the tools previously developed the by ABO for outcomes assessments will play an important role as examinees prepare for the exam.
In addition, a Study guide and Samples cases with scenario-based questions will be available on the ABO website in late May 2018.
11. What can be expected on the new examination? Specifically:
a. How many cases will the examination include?
Each examinee will have exposure to multiple sets of examiners during the examination process. Different scenario cases and questions will be presented by each examiner pair, and these cases will change for different examination windows offered. The number of cases an examinee is being tested on will vary as some examiners may present a scenario case with a full set of records and associated questions, and other examiners may include multiple scenario cases with partial records and associated questions. Some examiners may present single unrelated records that require individual analysis.
b. Will examinees be tested on Discrepancy Index (DI), cast radiograph evaluation (CRE), and cephalometric superimposition technique and interpretation?
Yes, assessment and case evaluation will remain a critical component of the new examination. The tools previously developed by the ABO for case evaluation will still be utilized within the new examination.
An example of this may include an examinee being asked to complete measurements on ABO-provided cases. In addition, a potential scenario case may include tracings that an examinee is asked to superimpose. Another way that the exam may test for an examinee’s knowledge on these tools may involve asking an examinee to review ABO-provided measurements and/or superimpositions and complete a growth analysis and/or discuss the growth effect versus the treatment effect of the case outcome.
c. Will the examination format be the same for all examinees, or will it vary for recent graduates vs. experienced orthodontic practitioners?
The examination format will be the same for recent graduates and experienced orthodontic practitioners.
d. How are examination questions and answers standardized?
For each of the questions in a case scenario, there are at least three predetermined model responses approved by the board: (a) a fully proficient model response, (b) a borderline proficient model response (c) a borderline not or not proficient model response. These questions and responses are sent through a thorough process to be reviewed, edited, refined, and validated prior to being used in an examination. The model responses are used to anchor examiners so that their assessment of candidates’ responses applies a consistent standard for all candidates.
During the examination, responses are scored by trained examiners who will use an anchored rating scale to ensure consistency in scoring. The scales will address the purpose(s) of the case (e.g., diagnosis, planning, implementation, critical analysis) and are standardized for all cases serving the specified purpose. Each unit of the scale is anchored with language that helps to ensure that the ABO’s standards are applied by all examiners.
The ABO’s examiner training program educates examiners in the intended application of the scales to candidate responses. The examiner training program provides practice opportunities as well as a required assessment of agreement with criterion ratings for a selection of responses. Examiners are recalibrated by means of an abbreviated refresher (calibration) training immediately prior to each testing and scoring cycle.
e. How are clinical examination results determined and presented?
The weighting system for the clinical examination details the percentage and number of points for each case defined for the four domains, such that the sum of weights for all problems in each version of the test across the domains equals 100%.
The rating scales for the clinical examination are weighted to achieve 25% allocations to each of the four domains covered in the examination.
Data Gathering and Diagnosis
Treatment Objectives and Planning
Treatment Implementation and Management
Critical Analysis and Outcomes Assessment
Points awarded to candidates are determined as a function of the ratings assigned and the weight of the question. Psychometric analyses are performed after each testing cycle.
Exam results are presented in a pass/fail format. Examinees who are unsuccessful on the examination will be provided with feedback outlining their level of success on the four main domains of the examination. Examinees who have been unsuccessful must re-take the entire examination in order to pass, and will not have the ability to be re-tested on individual sections.
Examination feedback will also be provided to residency programs similar to what is currently provided for the Written Examination.
12. What are the Exam eligibility requirements?
Examinees are eligible to take the Written Examination once they have completed at least 18 months of a CODA-accredited orthodontic program. All examinees who have graduated from a CODA accredited orthodontic program and successfully completed the ABO Written Examination are then eligible for the Scenario-based Oral Clinical Examination.
13. What are the dates of the new examination and when does registration open?
Registration for the February 2019 and November 2019 Clinical Examinations will open on May 22, 2018 and may be completed on the ABO website. A possible third examination will be added in 2019 if necessary. The February 2019 examination will be the last in-person case submission examination. Additional information may be reviewed on the Exam Timing.
14. What are the travel and logistical requirements of the scenario-based Clinical Examination?
The Clinical Examinations will continue to be held in St. Louis, Missouri at The ABO Exam Center.
15. Is Certification Renewal still required?
Yes, Certification Renewal is a critical component of the certification process because it requires ongoing proficiency in orthodontics and a commitment to life-long learning. The certification renewal format currently includes two pathway options to complete the necessary requirements.