Treatment outcome differences between pass and fail scores and correlation between cephalometric changes and cast-radiograph evaluation of the American Board of Orthodontics Neoh SP, Komoltri C, Viwattanatipa N

ORIGINAL ARTICLE

Year : 2018 | Volume : 7 | Issue : 1 | Page : 22

Treatment outcome differences between pass and fail scores and correlation between cephalometric changes and cast-radiograph evaluation of the American Board of Orthodontics

Siew Peng Neoh¹, Chulaluk Komoltri², Nita Viwattanatipa¹
¹ Department of Orthodontics, Faculty of Dentistry, Mahidol University, Bangkok, Thailand
² Department of Epidemiology, Faculty of Medicine, Siriraj Hospital, Mahidol University, Bangkok, Thailand

Date of Web Publication

15-Nov-2018

Correspondence Address:
Prof. Nita Viwattanatipa
Department of Orthodontics, Faculty of Dentistry, Mahidol University, Bangkok
Thailand

Source of Support: None, Conflict of Interest: None

Check

DOI: 10.4103/jos.JOS_33_18

Abstract

OBJECTIVE: Using the cast-radiograph evaluation (CRE) score of the American Board of Orthodontics (ABO), the purpose of this study was 1) to find the post-treatment discrepancies that contributed to low-quality outcomes and 2) to identify if there might be any correlation between cephalometric changes and post-treatment discrepancies.
MATERIALS AND METHODS: About 200 records submitted for the Thai Board of Orthodontics examination were analyzed. Overall, 23 parameters of the CRE scores and 12 cephalometric changes were collected. Based on the total CRE score, the cases were classified into three categories: pass (score <20), undetermined (score 20–30), and fail (score >30). Kruskall–Wallis was used to analyze the differences of mean CRE scores among these three categories. In addition, the cases were further classified into fixed appliance, two-phase and orthognathic surgery groups. Correlation tests were carried out to determine if there might be any association between cephalometric changes and CRE parameters.
RESULTS: Significant differences of mean CRE scores were found for all CRE components except interproximal contacts. Significant correlation coefficients with the total CRE scores were found for all parameters except interproximal contacts. Significant moderate association was found between lower incisor changes and CRE scores in the two-phase and orthognathic surgery group.
CONCLUSIONS: In order to improve treatment outcome quality, the top four parameters that orthodontists should pay attention to are occlusal contacts, occlusal relationship, marginal ridges, and alignment and rotations. Cephalometric changes were not suitable as weighting factors for total CRE scores.

Keywords: American board of orthodontics, cast-radiograph evaluation score, cephalometric changes, objective grading system, orthodontic treatment outcome

How to cite this article:
Neoh SP, Komoltri C, Viwattanatipa N. Treatment outcome differences between pass and fail scores and correlation between cephalometric changes and cast-radiograph evaluation of the American Board of Orthodontics. J Orthodont Sci 2018;7:22

How to cite this URL:
Neoh SP, Komoltri C, Viwattanatipa N. Treatment outcome differences between pass and fail scores and correlation between cephalometric changes and cast-radiograph evaluation of the American Board of Orthodontics. J Orthodont Sci [serial online] 2018 [cited 2023 Mar 22];7:22. Available from: https://www.jorthodsci.org/text.asp?2018/7/1/22/245519

Introduction

Over the years, there have been numerous attempts to formulate an objective means of determining the successful outcome of orthodontic treatment. Examples of these systems are the American Board of Orthodontics Objective Grading System, peer assessment rating index, index of complexity, outcome and need, dental esthetic index, and comprehensive clinical assessment (CCA).^{[1],[2],[3],[4],[5],[6],[7]}

The American Board of Orthodontics (ABO) developed an Objective Grading System (OGS) for post-treatment evaluation, using the ABO cast-radiograph evaluation (CRE).^[4],[8],[9] CRE scores have been tested for its validity and reliability in many studies to evaluate the quality of treated orthodontic cases in university and private practice patients.^{[3],[7],[10],[11],[12],[13],[14]} Mean total CRE scores reported were highly varied, ranging from 16.3^[15] to 45.5^[16]. These differences could be attributed to the clinicians' skill,^[13],[16] orthodontic appliances,^{[17],[18],[19]} extraction versus nonextraction,^[20] etc., Nonetheless, most of these studies were in agreement that using ABO CRE scores would help in improving the quality of their completed cases.

Although the CRE was considered the most rigid and sensitive tool for post-treatment outcome evaluation, it was not able to show the degree by which the case improved from its pretreatment condition. Several attempts have been made to search for parameters that might be associated with the CRE score. Such parameters could then be used as weighting factors for the orthodontic board examiners. One of the most studied parameters was the ABO discrepancy index (DI). However, the results have been contradictory. Some studies reported no correlation between the DI and CRE scores.^[9],[10] On the other hand, Campbell et al.^[21] and Pulfer et al.^[22] found a positive but weak correlation between the DI and CRE scores. In any case, cephalometric changes have not been studied in the past. Clinicians may have doubt whether greater discrepancies in post-treatment outcomes could be due to severe dento-skeletal problems.

Therefore, our research questioned if there could be any parameters of cephalometric changes associated with CRE scores. The specific purpose of this study was as follows:

Identifying the weaknesses of post-treatment outcomes in cases submitted for the Thai Board of Orthodontics (ThaBO) examination by using the CRE score
Evaluating the contribution of each CRE component to the total CRE score
Identifying if any parameters of cephalometric changes were associated with CRE scores.

Materials and Methods

This research was granted the certificate of approval by the institutional review board of Faculty of Dentistry/Faculty of Pharmacy, Mahidol University, COA No. MU-DT/PY-IRB 2018/037.2806.

Sample size

Sample size was calculated based on the expected correlation coefficient, r = 0.2, two-sided type I error (α) of 0.05, and type II error (β) of 0.2 (80% power) and found that 194 samples were needed.

Data collection

The samples for this retrospective study were selected from the archives of cases submitted for the ThaBO examination for the year 2014. The cases had been treated by orthodontic residents in university-affiliated, 3-year accredited training programs from various institutions. In total, 200 cases which were obtained from 25 residents who were required to submit 8 cases each that met the specifications of 10 malocclusion categories, resembling the ABO criteria.

The names of the residents, institutions, and patients were not exposed to the researcher, and each case was assigned a random number. They were collected as three main sections: (1) pretreatment characteristics and treatment type, (2) ABO CRE score, and (3) cephalometric changes.

Pretreatment characteristics and treatment type

Pretreatment characteristics were recorded for frequency according to the following subcategories: (1) malocclusion type, (2) missing teeth, (3) crowding, and (4) treatment type. The details are shown in [Table 1] and [Figure 1].

Table 1: Descriptive frequencies of pretreatment variables and treatment types

Click here to view

Figure 1: Pie charts of pretreatment variables and treatment types. (a) Malocclusion. (b) Missing teeth. (c) Crowding. (d) Treatment type

Click here to view

Cast-radiograph evaluation score

The raw data of ABO CRE scores were obtained from our previous study as our research was conducted as an extension project.^[9] Emphasis was placed on the main and subcomponents of the CRE data, which had not yet been reported elsewhere.

According to the methodology of the previous study, the 200 posttreatment records were assessed with the CRE score sheet using the ABO measuring gauge, by scoring various aspects on the study casts and panoramic radiographs to cover the following eight main components: (1) alignment and rotations, (2) marginal ridges, (3) buccolingual inclination, (4) overjet, (5) occlusal contacts, (6) occlusal relationships, (7) interproximal contacts, and (8) root angulation. Each component was further separated into maxillary and mandibular measurements, except for overjet (separated into anterior and posterior regions), occlusal contacts (buccal and lingual subcomponents), and occlusal relationships. In total, 23 parameters (total CRE score, 8 main components, 14 subcomponents) related to the CRE score were obtained [Table 2].

Table 2: Descriptive statistics of cast-radiograph evaluation components (n=200)

Click here to view

All cases were further divided into three categories for the pass/fail status according to the CRE scores as suggested by Casko et al.^[1] as follows:

Pass: CRE <20
Undetermined: CRE 20–30
Fail: CRE >30.

Cephalometric changes

Twelve measurements of cephalometric changes were obtained from the patients file records. Because most candidates did not report the same sets of measurements, only those cephalometric changes that were consistently reported in all 200 cases were selected for our study.

Statistical analysis

All data were subjected to statistical analyses using IBM SPSS Statistics for Windows, version 22.0 (IBM Corp., Armonk, NY, USA).

Test of normality

The Shapiro–Wilk Test signified that the CRE scores were not of normal distribution; hence, nonparametric tests were to be conducted. Histograms of the frequency and distribution for the scores of the eight main CRE components and total CRE are presented in [Figure 2].

Figure 2: Frequency and distribution of different cast-radiograph evaluation components and the total cast-radiograph evaluation score

Click here to view

Descriptive statistics

Descriptive frequencies of pretreatment characteristics (malocclusion type, missing teeth, crowding) and treatment type were analyzed [Table 1].

Descriptive statistics were also carried out for the scores of the total CRE, 8 main CRE components, and 14 CRE subcomponents to determine means, standard deviations (SD), and range [Table 2], as well as for the subcategories of pass, undetermined, and fail groups [Table 3]. Means, SD, and range were computed for cephalometric changes in absolute values according to the treatment type [Table 4].

Table 3: Results of Kruskall-Wallis and post hoc tests, across pass, undetermined, and fail cast-radiograph evaluation (n=200)

Click here to view

Table 4: Descriptive statistics of cephalometric changes (absolute values) according to treatment type

Click here to view

Comparative statistics

Comparisons of differences among means of all 23 CRE parameters across pass, undetermined, and fail categories were performed by Kruskal–Wallis tests to ascertain whether significant differences across these three categories existed. This was followed by post-hoc tests for significant parameters.

Correlation tests

Nonparametric Spearman's rank-order correlation tests were conducted between 22 CRE subcomponents with the total CRE scores, as well as between the total CRE scores with the 12 parameters of cephalometric changes according to the treatment type.

Standard error of measurement

Standard error of measurement (SEM) was determined using the intraclass correlation coefficient, where the two-way random model was selected for absolute agreement for both intraobserver and interobserver measurements.

Results

Descriptive frequencies of pretreatment characteristics and treatment type

The frequencies and percentages of pretreatment characteristics, such as malocclusion type, region of missing teeth, degree of crowding, as well as treatment type, are described in [Table 1] and [Figure 1]. According to the treatment type, the statistics showed that 50% were extraction cases, 27.5% were nonextraction cases, whereas orthognathic surgery and two-phase treatment were done in 13% and 9.5% of the total cases, respectively.

Frequency distributions and means of total cast-radiograph evaluation and its components

The frequency distributions of the scores of the total CRE and its eight main components are shown in [Figure 2]. The total CRE score was normally distributed, whereas the eight main CRE components displayed variably skewed distributions.

The means, SD, and range of the total CRE score and its main components are summarized in [Table 2]. The highest mean values came from buccolingual inclinations and alignment and rotations, with scores of 5.39 and 4.97, respectively. The mean of the total CRE score was 26.53 (SD 9.57), which falls into the undetermined category (CRE 20–30). The majority of cases also fell within this undetermined category, making up 43% of the total sample.

Comparisons of means of cast-radiograph evaluation components

Most CRE components exhibited significant differences in mean scores across the categories of pass, undetermined, and fail, except interproximal contacts and root angulation [Table 3] and [Figure 3]. Post-hoc tests showed significant differences in the means between pass and fail groups for the total CRE and all CRE components except interproximal contacts and root angulation (maxilla).

Figure 3: Clustered bar chart of distribution of means of eight main cast-radiograph evaluation components across pass, undetermined, and fail categories

Click here to view

Descriptive mean cephalometric changes

The means, SD, and range of cephalometric changes in absolute values according to the treatment type are shown in [Table 4].

Correlations between cast-radiograph evaluation components with total cast-radiograph evaluation score

Results of the Spearman's rank-order correlation tests performed between the CRE components with the total CRE score are displayed in [Figure 4] and [Table 5].

Figure 4: Conceptual framework and correlation coefficients between investigated variables

Click here to view

Table 5: Results of Spearman's rank-order correlation tests between cast-radiograph evaluation components with the total cast-radiograph evaluation score

Click here to view

In [Table 5], significant correlations at a moderate level can be noted for all variables except for interproximal contacts and maxillary root angulation. For the eight main CRE components, the correlation coefficients, r, ranking from highest to lowest were as follows: (1) occlusal contacts (0.62), (2) occlusal relationship (0.60), (3) marginal ridges (0.58), (4) alignment and rotations (0.57), (5) overjet (0.38), (6) buccolingual inclinations (0.38), and (7) root angulation (0.25) (P < 0.05).

Correlations between total cast-radiograph evaluation with cephalometric changes according to treatment type

Statistically significant correlations at moderate levels were found as follows: (1) total CRE score with L1-NB (° and mm.) in two-phase treatment (r = 0.479 and 0.473, respectively) and (2) total CRE score with L1-MP (°) in orthognathic surgery cases (r = −0.434) [Table 6] and [Figure 4].

Table 6: Results of Spearman's rank-order correlation tests between total cast-radiograph evaluation score with cephalometric changes according to treatment type

Click here to view

Standard error of measurement

Reliability tests showed that the intraclass correlation coefficient for intraobserver reliability was computed to be 0.89 and 0.81, while interobserver was 0.73.

Discussion

In this study, we aimed to identify the CRE components that contributed significantly to the total CRE score, and specifically in cases with scores that would potentially fail the ABO examination. Our results show that significant treatment outcome differences among pass, undetermined, and fail categories of cases submitted for the ThaBO examination were found in the following CRE components: (1) alignment and rotations, (2) buccolingual inclinations, (3) occlusal contacts, (4) marginal ridges, (5) occlusal relationship, (6) overjet, and (7) root angulations. However, there were no significant differences for interproximal contacts.

For overall cases, we found no correlation between any cephalometric changes and total CRE score. However, lower incisors inclination and position were significantly correlated with total CRE in the two-phase treatment and orthognathic surgery subgroups

Cast-radiograph evaluation components

The results of our study were generally in agreement with other past studies. According to [Table 7], the mean total CRE score of 26.5 (SD 9.6) in our study is found to be comparable to the majority of other university-based studies.^{[8],[12],[14],[23],[24]} In fact, there were several other studies that reported even greater mean total CRE scores (>30).^{[3],[7],[16],[17],[18]}

Table 7: Summary of results for cast-radiograph evaluation components and treatment duration from previous studies

Click here to view

Though this may seem rather alarming, a review of other studies done over the past two decades showed quite similar results when this pass criteria of CRE score <20 was used.^{[8],[12],[14],[18],[21],[23],[24],[25],[26],[27],[28],[30]}

However, the level of experience and skill of orthodontists may account for differences in mean CRE scores. Yang-Powers et al.^[16] reported that cases submitted for the ABO examination had significantly better scores, particularly for occlusal contacts and overjet components, which possibly reflected higher skill in finishing cases, compared with those that were treated consecutively by university residents and not necessarily for ABO examination purposes.

Correlations: Cast-radiograph evaluation components with total cast-radiograph evaluation score

Occlusal contacts

Occlusal contacts was ranked as the highest contributor, demonstrating significance at a moderate level with the total CRE score (P < 0.01). We found that lingual occlusal contacts (r = 0.542) contributed to the total CRE score slightly more than the buccal aspects (r = 0.399). These deficiencies in occlusal contacts could be due to (1) infraocclusion, (2) excessive mesiodistal crown tip, and (3) nonharmonized torque (buccolingual inclination) in the transverse dimension.

Despite this seemingly large contribution, many authors have suggested that occlusal settling could occur during the retention phase,^[11] which would actually improve and increase the number of occlusal contacts post-treatment.^{[31],[32],[33],[34]} Nevertheless, the potential of this occlusal settling period should not be used as justification for post-treatment deficiencies as reflected in the CRE scores. Therefore, it was recommended that the cases should be finished with as good quality occlusions as possible, rather than relying on any post-treatment changes for improvements.

Occlusal relationship

Deficiencies in occlusal relationship, once present, would almost always lead to high scores, as the discrepancies would usually be affected all the way from the canines right through to the second molars, and also affecting both right and left sides.

Marginal ridges

The presence of marginal ridge discrepancies is concomitant with those of occlusal contacts. Cases with deficient occlusal contacts, that is, vertical infraocclusion, excessive mesiodistal crown tip, and/or nonharmonized torque (buccolingual inclination), would have many scores deducted. We also found that these marginal ridge discrepancies were more prevalent in the maxilla (r = 0.520) than in the mandible (r = 0.398). This finding is in agreement with the ABO report, which stated that the most common mistakes in marginal ridge alignment occurred between the maxillary first and second molars, whereas the second most common problematic area was between the mandibular first and second molars.^[2]

Alignment and rotations

The results of the four field tests by the ABO study showed that the most commonly mal-aligned teeth post-treatment were the lateral incisors and second molars, which accounted for nearly 80% of the discrepancies.^[2]

Overjet

The post-treatment overjet problems may be attributed to the shovel shaped anatomy of upper incisors prevalent in the Asian phenotype. It was suspected that thicker lingual marginal ridges and larger tooth size of upper incisors in Asians would lead to increased overjet and a more mesially aligned dentition.^[3]

Buccolingual inclination

We found significant correlations in buccolingual inclination with stronger associations in the maxillary teeth (r = 0.379) compared with the mandibular teeth (r = 0.184). Similarly, Deguchi et al.^[3] identified the maxillary or mandibular second molars to be the main problematic areas. Along with Yang-Powers et al.,^[16] they inferred that this could be a result of inadequate torque control due to one of these factors: (1) the second molars were banded late in treatment, (2) difficulty of visual assessment (3) larger tooth size compared with arch length leading to molars flaring out, and (4) use of preadjusted appliances with inadequate torque.

Root angulation

Our result was in agreement with the ABO study, which reported that the mistakes in root angulation commonly occurred in the maxillary lateral incisors, canines, second premolars, and mandibular first premolars.^[2] However, it should be noted that the assessment of mesiodistal root angulations in panoramic radiographs should be approached with caution and reinforced with a thorough clinical examination of the dentition.^[35] Root angulation discrepancies detected by panoramic films could be related to mesiodistal tipping, insufficient torque, or tooth rotations.

Cephalometric changes

It was our assumption that greater cephalometric changes could be associated with higher CRE scores due to greater treatment difficulty. It should also be kept in mind that 13% of cases in this sample underwent orthognathic surgery and 9.5% underwent two-phase treatment. These treatment modalities could have some impact in reducing the difficulty of orthodontic treatment.

No significant correlations were found between any cephalometric changes with the total CRE score in the fixed appliances group. We can infer that the candidates were sufficiently skilled to finish their cases selected for fixed appliances to acceptable quality whether or not they performed a large or small amount of changes in the skeletal or dental components.

However, we found a significant moderate correlation existed between changes in the lower incisal inclination and position (L1-NB) with the total CRE score in cases which underwent two-phase treatment. We can suggest that though the first phase of treatment in such patients was successful in correcting the skeletal discrepancy by growth modification, further extractions were still required in the second phase for correction of protrusion or crowding. This would demand additional good control of tooth movement, which could, in turn, worsen certain aspects of the occlusion as measured by the CRE.

For the cases that underwent orthognathic surgery, a significant negative and moderate correlation was found between the L1-MP change (lower incisor inclinations) and the total CRE score. As the majority of orthognathic cases presented by the candidates were of skeletal type III patterns, we can expect that more dental decompensation of the pretreatment lower incisors would enable more accurate anteroposterior correction of the skeletal discrepancy, hence producing a better final occlusion and lower CRE score. If the dentition was not adequately decompensated presurgically, the surgical procedure could have been compromised. This could have resulted in a large remaining overjet or a residual Class III occlusal relationship postsurgically, producing less ideal CRE scores.

Limitations

The samples in this study were only limited to a select amount and type of orthodontic cases, treated by orthodontic residents in postgraduate training programmes in institutes across the country. All candidates were not aware that their submitted cases would be evaluated with the ABO CRE and they had never been trained to use this system. Hence, the results of this study may not reflect the overall quality of orthodontic treatment outcome to encompass those treated by more recent residents or practitioners in the private sector. In addition, the passing criteria used by the ThaBO was based on both outcome evaluation and case presentation which made the percentage passing/fail results differently from ABO.

Clinical implications

It is our recommendation that orthodontic training program directors should strongly emphasize inclusion of ABO CRE assessment in their curriculum in order to improve the overall quality of orthodontic treatment. Suggestions for future studies may involve the ABO case management form or the CCA as a supplement to the CRE.

Conclusions

Significant differences in the mean CRE scores among the three categories (pass, undetermined, and fail) were found for all CRE components except interproximal contacts
Significant correlation coefficients, r, were found, ranking from highest to lowest: (1) occlusal contacts (0.62), (2) occlusal relationship (0.60), (3) marginal ridges (0.58), (4) alignment and rotations (0.57), (5) overjet (0.38), (6) buccolingual inclinations (0.38), and (7) root angulation (0.25) (P < 0.05)
Cephalometric changes were not suitable as weighting factors for CRE scores of overall cases
There were moderately significant correlations between changes in lower incisor inclinations with the total CRE score in 2-phase treatment and orthognathic surgery cases.

Acknowledgement

The authors are grateful for the permission granted by the Royal College of Dental Surgeons of Thailand, Executive Board of Orthodontics Division for the use of the samples in this study. We are indebted to Dr. Weerawat Buapuean for his part in data collection. We are also thankful for the research grant supported by the Faculty of Dentistry, Mahidol University.

Financial support and sponsorship

Faculty of Dentistry, Mahidol University.

Conflicts of interest

There are no conflicts of interest.

References

1.	Casko JS, Vaden JL, Kokich VG, Damone J, James RD, Cangialosi TJ, et al. Objective grading system for dental casts and panoramic radiographs. American board of orthodontics. Am J Orthod Dentofacial Orthop 1998;114:589-99.
2.	The American Board of Orthodontics. The Anerican Board of Orthodontics Grading System for Dental Casts and Panoramic Radiographs. Saint Louis: The American Board of Orthodontics; 2012.
3.	Deguchi T, Honjo T, Fukunaga T, Miyawaki S, Roberts WE, Takano-Yamamoto T, et al. Clinical assessment of orthodontic outcomes with the peer assessment rating, discrepancy index, objective grading system, and comprehensive clinical assessment. Am J Orthod Dentofacial Orthop 2005;127:434-43.
4.	Dyken RA, Sadowsky PL, Hurst D. Orthodontic outcomes assessment using the peer assessment rating index. Angle Orthod 2001;71:164-9.
5.	Lobb WK, Ismail AI, Andrews CL, Spracklin TE. Evaluation of orthodontic treatment using the dental aesthetic index. Am J Orthod Dentofacial Orthop 1994;106:70-5.
6.	Onyeaso CO, Begole EA. Relationship between index of complexity, outcome and need, dental aesthetic index, peer assessment rating index, and American board of orthodontics objective grading system. Am J Orthod Dentofacial Orthop 2007;131:248-52.
7.	Pinskaya YB, Hsieh TJ, Roberts WE, Hartsfield JK. Comprehensive clinical evaluation as an outcome assessment for a graduate orthodontics program. Am J Orthod Dentofacial Orthop 2004;126:533-43.
8.	Deguchi T, Terao F, Aonuma T, Kataoka T, Sugawara Y, Yamashiro T, et al. Outcome assessment of lingual and labial appliances compared with cephalometric analysis, peer assessment rating, and objective grading system in angle class II extraction cases. Angle Orthod 2015;85:400-7.
9.	Viwattanatipa N, Buapuean W, Komoltri C. Relationship between discrepancy index and the objective grading system in thai board of orthodontics patients. Orthod Waves 2016;75:54-63.
10.	Vu CQ, Roberts WE, Hartsfield JK Jr., Ofner S. Treatment complexity index for assessing the relationship of treatment duration and outcomes in a graduate orthodontics clinic. Am J Orthod Dentofacial Orthop 2008;133:9.e1-13.
11.	Knierim K, Roberts WE, Hartsfield J Jr. Assessing treatment outcomes for a graduate orthodontics program: Follow-up study for the classes of 2001-2003. Am J Orthod Dentofacial Orthop 2006;130:648-55, 655.e1-3.
12.	Cook DR, Harris EF, Vaden JL. Comparison of university and private-practice orthodontic treatment outcomes with the american board of orthodontics objective grading system. Am J Orthod Dentofacial Orthop 2005;127:707-12.
13.	Abei Y, Nelson S, Amberman BD, Hans MG. Comparing orthodontic treatment outcome between orthodontists and general dentists with the ABO index. Am J Orthod Dentofacial Orthop 2004;126:544-8.
14.	Mislik B, Konstantonis D, Katsadouris A, Eliades T. University clinic and private practice treatment outcomes in class I extraction and nonextraction patients: A comparative study with the American board of orthodontics objective grading system. Am J Orthod Dentofacial Orthop 2016;149:253-8.
15.	Struble BH, Huang GJ. Comparison of prospectively and retrospectively selected american board of orthodontics cases. Am J Orthod Dentofacial Orthop 2010;137:6.e1-8.
16.	Yang-Powers LC, Sadowsky C, Rosenstein S, BeGole EA. Treatment outcome in a graduate orthodontic clinic using the American board of orthodontics grading system. Am J Orthod Dentofacial Orthop 2002;122:451-5.
17.	Djeu G, Shelton C, Maganzini A. Outcome assessment of invisalign and traditional orthodontic treatment compared with the american board of orthodontics objective grading system. Am J Orthod Dentofacial Orthop 2005;128:292-8.
18.	Detterline DA, Isikbay SC, Brizendine EJ, Kula KS. Clinical outcomes of 0.018-inch and 0.022-inch bracket slot using the ABO objective grading system. Angle Orthod 2010;80:528-32.
19.	Jain M, Varghese J, Mascarenhas R, Mogra S, Shetty S, Dhakar N, et al. Assessment of clinical outcomes of roth and MBT bracket prescription using the American board of orthodontics objective grading system. Contemp Clin Dent 2013;4:307-12. [PUBMED] [Full text]
20.	Cansunar HA, Uysal T. Relationship between pretreatment case complexity and orthodontic clinical outcomes determined by the American board of orthodontics criteria. Angle Orthod 2014;84:974-9.
21.	Campbell CL, Roberts WE, Hartsfield JK, Jr., Qi R. Treatment outcomes in a graduate orthodontic clinic for cases defined by the American board of orthodontics malocclusion categories. Am J Orthod Dentofacial Orthop 2007;132:822-9.
22.	Pulfer RM, Drake CT, Maupome G, Eckert GJ, Roberts WE. The association of malocclusion complexity and orthodontic treatment outcomes. Angle Orthod 2009;79:468-72.
23.	Anthopoulou C, Konstantonis D, Makou M. Treatment outcomes after extraction and nonextraction treatment evaluated with the American board of orthodontics objective grading system. Am J Orthod Dentofacial Orthop 2014;146:717-23.
24.	Wes Fleming J, Buschang PH, Kim KB, Oliver DR. Posttreatment occlusal variability among angle class I nonextraction patients. Angle Orthod 2008;78:625-30.
25.	Chaison ET, Liu X, Tuncay OC. The quality of treatment in the adult orthodontic patient as judged by orthodontists and measured by the objective grading system. Am J Orthod Dentofacial Orthop 2011;139:S69-75.
26.	Chalabi O, Preston CB, Al-Jewair TS, Tabbaa S. A comparison of orthodontic treatment outcomes using the objective grading system (OGS) and the peer assessment rating (PAR) index. Aust Orthod J 2015;31:157-64.
27.	Eberting JJ, Straja SR, Tuncay OC. Treatment time, outcome, and patient satisfaction comparisons of damon and conventional brackets. Clin Orthod Res 2001;4:228-34.
28.	Fleming JW. An Analysis of Variability in Class I Non-Extraction Treatment Outcomes in a Resident Clinic Using the American Board of Orthodontics Objective Grading System[Thesis]. Missouri: Saint Louis University; 2007.
29.	Hsieh TJ, Pinskaya Y, Roberts WE. Assessment of orthodontic treatment outcomes: Early treatment versus late treatment. Angle Orthod 2005;75:162-70.
30.	Santiago JJ, Martínez CJ. Use of the objective grading system of the American board of orthodontics to evaluate treatment at the orthodontic graduate program clinic, university of Puerto Rico, 2007-2008. P R Health Sci J 2012;31:29-34.
31.	Aszkler RM, Preston CB, Saltaji H, Tabbaa S. Long-term occlusal changes assessed by the American board of orthodontics' model grading system. Am J Orthod Dentofacial Orthop 2014;145:173-8.
32.	Başçiftçi FA, Uysal T, Sari Z, Inan O. Occlusal contacts with different retention procedures in 1-year follow-up period. Am J Orthod Dentofacial Orthop 2007;131:357-62.
33.	Dinçer M, Meral O, Tümer N. The investigation of occlusal contacts during the retention period. Angle Orthod 2003;73:640-6.
34.	Sari Z, Uysal T, Başçiftçi FA, Inan O. Occlusal contact changes with removable and bonded retainers in a 1-year retention period. Angle Orthod 2009;79:867-72.
35.	Bouwens DG, Cevidanes L, Ludlow JB, Phillips C. Comparison of mesiodistal root angulation with posttreatment panoramic radiographs and cone-beam computed tomography. Am J Orthod Dentofacial Orthop 2011;139:126-32.