Apriori sample size estimation and reporting in original articles published from 2012 to 2020 in two Asian orthodontic journals Ramteke S, Kumar SS, Madhan B

ORIGINAL ARTICLE

Year : 2022 | Volume : 11 | Issue : 1 | Page : 39

Apriori sample size estimation and reporting in original articles published from 2012 to 2020 in two Asian orthodontic journals

Shivangi Ramteke, Sekar Santhosh Kumar, Balasubramanian Madhan
Department of Dentistry, Division of Orthodontics, Jawaharlal Institute of Postgraduate Medical Education and Research (JIPMER), Puducherry, India

Date of Submission	01-Jul-2021
Date of Decision	26-Apr-2022
Date of Acceptance	23-May-2022
Date of Web Publication	24-Aug-2022

Correspondence Address:
Balasubramanian Madhan
Department of Dentistry, Division of Orthodontics, Jawaharlal Institute of Postgraduate Medical Education and Research, Dhanvantri Nagar, Puducherry - 605 006
India

Source of Support: None, Conflict of Interest: None

Check

DOI: 10.4103/jos.jos_159_21

Abstract

AIMS: To evaluate the proportion and completeness of reporting apriori sample size estimation (SSE) in research articles published in the Journal of Orthodontic Science (JOS) and the Journal of Indian Orthodontic Society (JIOS).
MATERIALS AND METHODS: All online research articles published in both journals from 2012 to 2020 were screened. Those reporting apriori SSE were analyzed for the study design and completeness of SSE reporting (outcome parameter and assumptions, Type I error, Power/type II error, one or two-tailed test, the method used, justifications for assumptions, adjustments in sample size, and the target sample size). Chi-square or Fisher exact test was used to analyze the differences between the journals in the proportions of articles reporting these characteristics.
RESULTS: Twenty articles (13.2%) in JOS and 24 (8.3%) in JIOS, have reported apriori SSE [χ² (1, N = 440) = 2.573, P=0.10]. Non-clinical and quasi-experimental studies formed nearly two-thirds of articles reporting SSE in JOS. Quasi-experimental (34%), randomized controlled trials (28%), and cross-sectional studies (24%) formed the bulk of articles reporting SSE in JIOS. Type II error/power was the most frequently reported characteristic in both (75% and 95.8% in JOS and JIOS, respectively), and the number of tails was the least (5% and 0%, respectively). More articles in JOS than JIOS reported the outcome variable used [65% vs. 12.5%, χ² (1, N = 44) = 12.99, P<.001] and provided justifications for the assumptions [70% vs 33.3%, χ² (1, N = 44) = 5.86, P = 0.01].
CONCLUSION: The extent and completeness of reporting apriori SSE are suboptimal in these journals and require prompt and stringent curative measures.

Keywords: Journal of Indian orthodontic society, Journal of orthodontic science, sample size, study design

How to cite this article:
Ramteke S, Kumar SS, Madhan B. Apriori sample size estimation and reporting in original articles published from 2012 to 2020 in two Asian orthodontic journals. J Orthodont Sci 2022;11:39

How to cite this URL:
Ramteke S, Kumar SS, Madhan B. Apriori sample size estimation and reporting in original articles published from 2012 to 2020 in two Asian orthodontic journals. J Orthodont Sci [serial online] 2022 [cited 2023 Jun 10];11:39. Available from: https://www.jorthodsci.org/text.asp?2022/11/1/39/354496

Introduction

Estimating the number of samples required for the study is vital in planning a scientific investigation.^[1],[2] Studies with inadequate sample size may not have adequate power and compromise the validity of the conclusions. On the contrary, including more samples than required raises ethical concerns as it exposes subjects to unnecessary risk and wastes resources.^[3] Hence, using an optimal sample size is vital in any scientific inquiry.^[4]

Sample size estimation is best done before starting a study (Apriori). Of the various parameters required for SSE, the outcome parameter is an essential study-dependent variable. The assumed values for the same (effect size, event rate, variance, etc.)^[5] can be deduced from existing literature, a pilot study, or a forced educated guess. However, they must be justifiable in terms of clinical relevance and feasibility in all cases. The alpha (Type I or False-positive) error is the probability of rejecting the null hypothesis when it is true. The beta (Type II or false-negative) error is the probability of accepting the null hypothesis when it is false. Conventionally, the maximum permissible extent of these are 5% (0.05) and 20% (0.2), respectively. Power is (1- β), and hence the minimal acceptable value is 80% (0.8).^[6] In hypothesis testing, a two-tailed test implies that the hypothesis is nondirectional, and the alpha level is split (as 2.5%) between both tails of the distribution. Generally, these need more samples than a one-tailed test to reach the same power.^[7] Finally, in prospective studies, it may be prudent to appropriately inflate the sample size to compensate for the expected attrition (10-20%) or non-response from participants. Though many methods like formulas,^[8] graphics (e.g., Altman's nomogram),^[9] and tables^[10] are in vogue for SSE, computer software is becoming increasingly popular due to its simplicity, versatility, and economy of time. It is essential that the reporting of SSE be precise and complete with all this information to allow verification and replication of the procedure.

Despite consensus on its paramount importance, under-utilization and poor reporting of sample size estimation are widespread in the medical and dental literature. For example, only 34% of the two-arm parallel-group randomized clinical trials (RCT) with a single primary outcome published in six general medical journals of repute between 2015 - 2016 were found to report all data required to calculate the sample size, had an accurate calculation, and used accurate assumptions for the control group.^[11] Koletsi et al.^[12] noted that only 29.3% of 413 RCTs published over 20 years in eight leading dental journals contained enough information to replicate the sample size estimation. Orthodontic journals are no exception to this scenario. For eight top orthodontic journals evaluated for 20 years, only 29.5% of the RCTs had reported complete sample size estimation.^[13] Most investigations in this regard are restricted to RCTs, and there is a paucity of literature regarding the status of other study designs.

The Journal of Orthodontic Science (JOS), the Saudi Orthodontic Society's official publication, is an international, peer-reviewed journal started in 2012. The Journal of Indian Orthodontic Society (JIOS) is the official journal of The Indian Orthodontic Society, published since 1968. Both publish original research papers in all orthodontics expanses, with their online versions available from 2012 and 2001, respectively. As the reporting of sample size estimation in these journals has not been investigated, the current study was undertaken to evaluate sample size estimation in research publications of these journals in terms of study design, the method used, and completeness of reporting.

Materials and Method

All the scientific publications in JOS and JIOS from 2012 till 2020, available at http://www.journalonweb.com/jos and https://journals.sagepub.com/loi/jioa, respectively, were screened for the content. Publications identified as reporting original research amongst them were perused for sample size reporting in all sections of the manuscript. Those reporting apriori sample size estimation were categorized by their study design (i.e., Non-clinical, Cross-sectional, Case-control, Cohort, Quasi-experiment, and Randomised controlled trials). They were also analyzed for completeness in reporting the characteristics required for sample size estimation as applicable to that particular instance. These included the outcome variable and its assumptions, type I error, type II error/power, one or two-tailed test, the method (table, nomogram, formula, or software), justification for assumptions (citation of relevant literature), adjustments if any (e.g., attrition, non-response, etc.), and the target sample size. Only those characteristics whose information was explicitly mentioned were considered as reported.

Data Acquisition and Analysis

Before initiating the study, a consensus discussion on the classification of study designs and parameters required for sample size estimation was held among the assessors. All three authors did the screening and data collection independently and were later cross-verified for concordance. For discordant entries, the differences were resolved by a group discussion before the final data entry.

The number and proportion of original articles reporting apriori sample size estimation were calculated for each journal. The number and proportion of original articles reporting each variable required to estimate the sample size were also assessed. The difference in the proportion of original articles reporting sample size estimation in the two journals was analyzed for statistical significance with the Chi-square test. The differences in the proportion of articles documenting the variables used in sample size estimation were evaluated with the Chi-square test or Fisher's exact test. All the statistical analyses were performed in OpenEpi Version 3.01.^[14]

Results

Twenty out of 152 (13.2%) in JOS and 24 out of 288 (8.3%) original articles in JIOS had reported apriori sample size estimation [Table 1]. The difference was not statistically significant [χ² (1, N = 440)= 2.573, P =0.10]. The distribution of articles based on study designs is given in [Figure 1]. Non-clinical and quasi-experimental studies formed nearly two-thirds of articles reporting sample size estimation in JOS. For JIOS, quasi-experimental (34%), RCT (28%), and cross-sectional studies (24%) formed the bulk.

Table 1: Apriori sample size reporting in original articles published from 2012 to 2020

Click here to view

Figure 1: Distribution of articles reporting apriori sample size estimation by study design

Click here to view

The data on reporting of variables used in A-priori sample size estimation for both the journals are presented in [Table 2]. Type II error/power was the most frequently reported characteristic in both the journals (75% and 95.8% in JOS and JIOS, respectively). Whether the test was one or two-tailed was the most frequently missing information in both the journals (reported in 5% and 0% in JOS and JIOS, respectively). More articles in JOS reported the outcome variable used for sample size estimation than in JIOS, and the difference was statistically significant [65% vs. 12.5%, χ² (1, N = 44) = 12.99, P <.001]. The same was true for providing justifications for the assumptions used in the calculation [70% vs 33.3%, χ² (1, N = 44) = 5.86, P =0.01]. Though there were differences between the two journals for other variables evaluated in the study, none reached statistical significance. [Figure 2] provides the details of the methods used for SSE. G* Power (Heinrich Heine Universität Düsseldorf), PS: Power and sample size calculation,^[15] and Stata statistical software (StataCorp, College Station, TX, USA) was the common software used for SSE in the reported articles. Surprisingly, 8.3% of these articles in JIOS and nearly one-fourth in JOS failed to explicitly report the study's final (Target) sample size.

Table 2: Reporting of SSE parameters in original articles published from 2012 to 2020

Click here to view

Figure 2: Distribution of articles by methods used for sample size estimation

Click here to view

Discussion

Precise apriori sample size estimation and complete reporting of the process are characteristics of a good research article. However, as there is no information on the extent and completeness of sample-size reporting in original articles published in the Journal of Orthodontic Science and the Journal of Indian Orthodontic Society, the current study evaluated this for online articles published from 2012 to 2020.

Overall, only 13.2% and 8.3% of original JOS and JIOS articles reported apriori sample size estimation. For articles published between 2005 and 2008, this proportion was found to be higher for the American Journal of Orthodontics and Dentofacial Orthopedics (AJODO) (21.1%) and substantially low for Revista Dental Press de Ortodontia e Ortopedia Facial (3.9%).^[16] Another study that evaluated six dental specialty journals with the highest impact factor in 2008 found SSE reporting in 7.3% of articles on average, with AJODO standing at 2.9%.^[17]

An analysis based on study designs indicated that Quasi-experimental studies comprised nearly one-third of articles reporting SSE in both JOS and JIOS. Non-clinical articles contributed to another third of JOS. The lack of exactly similar studies precludes a direct comparison of these results. For RCTs published in eight leading orthodontic journals over twenty years up to 2012, Koletsi et al.^[13] noted complete SSE reporting in 29.5%, probably higher if partial reporting was also considered. A similar study on 50 issues up to 2013 in four leading orthodontic journals noted SSE description at least partially in 56.2% of RCTs.^[18] The higher reporting of SSE seen in these studies than ours is understandable given that they included only RCTs. Trials are generally more robust than observational studies in planning and reporting and are more likely to fare better. A recent survey of journals in five medical specialties by Tripathi et al.^[19] corroborated that the interventional studies show higher SSE reporting than observational studies.

The same reason could also hold for more comprehensive SSE reporting in RCTs than in other study designs. Koletsi et al.^[13] noted adequate information to verify SSE only in 29.5% of RCTs published over 20 years in 8 leading orthodontic journals. Tripathi et al.^[19] reported this figure to be less than 50% for six medical specialty journals. In this study, only one article in JOS and none in JIOS had explicitly reported the complete information required to verify the accuracy of SSE without ambiguity. Unfortunately, 8.3% of articles in JIOS and 25% in JOS failed to mention even the final sample size after estimation.

These findings highlight the suboptimal and incomplete quality of SSE reporting in these two journals, hence the compelling need for imminent and proactive measures to improve them. A multi-pronged approach involving all the stakeholders of the journal is the need of the hour. Firstly, the investigators must recognize the importance of SSE in planning their study and take the onus for adequately reporting it in all ensuing publications. Second, the review boards/ethical committees must seriously note missing SSE during approval. Third, reviewers must treat the failure to report SSE as a significant omission of information and highlight this concern during the peer review. Finally, the journals must go beyond just recommendations and enforce strict compliance to current reporting guidelines like CONSORT, STROBE, etc., even while accepting articles for review, let alone publication.

Limitations and future scope

Very few articles described all the parameters required for SSE. Hence, replicating the process to verify its accuracy was not performed, an important limitation of the study. Appraising the accuracy and clinical relevance of the estimates used for SSE in these articles and evaluating the changes in the reporting SSE over the years are two possible areas of future inquiry.

Conclusions

The following are the salient conclusions of this study.

Apriori sample size estimation was reported in only 13.2% and 8.3% of original articles in the Journal of Orthodontic Science and the Journal of Indian Orthodontic Society.
Quasi-experimental studies formed nearly one-third of articles reporting SSE in both these journals.
Type II error/power was the most frequently reported characteristic in SSE in both the journals and the number of tails was the least.

Overall, the extent and completeness of reporting sample size estimation were unsatisfactory, highlighting the need for early remedial action.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

References

1.	Last JM. Making the dictionary of epidemiology. Int J Epidemiol 1996;25:1098-101.
2.	Macfarlane TV. Sample size determination for research projects. J Orthod 2003;30:99-100.
3.	Faber J, Fonseca LM. How sample size influences research outcomes. Dental Press J Orthod 2014;19:27-9.
4.	Freiman JA, Chalmers TC, Smith H, Kuebler RR. The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial. Survey of 71 “negative” trials. N Engl J Med 1978;299:690-4.
5.	Jain S, Dubey S, Jain S. Designing and validation of questionnaire. Int Dent Med J Adv Res 2016;2:1-3.
6.	Beck TW. The importance of a priori sample size estimation in strength and conditioning research. J Strength Cond Res 2013;27:2323-37.
7.	Chan YH. Randomised controlled trials (RCTs)--sample size: The magic number? Singapore Med J 2003;44:172-4.
8.	Peacock J, Peacock P. Oxford Handbook of Medical Statistics. OUP Oxford; 2011. 540 p.
9.	Altman DG. Statistics and ethics in medical research: III How large a sample? Br Med J 1980;281:1336-8.
10.	Röhrig B, du Prel J-B, Wachtlin D, Kwiecien R, Blettner M. Sample size calculation in clinical trials. Dtsch Arztebl Int 2010;107:552-6.
11.	Charles P, Giraudeau B, Dechartres A, Baron G, Ravaud P. Reporting of sample size calculation in randomised controlled trials: Review. BMJ 2009;338:b1732.
12.	Koletsi D, Fleming PS, Seehra J, Bagos PG, Pandis N. Are sample sizes clear and justified in RCTs published in dental journals? PLoS One 2014;9:e85949.
13.	Koletsi D, Pandis N, Fleming PS. Sample size in orthodontic randomized controlled trials: Are numbers justified? Eur J Orthod 2014;36:67-73.
14.	Sullivan KM, Dean A, Soe MM. OpenEpi: A web-based epidemiologic and statistical calculator for public health. Public Health Rep 2009;124:471-4.
15.	Dupont WD, Plummer WD. Power and sample size calculations. A review and computer program. Control Clin Trials 1990;11:116-28.
16.	Normando D. Analysis of the use of sample size calculation and error of method in researches published in Brazilian and international orthodontic journals. Dental Press J Orthod 2011;16:33-5.
17.	Pandis N, Polychronopoulou A, Madianos P, Makou M, Eliades T. Reporting of research quality characteristics of studies published in 6 major clinical dental specialty journals. J Evid Based Dent Pract 2011;11:75-83.
18.	Lempesi E, Koletsi D, Fleming PS, Pandis N. The reporting quality of randomized controlled trials in orthodontics. J Evid Based Dent Pract 2014;14:46-52.
19.	Tripathi R, Khatri N, Mamde A. Sample size and sampling considerations in published clinical research articles. J Assoc Physicians India 2020;68:14-8.