# Design and Analysis Methods for Trials with AI-Based Diagnostic Devices for Breast Cancer

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Materials and Methods

#### 2.1. Objective 1: Is the Concordance Rate between the AI-Based Device and Radiologists as High as That among Radiologists?

#### 2.1.1. Statistical Testing Method

#### 2.1.2. Power and Sample Size Calculation

- (1)
- Specify ($\alpha ,1-\beta $), expected concordance rate among radiologists ${p}_{r}$, similarity margin ${\delta}_{1}$ and hypothetical correlation coefficients ${\rho}_{r1},{\rho}_{r2},{\rho}_{ss},{\rho}_{s1}$ and ${\rho}_{s2}$.
- (2)
- Calculate ${\sigma}_{1}^{2}$ using (3).
- (3)
- Obtain sample size using (2).

#### 2.2. Objective 2: Is the AI-Based Device More Concordant with Experienced Radiologists Than with Junior Radiologists?

#### 2.2.1. Statistical Testing Method

#### 2.2.2. Power and Sample Size Calculation

- 1.
- Specify ($\alpha ,1-\beta $), expected concordance rate between the AI-based device and a highly experienced radiologist ${p}_{x}$, clinically meaningful difference in concordance rates ${\delta}_{2}$ and correlation coefficients ${\rho}_{xx},{\rho}_{yy}$ and ${\rho}_{xy}$.
- 2.
- Calculate ${\sigma}_{2}^{2}$ using (6).
- 3.
- Obtain the required sample size using (5).

## 3. Numerical Studies and Results

## 4. Discussion and Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## Appendix A

Ground Truth | Lesion Type |
---|---|

Shape | Oval |

Round | |

Irregular | |

Margin | Circumscribed |

Indistinct | |

Angular | |

Microlobulated | |

Spiculated | |

Orientation | Parallel |

Not parallel | |

Echo pattern | Anechoic |

Hypoechoic | |

Complex cystic and solid | |

Isoechoic | |

Hyperechoic | |

Heterogeneous | |

Posterior features | No features |

Enhancement | |

Shadowing | |

Combined pattern |

#### Appendix A.1. Derivation of ρ_{1}

#### Appendix A.2. The Limit of ${\widehat{\sigma}}_{1}^{2}$ under ${\overline{H}}_{1}$

#### Appendix A.3. Derivation of ρ_{2}

#### Appendix A.4. The Limit of ${\widehat{\sigma}}_{2}^{2}$ under ${\overline{H}}_{2}$

## References

- Zhang, Z.; Sejdic, E. Radiological images and machine learning: Trends, perspectives, and prospects. Comput. Biol. Med.
**2019**, 108, 354–370. [Google Scholar] [CrossRef] [PubMed] [Green Version] - DSickles, E.A.; D’Orsi, C.J.; Bassett, L.W.; Appleton, C.M.; Berg, W.A.; Burnside, E.S. ACR BI-RADS®Atlas, Breast Imaging Reporting and Data System; American College of Radiology: Reston, VA, USA, 2013. [Google Scholar]
- Wu, W.J.; Lin, S.W.; Moon, W.K. Combining support vector machine with genetic algorithm to classify ultrasound breast tumor images. Comput. Med. Imaging Graph.
**2012**, 36, 627–633. [Google Scholar] [CrossRef] [PubMed] - Liu, B.; Cheng, H.D.; Huang, J.; Tian, J.; Tang, X.; Liu, J. Fully automatic and segmentation-robust classification of breast tumors based on local texture analysis of ultrasound images. Pattern Recogn.
**2010**, 43, 280–298. [Google Scholar] [CrossRef] - Shan, J.; Cheng, H.D.; Wang, Y. Completely automated segmentation approach for breast ultrasound images using multiple-domain features. Ultrasound Med. Biol.
**2012**, 38, 262–275. [Google Scholar] [CrossRef] [PubMed] - Cheng, H.D.; Shan, J.; Ju, W.; Guo, Y.; Zhang, L. Automated breast cancer detection and classification using ultrasound images: A survey. Pattern Recogn.
**2010**, 43, 299–317. [Google Scholar] [CrossRef] [Green Version] - Wu, G.G.; Zhou, L.Q.; Xu, J.W.; Wang, J.Y.; Wei, Q.; Deng, Y.B.; Cui, X.W.; Dietrich, C.F. Artificial intelligence in breast ultrasound. World J. Radiol.
**2019**, 11, 19–26. [Google Scholar] [CrossRef] [PubMed] - O’Connell, A.M. Diagnostic Performance of An Artificial Intelligence System in Breast Ultrasound. J. Ultrasound Med.
**2021**. [Google Scholar] [CrossRef] [PubMed] - Liang, K.Y.; Zeger, S. Longitudinal data analysis using generalized linear models. Biometrika
**1986**, 73, 13–22. [Google Scholar] [CrossRef] - Emrich, I.J.; Piedmonte, M.R. A method for generating high dimensional multivariate binary variables. Am. Stat.
**1991**, 45, 302–304. [Google Scholar] - Jung, S.H.; Barnhart, H.X.; Sohn, I.; Stinnett, S.S.; Wallace, D.K. Sample Size for Comparing Correlated Concordance Rates. J. Biopharm. Stat.
**2008**, 18, 359–369. [Google Scholar] [CrossRef] [PubMed] - Qureshi, A.; Lakhtakia, R.; Bahri, M.A.; Al Haddabi, I.; Saparamadu, A.; Shalaby, A.; Al Riyami, M.; Rizvi, G. Gleason’s Grading of Prostatic Adenocarcinoma: Inter-Observer Variation Among Seven Pathologists at a Tertiary Care Center in Oman. Asian Pac. J. Cancer Prev.
**2016**, 17, 4867–4868. [Google Scholar] [PubMed]

**Table 1.**Sample size (empirical type I error rate, empirical power), $n(\widehat{\alpha},1-\widehat{\beta})$, under various design settings of (${p}_{r},{\delta}_{1},{\rho}_{1},1-\beta $) for the first type of study objective.

${\mathit{p}}_{\mathit{r}}$ | ${\mathit{\delta}}_{1}$ | ${\mathit{\rho}}_{1}$ | $1-\mathit{\beta}=0.8$ | $1-\mathit{\beta}=0.9$ |
---|---|---|---|---|

0.3 | 0.05 | 0.1 | $210(0.044,0.808)$ | $290(0.051,0.910)$ |

0.3 | $206(0.048,0.812)$ | $285(0.047,0.903)$ | ||

0.5 | $200(0.049,0.805)$ | $275(0.049,0.910)$ | ||

0.7 | $186(0.054,0.811)$ | $256(0.056,0.903)$ | ||

0.1 | 0.1 | $56(0.041,0.829)$ | $76(0.045,0.915)$ | |

0.3 | $55(0.047,0.823)$ | $75(0.042,0.914)$ | ||

0.5 | $53(0.048,0.822)$ | $73(0.052,0.921)$ | ||

0.7 | $50(0.061,0.822)$ | $68(0.060,0.913)$ | ||

0.5 | 0.05 | 0.1 | $249(0.047,0.804)$ | $344(0.048,0.904)$ |

0.3 | $245(0.051,0.808)$ | $338(0.053,0.901)$ | ||

0.5 | $237(0.045,0.812)$ | $327(0.050,0.907)$ | ||

0.7 | $220(0.053,0.798)$ | $304(0.050,0.904)$ | ||

0.1 | 0.1 | $66(0.052,0.815)$ | $90(0.048,0.911)$ | |

0.3 | $65(0.049,0.824)$ | $88(0.046,0.914)$ | ||

0.5 | $63(0.051,0.831)$ | $86(0.048,0.912)$ | ||

0.7 | $58(0.054,0.813)$ | $80(0.054,0.909)$ | ||

0.7 | 0.05 | 0.1 | $210(0.052,0.804)$ | $290(0.054,0.902)$ |

0.3 | $206(0.050,0.800)$ | $285(0.048,0.906)$ | ||

0.5 | $200(0.052,0.802)$ | $275(0.051,0.906)$ | ||

0.7 | $186(0.055,0.806)$ | $256(0.049,0.899)$ | ||

0.1 | 0.1 | $56(0.055,0.821)$ | $76(0.054,0.909)$ | |

0.3 | $55(0.055,0.816)$ | $75(0.049,0.904)$ | ||

0.5 | $53(0.052,0.814)$ | $73(0.054,0.911)$ | ||

0.7 | $50(0.060,0.821)$ | $68(0.058,0.912)$ |

**Table 2.**Sample size (empirical type I error rate, empirical power), $n(\widehat{\alpha},1-\widehat{\beta})$, under various design settings of (${p}_{x},{\delta}_{2},{\rho}_{2},1-\beta $) for the second type of study objective.

${\mathit{p}}_{\mathit{x}}$ | ${\mathit{\delta}}_{2}$ | ${\mathit{\rho}}_{2}$ | $1-\mathit{\beta}=0.8$ | $1-\mathit{\beta}=0.9$ |
---|---|---|---|---|

0.3 | 0.05 | 0.1 | $348(0.052,0.810)$ | $465(0.052,0.898)$ |

0.3 | $328(0.050,0.812)$ | $438(0.048,0.894)$ | ||

0.5 | $298(0.049,0.807)$ | $397(0.047,0.900)$ | ||

0.7 | $245(0.053,0.807)$ | $327(0.048,0.907)$ | ||

0.1 | 0.1 | $86(0.054,0.816)$ | $113(0.050,0.905)$ | |

0.3 | $81(0.053,0.820)$ | $107(0.047,0.912)$ | ||

0.5 | $74(0.047,0.825)$ | $98(0.047,0.914)$ | ||

0.7 | $63(0.053,0.844)$ | $83(0.048,0.934)$ | ||

0.5 | 0.05 | 0.1 | $434(0.050,0.798)$ | $580(0.048,0.902)$ |

0.3 | $409(0.050,0.810)$ | $546(0.049,0.904)$ | ||

0.5 | $370(0.049,0.806)$ | $495(0.052,0.905)$ | ||

0.7 | $304(0.054,0.802)$ | $406(0.050,0.905)$ | ||

0.1 | 0.1 | $111(0.053,0.816)$ | $148(0.053,0.909)$ | |

0.3 | $105(0.047,0.814)$ | $140(0.051,0.909)$ | ||

0.5 | $96(0.050,0.811)$ | $127(0.052,0.911)$ | ||

0.7 | $79(0.052,0.829)$ | $105(0.050,0.913)$ | ||

0.7 | 0.05 | 0.1 | $382(0.051,0.803)$ | $511(0.052,0.900)$ |

0.3 | $360(0.052,0.797)$ | $481(0.048,0.901)$ | ||

0.5 | $327(0.046,0.804)$ | $436(0.055,0.902)$ | ||

0.7 | $269(0.047,0.811)$ | $359(0.053,0.909)$ | ||

0.1 | 0.1 | $103(0.050,0.815)$ | $136(0.049,0.914)$ | |

0.3 | $97(0.054,0.817)$ | $129(0.049,0.912)$ | ||

0.5 | $89(0.050,0.821)$ | $117(0.050,0.917)$ | ||

0.7 | $74(0.048,0.840)$ | $98(0.045,0.925)$ |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Liu, L.; Parker, K.J.; Jung, S.-H.
Design and Analysis Methods for Trials with AI-Based Diagnostic Devices for Breast Cancer. *J. Pers. Med.* **2021**, *11*, 1150.
https://doi.org/10.3390/jpm11111150

**AMA Style**

Liu L, Parker KJ, Jung S-H.
Design and Analysis Methods for Trials with AI-Based Diagnostic Devices for Breast Cancer. *Journal of Personalized Medicine*. 2021; 11(11):1150.
https://doi.org/10.3390/jpm11111150

**Chicago/Turabian Style**

Liu, Lu, Kevin J. Parker, and Sin-Ho Jung.
2021. "Design and Analysis Methods for Trials with AI-Based Diagnostic Devices for Breast Cancer" *Journal of Personalized Medicine* 11, no. 11: 1150.
https://doi.org/10.3390/jpm11111150