A quantitative comparison between Focal loss and Binary Cross-Entropy loss in Brain Tumor Auto-segmentation using U-net
-
Mahdi Shafiei Neyestanak
,
Hamid Jahani
,
Mohsen Khodarahmi
,
Javad Zahiri
,
Mostafa Hosseini
,
Amirali Fatoorchi
,
Mir Saeed Yekaninejad
Abstract
Purpose: Brain tumors are among the fatal cancers and cause the death of many people annually. Early diagnosis of a brain tumor can help save the patient’s life.
Method: We have collected a dataset consisting of 314 brain MRI images in all planes taken by giving a contrast medium with the dimension of 800*512, which offers the highest resolution. First, skull stripping has been implemented to separate the brain from other parts in the images. Next, we have annotated the tumors in the images under the supervision of experienced radiologists to create ground truth. To determine the most effective model versions for all three loss functions, hyperparameter tuning was performed. Following the comparison, the study further evaluates the effectiveness of two loss functions, Binary Cross-Entropy (BCE) and Focal loss, specifically in handling tumor regions within the dataset.
Result: The two proposed loss functions were evaluated using 5-fold cross-validation, and the average precision, recall, and f1 were 76.16%, 71.9%, and 74.52 for BCE loss and 82.92%, 79.32%, and 81% for the Focal loss on the test data, respectively. Moreover, the accuracy for BCE loss was 99.03% and 99.44% for the Focal loss.
Conclusion: We recommend using BCE loss cautiously in classification tasks without data imbalance and emphasize the adoption of Focal loss for more accurate and reliable results in brain tumor segmentation.
References
1. Khazaei Z, Goodarzi E, Borhaninejad V, Iranmanesh F, Mirshekarpour H, Mirzaei B, et al. The association between incidence and mortality of brain cancer and human development index (HDI): an ecological study. BMC Public Health. 2020 Nov 12;20(1):1696.
2. Ostrom QT, Gittleman H, Truitt G, Boscia A, Kruchko C, Barnholtz-Sloan JS. CBTRUS statistical report: Primary brain and other central nervous system tumors diagnosed in the United States in 2011-2015. Neuro Oncol. 2018 Oct 1;20(suppl_4):iv1–86.
3. Al-Okaili RN, Krejza J, Wang S, Woo JH, Melhem ER. Advanced MR imaging techniques in the diagnosis of intraaxial brain tumors in adults. Radiographics. 2006 Oct;26 Suppl 1(suppl_1):S173-89.
4. Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M, Kirby JS, et al. Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci Data. 2017 Sep 5;4(1):170117.
5. Borole VY, Nimbhore S, Kawthekar SS. Image Processing Techniques for Brain Tumor Detection: A Review. 2015 [cited 2024 Dec 18]; Available from: https://www.semanticscholar.org/paper/Image-Processing-Techniques-for-Brain-Tumor-A-Borole-Nimbhore/3975ab9ca18c1ef9d38c2a0b39efc703e6ebcd67
6. Thaha MM, Kumar KPM, Murugan BS, Dhanasekeran S, Vijayakarthick P, Selvi AS. Brain tumor segmentation using convolutional neural networks in MRI images. J Med Syst. 2019 Jul 24;43(9):294.
7. Liu Z, Tong L, Chen L, Jiang Z, Zhou F, Zhang Q, et al. Deep learning based brain tumor segmentation: a survey. Complex Intell Syst. 2023 Feb;9(1):1001–26.
8. Corso JJ, Sharon E, Dube S, El-Saden S, Sinha U, Yuille A. Efficient multilevel brain tumor segmentation with integrated bayesian model classification. IEEE Trans Med Imaging. 2008 May;27(5):629–40.
9. Clark MC, Hall LO, Goldgof DB, Velthuizen R, Murtagh FR, Silbiger MS. Automatic tumor segmentation using knowledge-based techniques. IEEE Trans Med Imaging. 1998 Apr;17(2):187–201.
10. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Commun ACM. 2017 May 24;60(6):84–90.
11. Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, et al. Brain tumor segmentation with Deep Neural Networks. Med Image Anal. 2017 Jan;35:18–31.
12. Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE; 2015. p. 3431–40.
13. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Lecture Notes in Computer Science. Cham: Springer International Publishing; 2015. p. 234–41. (Lecture notes in computer science).
14. Milletari F, Navab N, Ahmadi S-A. V-net: Fully convolutional Neural Networks for volumetric medical image segmentation [Internet]. arXiv [cs.CV]. 2016. Available from: https://www.semanticscholar.org/paper/V-Net%3A-Fully-Convolutional-Neural-Networks-for-Milletar%C3%AC-Navab/50004c086ffd6a201a4b782281aaa930fbfe6ecf
15. Ciresan D, Giusti A, Gambardella L, Schmidhuber J. Deep neural networks segment neuronal membranes in electron microscopy images. Pereira F, Burges CJ, Bottou L, Weinberger KQ, editors. Neural Inf Process Syst. 2012 Dec 3;25:2852–60.
16. Li G, Zheng Y, Cui J, Gai W, Qi M. DIM-UNet: Boosting medical image segmentation via diffusion models and information bottleneck theory mixed with MLP. Biomed Signal Process Control. 2024 May;91(106026):106026.
17. Liu Y, Wang H, Chen Z, Huangliang K, Zhang H. TransUNet＋: Redesigning the skip connection to enhance features in medical image segmentation. Knowl Based Syst. 2022 Nov;256(109859):109859.
18. Isensee F, Kickingereder P, Wick W, Bendszus M, Maier-Hein KH. Brain tumor segmentation and radiomics survival prediction: Contribution to the BRATS 2017 challenge. In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. Cham: Springer International Publishing; 2018. p. 287–97. (Lecture notes in computer science).
19. Liaw R, Liang E, Nishihara R, Moritz P, Gonzalez JE, Stoica I. Tune: A research platform for distributed model selection and training [Internet]. arXiv [cs.LG]. 2018. Available from: http://arxiv.org/abs/1807.05118
20. van Beers F, Lindström A, Okafor E, Wiering M. Deep neural networks with intersection over union loss for binary image segmentation. In: Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods [Internet]. SCITEPRESS - Science and Technology Publications; 2019. Available from: https://research.rug.nl/files/87088047/ICPRAM_2019_35.pdf
21. Kingma DP, Ba J. Adam: A method for stochastic optimization [Internet]. arXiv [cs.LG]. 2014. Available from: http://arxiv.org/abs/1412.6980
22. Lin T-Y, Goyal P, Girshick R, He K, Dollar P. Focal loss for dense object detection. In: 2017 IEEE International Conference on Computer Vision (ICCV). IEEE; 2017. p. 2999–3007.
23. Smith SM. Fast robust automated brain extraction. Hum Brain Mapp. 2002 Nov;17(3):143–55.
24. Zou KH, Warfield SK, Bharatha A, Tempany CMC, Kaus MR, Haker SJ, et al. Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol. 2004 Feb;11(2):178–89.
25. Smeulders AWM, Chu DM, Cucchiara R, Calderara S, Dehghan A, Shah M. Visual tracking: An experimental survey. IEEE Trans Pattern Anal Mach Intell. 2014 Jul;36(7):1442–68.
26. Tan M, Wu F, Kong D, Mao X. Automatic liver segmentation using 3D convolutional neural networks with a hybrid loss function. Med Phys. 2021 Apr;48(4):1707–19.
27. Huttenlocher DP, Klanderman GA, Rucklidge WJ. Comparing images using the Hausdorff distance. IEEE Trans Pattern Anal Mach Intell. 1993;15(9):850–63.
28. Pike VW. Considerations in the development of reversibly binding PET radioligands for brain imaging. Curr Med Chem. 2016;23(18):1818–69.
29. Kishida I, Nakayama H. Empirical study of easy and hard examples in CNN training [Internet]. arXiv [cs.CV]. 2019. p. 179–88. Available from: https://www.researchgate.net/publication/337808063_Empirical_Study_of_Easy_and_Hard_Examples_in_CNN_Training
30. Wong KCL, Moradi M, Tang H, Syeda-Mahmood T. 3D segmentation with exponential logarithmic loss for highly unbalanced object sizes [Internet]. arXiv [cs.CV]. 2018. Available from: https://www.semanticscholar.org/paper/3D-Segmentation-with-Exponential-Logarithmic-Loss-Wong-Moradi/88b3a6c3900b189f02cb8a3bfae95214f90ca585
31. Ullah F, Nadeem M, Abrar M, Al-Razgan M, Alfakih T, Amin F, et al. Brain Tumor Segmentation from MRI images using handcrafted convolutional neural network. Diagnostics (Basel). 2023 Aug 11;13(16):2650.
32. Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J. UNet++: A Nested U-Net Architecture for Medical Image Segmentation [Internet]. arXiv [cs.CV]. 2018. Available from: http://arxiv.org/abs/1807.10165
33. Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, et al. Attention U-Net: Learning where to look for the pancreas [Internet]. arXiv [cs.CV]. 2018. Available from: http://arxiv.org/abs/1804.03999
34. Huang D, Wang M, Zhang L, Li H, Ye M, Li A. Learning rich features with hybrid loss for brain tumor segmentation. BMC Med Inform Decis Mak. 2021 Jul 30;21(Suppl 2):63.
35. Salehi SSM, Erdogmus D, Gholipour A. Tversky loss function for image segmentation using 3D fully convolutional deep networks. In: Machine Learning in Medical Imaging. Cham: Springer International Publishing; 2017. p. 379–87. (Lecture notes in computer science).
36. Bauer S, Nolte L-P, Reyes M. Segmentation of brain tumor images based on atlas-registration combined with a Markov-Random-Field lesion growth model. In: 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. IEEE; 2011. p. 2018–21.
37. Menze BH, Van Leemput K, Lashkari D, Riklin-Raviv T, Geremia E, Alberts E, et al. A generative probabilistic model and discriminative extensions for brain lesion segmentation--with application to tumor and stroke. IEEE Trans Med Imaging. 2016 Apr;35(4):933–46.
38. Zhang W, Li R, Deng H, Wang L, Lin W, Ji S, et al. Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. Neuroimage. 2015 Mar;108:214–24.
39. Pereira S, Pinto A, Alves V, Silva CA. Brain tumor segmentation using Convolutional Neural Networks in MRI images. IEEE Trans Med Imaging. 2016 May;35(5):1240–51.
1. Khazaei Z, Goodarzi E, Borhaninejad V, Iranmanesh F, Mirshekarpour H, Mirzaei B, et al. The association between incidence and mortality of brain cancer and human development index (HDI): an ecological study. BMC Public Health. 2020 Nov 12;20(1):1696.
2. Ostrom QT, Gittleman H, Truitt G, Boscia A, Kruchko C, Barnholtz-Sloan JS. CBTRUS statistical report: Primary brain and other central nervous system tumors diagnosed in the United States in 2011-2015. Neuro Oncol. 2018 Oct 1;20(suppl_4):iv1–86.
3. Al-Okaili RN, Krejza J, Wang S, Woo JH, Melhem ER. Advanced MR imaging techniques in the diagnosis of intraaxial brain tumors in adults. Radiographics. 2006 Oct;26 Suppl 1(suppl_1):S173-89.
4. Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M, Kirby JS, et al. Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci Data. 2017 Sep 5;4(1):170117.
5. Borole VY, Nimbhore S, Kawthekar SS. Image Processing Techniques for Brain Tumor Detection: A Review. 2015 [cited 2024 Dec 18]; Available from: https://www.semanticscholar.org/paper/Image-Processing-Techniques-for-Brain-Tumor-A-Borole-Nimbhore/3975ab9ca18c1ef9d38c2a0b39efc703e6ebcd67
6. Thaha MM, Kumar KPM, Murugan BS, Dhanasekeran S, Vijayakarthick P, Selvi AS. Brain tumor segmentation using convolutional neural networks in MRI images. J Med Syst. 2019 Jul 24;43(9):294.
7. Liu Z, Tong L, Chen L, Jiang Z, Zhou F, Zhang Q, et al. Deep learning based brain tumor segmentation: a survey. Complex Intell Syst. 2023 Feb;9(1):1001–26.
8. Corso JJ, Sharon E, Dube S, El-Saden S, Sinha U, Yuille A. Efficient multilevel brain tumor segmentation with integrated bayesian model classification. IEEE Trans Med Imaging. 2008 May;27(5):629–40.
9. Clark MC, Hall LO, Goldgof DB, Velthuizen R, Murtagh FR, Silbiger MS. Automatic tumor segmentation using knowledge-based techniques. IEEE Trans Med Imaging. 1998 Apr;17(2):187–201.
10. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Commun ACM. 2017 May 24;60(6):84–90.
11. Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, et al. Brain tumor segmentation with Deep Neural Networks. Med Image Anal. 2017 Jan;35:18–31.
12. Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE; 2015. p. 3431–40.
13. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Lecture Notes in Computer Science. Cham: Springer International Publishing; 2015. p. 234–41. (Lecture notes in computer science).
14. Milletari F, Navab N, Ahmadi S-A. V-net: Fully convolutional Neural Networks for volumetric medical image segmentation [Internet]. arXiv [cs.CV]. 2016. Available from: https://www.semanticscholar.org/paper/V-Net%3A-Fully-Convolutional-Neural-Networks-for-Milletar%C3%AC-Navab/50004c086ffd6a201a4b782281aaa930fbfe6ecf
15. Ciresan D, Giusti A, Gambardella L, Schmidhuber J. Deep neural networks segment neuronal membranes in electron microscopy images. Pereira F, Burges CJ, Bottou L, Weinberger KQ, editors. Neural Inf Process Syst. 2012 Dec 3;25:2852–60.
16. Li G, Zheng Y, Cui J, Gai W, Qi M. DIM-UNet: Boosting medical image segmentation via diffusion models and information bottleneck theory mixed with MLP. Biomed Signal Process Control. 2024 May;91(106026):106026.
17. Liu Y, Wang H, Chen Z, Huangliang K, Zhang H. TransUNet＋: Redesigning the skip connection to enhance features in medical image segmentation. Knowl Based Syst. 2022 Nov;256(109859):109859.
18. Isensee F, Kickingereder P, Wick W, Bendszus M, Maier-Hein KH. Brain tumor segmentation and radiomics survival prediction: Contribution to the BRATS 2017 challenge. In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. Cham: Springer International Publishing; 2018. p. 287–97. (Lecture notes in computer science).
19. Liaw R, Liang E, Nishihara R, Moritz P, Gonzalez JE, Stoica I. Tune: A research platform for distributed model selection and training [Internet]. arXiv [cs.LG]. 2018. Available from: http://arxiv.org/abs/1807.05118
20. van Beers F, Lindström A, Okafor E, Wiering M. Deep neural networks with intersection over union loss for binary image segmentation. In: Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods [Internet]. SCITEPRESS - Science and Technology Publications; 2019. Available from: https://research.rug.nl/files/87088047/ICPRAM_2019_35.pdf
21. Kingma DP, Ba J. Adam: A method for stochastic optimization [Internet]. arXiv [cs.LG]. 2014. Available from: http://arxiv.org/abs/1412.6980
22. Lin T-Y, Goyal P, Girshick R, He K, Dollar P. Focal loss for dense object detection. In: 2017 IEEE International Conference on Computer Vision (ICCV). IEEE; 2017. p. 2999–3007.
23. Smith SM. Fast robust automated brain extraction. Hum Brain Mapp. 2002 Nov;17(3):143–55.
24. Zou KH, Warfield SK, Bharatha A, Tempany CMC, Kaus MR, Haker SJ, et al. Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol. 2004 Feb;11(2):178–89.
25. Smeulders AWM, Chu DM, Cucchiara R, Calderara S, Dehghan A, Shah M. Visual tracking: An experimental survey. IEEE Trans Pattern Anal Mach Intell. 2014 Jul;36(7):1442–68.
26. Tan M, Wu F, Kong D, Mao X. Automatic liver segmentation using 3D convolutional neural networks with a hybrid loss function. Med Phys. 2021 Apr;48(4):1707–19.
27. Huttenlocher DP, Klanderman GA, Rucklidge WJ. Comparing images using the Hausdorff distance. IEEE Trans Pattern Anal Mach Intell. 1993;15(9):850–63.
28. Pike VW. Considerations in the development of reversibly binding PET radioligands for brain imaging. Curr Med Chem. 2016;23(18):1818–69.
29. Kishida I, Nakayama H. Empirical study of easy and hard examples in CNN training [Internet]. arXiv [cs.CV]. 2019. p. 179–88. Available from: https://www.researchgate.net/publication/337808063_Empirical_Study_of_Easy_and_Hard_Examples_in_CNN_Training
30. Wong KCL, Moradi M, Tang H, Syeda-Mahmood T. 3D segmentation with exponential logarithmic loss for highly unbalanced object sizes [Internet]. arXiv [cs.CV]. 2018. Available from: https://www.semanticscholar.org/paper/3D-Segmentation-with-Exponential-Logarithmic-Loss-Wong-Moradi/88b3a6c3900b189f02cb8a3bfae95214f90ca585
31. Ullah F, Nadeem M, Abrar M, Al-Razgan M, Alfakih T, Amin F, et al. Brain Tumor Segmentation from MRI images using handcrafted convolutional neural network. Diagnostics (Basel). 2023 Aug 11;13(16):2650.
32. Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J. UNet++: A Nested U-Net Architecture for Medical Image Segmentation [Internet]. arXiv [cs.CV]. 2018. Available from: http://arxiv.org/abs/1807.10165
33. Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, et al. Attention U-Net: Learning where to look for the pancreas [Internet]. arXiv [cs.CV]. 2018. Available from: http://arxiv.org/abs/1804.03999
34. Huang D, Wang M, Zhang L, Li H, Ye M, Li A. Learning rich features with hybrid loss for brain tumor segmentation. BMC Med Inform Decis Mak. 2021 Jul 30;21(Suppl 2):63.
35. Salehi SSM, Erdogmus D, Gholipour A. Tversky loss function for image segmentation using 3D fully convolutional deep networks. In: Machine Learning in Medical Imaging. Cham: Springer International Publishing; 2017. p. 379–87. (Lecture notes in computer science).
36. Bauer S, Nolte L-P, Reyes M. Segmentation of brain tumor images based on atlas-registration combined with a Markov-Random-Field lesion growth model. In: 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. IEEE; 2011. p. 2018–21.
37. Menze BH, Van Leemput K, Lashkari D, Riklin-Raviv T, Geremia E, Alberts E, et al. A generative probabilistic model and discriminative extensions for brain lesion segmentation--with application to tumor and stroke. IEEE Trans Med Imaging. 2016 Apr;35(4):933–46.
38. Zhang W, Li R, Deng H, Wang L, Lin W, Ji S, et al. Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. Neuroimage. 2015 Mar;108:214–24.
39. Pereira S, Pinto A, Alves V, Silva CA. Brain tumor segmentation using Convolutional Neural Networks in MRI images. IEEE Trans Med Imaging. 2016 May;35(5):1240–51.
| Files | ||
| Issue | Vol 11 No 1 (2025): . | |
| Section | Articles | |
| Keywords | ||
| Brain tumor segmentation Deep Learning Convolutional Neural Network U-net- architecture Diagnosis MRI image | ||
| Rights and permissions | |
|
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
How to Cite
1.
Shafiei Neyestanak M, Jahani H, Khodarahmi M, Zahiri J, Hosseini M, Fatoorchi A, Yekaninejad MS. A quantitative comparison between Focal loss and Binary Cross-Entropy loss in Brain Tumor Auto-segmentation using U-net. JBE. 2025;11(1):15-35.
