Skip to Content
Applied SciencesApplied Sciences
  • Article
  • Open Access

27 July 2024

26 Pages

The Automated Generation of Medical Reports from Polydactyly X-ray Images Using CNNs and Transformers

,
,
and
1
Department of Computing, Federal University of Piauí—UFPI, Teresina 64049-550, PI, Brazil
2
EFREI Research Laboratory, École d’Ingénieurs Généraliste du Numérique—EFREI, 75003 Paris, France
*
Author to whom correspondence should be addressed.

Abstract

Pododactyl radiography is a non-invasive procedure that enables the detection of foot pathologies, as it provides detailed images of structures such as the metatarsus and phalanges, among others. This examination holds potential for employment in CAD systems. Our proposed methodology employs generative artificial intelligence to analyze pododactyl radiographs and generate automatic medical reports. We used a dataset comprising 16,710 exams, including images and medical reports on pododactylys. We implemented preprocessing of the images and text, as well as data augmentation techniques to improve the representativeness of the dataset. The proposed CAD system integrates pre-trained CNNs for feature extraction from the images and Transformers for report interpretation and generation. Our objective is to provide reports describing pododactyl pathologies, such as plantar fasciitis, bunions, heel spurs, flat feet, and lesions, among others, offering a second opinion to the specialist. The results are promising, with BLEU scores (1 to 4) of 0.612, 0.552, 0.507, and 0.470, respectively, a METEOR score of 0.471, and a ROUGE-L score of 0.633, demonstrating the model’s ability to generate reports with qualities close to those produced by specialists. We demonstrate that generative AI trained with pododactyl radiographs has the potential to assist in diagnoses from these examinations.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.