Application of Cognitive Information Systems in Medical Image Semantic Analysis

Abstract

Cognitive information systems create a new class of intelligent systems focused on semantic data analysis tasks. Such systems are based on cognitive resonance processes, which use a knowledge-based perception model, to analyze and semantically classify visual data. Such systems can therefore be used for image analysis and classification, including semantic analysis of medical images, aimed at supporting diagnostic processes and determining the severity of lesions visualized by diagnostic imaging methods. This paper will describe various types of cognitive information systems designed for lesion recognition in selected abdominal and coronary structures, as well as skeletal parts of the human body, made visible by the application of various modalities in medical diagnostic imaging procedures. In this paper, a new generation of cognitive systems will also be described, and when compared to existing systems, will have the ability to perform extended cognitive resonance processes. Inference based on extended resonance inference allows the system to acquire additional knowledge, as well as expand the knowledge base used for semantic analysis. This paper will also propose the implementation of new efficient formal grammars, which increase the efficiency of lesion recognition in selected medical images to over 90%.

1. Introduction and Literature Review

Data analysis is inextricably linked to the process of understanding data. The description of interpreted data sets and information includes both the elements of statistical analysis of the data and the assessment of the impact of their meaning on a given area of knowledge. Interpretation of analyzed data sets is an attempt to assess the degree of significance of individual features contained in the analyzed data sets relative to the entire body of information that relates to the data. Feature extraction is therefore of vital importance in the entire process of data analysis, for it is on these features that the main analysis process will be based. The interpretation of the individual features of the analyzed dataset leads to a determination of the degree of their relevance to the entire dataset and an assessment of their significance and importance. The identification in the analyzed dataset of the features that are the most significant and have the greatest impact on the variability of the analyzed data is one of the basic steps in proper data analysis.

The process of proper data analysis consists of the following steps:

• Data acquisition and the assessment of the degree of relevance in the analysis process;
• Selection of the relevant data, elimination of the irrelevant data;
• Description of the interpreted data sets;
• Attempt to identify the characteristics of the analyzed data;
• Evaluation of the degree of importance of the various characteristics for the proved process of data analysis;
• Creation of a set of characteristics for the analyzed data;
• Meaningful description and interpretation of the data;

In addition:

8.

The ability to modify the input data set with the entire process to indicate the characteristics of the data.

The processes of meaningful description and interpretation of data are characteristic issues of meaningful data analysis. This kind of analysis is possible when we deal with the phenomenon of cognitive resonance [1,2,3,4]. This phenomenon occurs when there is a process of comparing a set of characteristics with a set of expert knowledge contained in knowledge bases. Thus, such a process of comparison occurs when an expert knowledge base is formed based on knowledge acquired from various sources. Based on the accumulated knowledge contained in the knowledge bases in the system, a process of generating expectations about the analyzed data occurs, and these expectations provide a certain benchmark for what is expected and what is not expected in the analysis process. These expectations make it possible, in a sense, to predict the results of the analysis process carried out and to adopt the solutions that are supported by the actual knowledge of the analyzed subject matter. The desirability of the creation of expert databases for their use in the process of predicting certain expected solutions and decisions is unquestionable and triggers the process of the correct evaluation of the results obtained. At the same time, it should be noted that the effectiveness of the process of generating expectations of anticipated possible solutions depends on the correctness of the expert knowledge base built [4].

The more extensive this set is, the more relevant the predictions about the analyzed data set can be. At the same time, a properly functioning system should provide opportunities for its continuous enrichment and expansion with new solutions and knowledge elements.

Expectations generated based on existing knowledge about the analysis process being carried out do not give the possibility for the full interpretation of the data [5,6]. The results are formed by comparing the expectations with the characteristics of the analyzed data sets. The construction of a set of such characteristics is possible based on a complete description of the characteristics that are typical of the analyzed datasets and that are not present in other datasets. This is because it allows for the unambiguous classification of the data sets based on a description that unambiguously identifies a class. Features that identify specific data sets and are also their characteristic markers are compared with expectations, that are generated based on expert knowledge. This phenomenon is called cognitive resonance (Figure 1) [4].

Cognitive resonance processes are based on the knowledge-based perception model [4]. It occurs when there is a process of comparing expectations and features, and as a result of this comparison it is possible to determine the correspondence between the pairs of features being compared and the expectations assigned to them. If such correspondences do not occur, then the whole process of meaningful analysis is not possible. The lack of congruence between these pairs is characteristic of cognitive dissonance.

Cognitive resonance leads to the next stage of data analysis, which is data understanding. The identification of compatible pairs of characteristics and expectations allows for the analyzed data to be fully attributed its unique characteristics and knowledge, and when contained in the relevant formal grammars, makes the inference system correctly recognize the analyzed data. The process of automatic interpretation of data will make it possible to understand the data based on the rules of grammatical inference based on the semantic description of the data contained in the formal definition. The process of understanding the data is carried out using the semantic rules contained in the grammatical definitions using linguistic formal language.

Formal grammars are used to implement the process of understanding data. The choice of the right type of grammar depends on the type of data it will be used to describe. The choice of the right grammar (string, tree, or graph) implies the use of the right path of data processing and interpretation in the analysis process [7].

Information systems designed for data understanding tasks have emerged as a new solution to facilitate automatic data understanding processes. They were discussed in the works of [8], and their advantage is that they allowed for the automatic understanding of data.

The process of the automatic understanding of data is carried out by cognitive systems. These systems are used to meaningfully describe and interpret data based on cognitive analysis, the main stage of which is the occurrence of the phenomenon of cognitive resonance.

Information systems based on the operation of the phenomenon of cognitive resonance in data analysis processes use the methods and techniques of structural inference to match patterns. The structure of the analyzed data (usually in the form of an image) is compared with the structure of the pattern defined in the system. Such a comparison is carried out using strings of inference rules to generate a pattern in an unambiguous way. These rules (production form) are defined in the definition of the grammar, defined by the analysis process being carried out. This grammar further defines the formal language. Recognized data are assigned to the class in which there is a pattern representing the class.

Cognitive analysis applied to cognitive systems is conducted using a syntactic approach. This approach uses functional blocks. The input image at the very beginning of the analysis process is preprocessed with stages of filtering and preprocessing, the approximation of shapes or positions of the analyzed objects, and the encoding of the data/image by the terminal components of the input language.

The implementation of preprocessing creates a new representation of the data in the form of hierarchical semantic tree structures and provides opportunities to derive this representation from the initial symbol of the grammar.

During data preprocessing, the cognitive system also conducts the process of segmentation and identification of the primary components, and further determines the relationships that exist between the aforementioned stages [7]. The classification stage consists of recognizing whether the representation of input data belongs to the class of data generated by a formal language, which is defined by a specific grammar. These grammars are included in the classes of string, tree, and graph grammars. The process of recognition using grammars is carried out during the semantic analysis carried out by the system.

In this way, cognitive analysis influences the process of understanding the data. Thus, cognitive information systems are systems for the meaningful interpretation of data and the semantic analysis of data.

Various classes of meaningful data analysis systems are described in the literature, which include such classes of systems as [9,10]:

• UBIAS systems (Understanding Based Image Analysis Systems)—dedicated to the process of understanding image data.
• E-UBIAS systems (Extended UBIAS)—designed for the task of meaningful analysis of image data, enriched with the system’s learning capabilities for new solutions.

The UBIAS and E-UBIAS class systems are designed for image data analysis, especially for medical image analysis.

This paper will describe the application of a new generation of cognitive systems that have the ability to perform extended cognitive resonance processes to analyze selected medical images. Inference based on extended cognitive resonance will enable the system to acquire additional knowledge, as well as expand the knowledge base used for the semantic analysis of images. This paper will also propose a definition of new graph grammars that increase the efficiency of lesion recognition in selected medical images to more than 90%.

The use of cognitive systems to analyze selected medical images allows for the creation of effective methods supporting medical diagnostics with an effectiveness slightly exceeding other advanced medical imaging analysis techniques. Among such advanced approaches, there are numerous applications of explainable AI [11,12] and complex NN with deep learning methods [13,14]. It is worth noting, however, that each of the above-mentioned approaches for the analysis of medical images involves different structures or data sets, which makes it difficult to compare them directly in terms of effectiveness but may allow for the creation of hybrid solutions supporting the diagnosis of the most difficult cases.

2. UBIAS and E-UBIAS Systems—Main Methodology

UBIAS class systems are dedicated to the task of meaningful analysis of image data with particular emphasis on the analysis of medical imaging.

The operation of UBIAS class systems is based on the use of linguistic techniques for the description of analyzed data sets. The description of medical imaging can be varied depending on the type of analyzed data. UBIAS systems are used to analyze various types of morphological changes in the analyzed structures visible on medical images. These images can be recorded using a variety of medical image recorders that make changes within the examined structure visible.

The most common forms of recording medical imaging are EEG, ECG, ultrasound, MRI, CT, and X-ray [9].

Instead, soft structures—tissues and bones—are examined. The possible lesions in individual tissues and bones are diverse, and their course varies for different patients. A peculiar feature of medical imaging is that, extremely often, images showing similar lesions indicate different anomalies or disease entities. The process of computer analysis and understanding of medical images is therefore extremely complex as it relates to the analysis of human health.

UBIAS systems have been used to analyze medical imaging for the interpretation of long bone fractures, wrist bone deformations and deformations of the bones of the hand and foot, changes in the coronary artery area, deformations of the central nervous system of the spinal cord, as well as lesions of the pancreas and liver.

Extensions of the UBIAS class systems are the E-UBIAS systems, the main features of which are the ability to teach the systems new solutions, based on new knowledge in the field of analyzed data and new diagnoses of occurring changes, deformations, and disease entities. E-UBIAS, like UBIAS, is used for the automatic interpretation of visible changes of analyzed structures on medical images. However, it also has the ability to analyze unfamiliar (undefined) solutions through the use of learning methods.

A characteristic feature of UBIAS systems is their high versatility. Image analysis systems are designed to analyze a variety of image types as well as a variety of features of these images. Systems of this class provide opportunities to develop new definitions and modify the originally introduced solutions.

UBIAS systems work properly based on the phenomenon of cognitive resonance, which for this class of systems is presented in Figure 2.

UBIAS systems are used to analyze medical images depicting various deformations or lesions appearing in a variety of human organs. These images undergo a process of analysis to reveal/indicate the area on the image where the observed change is visible. Such systems are used for the meaningful analysis of image data with the characteristic lesions imaged in different organs and in a variety of positions.

A schematic view of the UBIAS and E-UBIAS systems is shown in Figure 3 and Figure 4.

Figure 3 shows how the system works based on sufficient knowledge to complete the full analysis process, while Figure 4 shows an elaborate inference system in which a step of teaching the system new solutions has been added.

The proper meaning of the analysis of medical imaging is implemented using linguistic data description techniques. These techniques are used to introduce the proper definition of a formal grammar, the components of which depend on the type of grammar chosen.

With the use of string grammars, a set of terminal symbols, a set of non-terminal symbols, as well as a production set, and a starting symbol of the grammar are defined. String grammars find their application in the case of describing medical imaging, on which deformations of the edges of a given structure are analyzed, such as spinal deformations or bone fractures.

Graph grammars, on the other hand, find their application in cases where it is necessary to apply a description of the structure under study by means of a graph and to determine the emergence of particular relationships between the elements of a given structure. When using graph grammars, a set of terminal symbols, a set of non-terminal symbols, a set of productions, and a starting symbol of the grammar are defined.

The research material for the development of UBIAS and E-UBIAS class systems represented a variety of medical images depicting the diverse changes of the organs under study.

The UBIAS and E-UBIAS class systems have a wide range of applications. The special importance of these classes of cognitive information systems can be indicated in the field of medicine, where these systems are dedicated to the following tasks:

• Supporting the diagnostic process of patients;
• Recognizing a variety of lesions;
• Understanding the causes of the diagnosed disease’s state;
• Predicting the prognosis of diagnosed lesions;
• Planning the treatment process and the curative and convalescent process.

3. Results

The proposed systems of meaningful analysis and understanding of medical data include the analysis of various areas of the human body and various lesions visualized on medical images of these organs. Within the framework of this conducted research, the following medical images depicting the structure of successively cited organs of the human body were subjected to semantic analysis [10,15]:

• Central nervous system and spinal cord—X-ray images;
• Deformations of the bones of the foot—X-ray images for the various projections of the foot;
• Fractures of long bones—X-ray images;
• Deformations of the metacarpal bones—X-ray images;
• Deformations of the bones of the hand—X-ray images.

An example of the developed cognitive solution will be presented for the analysis of wrist bone and hand bone imaging.

For the correct analysis of the imaging of the wrist bones and the hand bones, the medical nomenclature of the individual bones was adopted for the correct structures of the wrist and hand.

The correct image of the bones of the hand is shown in Figure 5, in which the bones of the palm of the right hand were imaged, divided into the bones of the fingers, metacarpals, and wrist.

In order to define the corresponding representations using formal or graph grammars, it is necessary to introduce a new linguistic representation for the lesions under analysis. For this purpose, the medical image is subjected to preprocessing operations, the purpose of which are to clearly distinguish the different parts of the analyzed medical structure in the image. To achieve this, a number of image preprocessing operations such as thresholding, binarization, skeletonization, and edge detection are applied [7,16].

After applying the above-mentioned preprocessing operations, we usually obtain images that are slightly noisy and that show discontinuous contours of the individual hand bones. In order to obtain an image suitable for further analysis and the introduction of a linguistic representation, it is also necessary to use high-pass filtering, which eliminates individual artifacts from the image, as well as to use morphological operations to close the images and fill in the contours of the visible bone elements.

The result is an image that makes the contours of the studied structure visible with the help of which we can introduce a new representation in the form of a linguistic graph, as shown in Figure 6. This representation allows us to define all of the independent fragments of the studied structure, determine their number, as well as their mutual spatial relationships. These are important parameters, since disease processes can, on the one hand, change the shape of the studied structures, but also cause their disappearance (i.e., reduce the number of visible elements), or cause their movement (e.g., as a result of trauma or injury). Once the linguistic representation is in place and all of the required sets of grammars (i.e., vertex labels of the graph, edge labels, terminal elements, non-terminal elements, and relations between structures) are defined, it is possible to define sets of grammar rules that will describe normal structures and those with pathological changes, as shown in

Table 1. The names of hand bones.

Abbreviation	Latin Name	English Name
u	ulna	ulna
s	os scaphoideum	fibula
l	os lunatum	lunar bone
t	os triquetrum	triangular bone
p	os pisiforme	pea bone
tz	os trapezium	greater quadrilateral bone
tm	os trapezoideum	lesser quadrilateral bone
c	os capitatum	head bone
h	os hamatum	hook bone
m	os metacarpale	metacarpal bone
ses	ossa sesamoidea	mandible
pip	phalanx proximalis	proximal phalanx
pm	phalanx media	middle phalanx
pd	phalanx distalis	phalanx

and

Table 2. Elements of G_hand grammar definition.

Set Name	Set Symbol	Set Elements
Nh	non-terminal symbols	ST, ULNA, OS SCAPHODEUM, OS LUNATUM, OS TRIQUETRUM, OS PISIFORME, OS TRAPEZIUM, OS TRAPEZOIDEUM, OS CAPITATUM, OS HAMATUM, m1, m2, m3, m4, m5, SES1, SES2, PIP1, PIP2, PIP3, PIP4, PIP5, PM2, PM3, PM4, PM5, PD1, PD2, PD3, PD4, PD5
Th	terminal symbols	r, u, s, l, t, p, tm, tz, c, h, m1, m2, m3, m4, m5, s1, s2, pip1, pip2, pip3, pip4, pip5, pm2, pm3, pm4, pm5, pd1, pd2, pd3, pd4, pd5
Sh	start symbol	ST
Γh	finite set of edge labels of Ghand grammar	p, q, r, s, t, u, v, w, x, y, z
Ph	production set

.

The final step in linguistic analysis using graph grammars is the implementation of a parser, which will analyze all new cases and thanks to the use of a parser control table, will allow for their classification into one of the defined classes describing correct cases or showing pathological changes [5].

Having defined a graph grammar and implemented a parser for recognizing new diagnostic cases, the proper classification of new diagnostic images is carried out in such a way that the recorded images are subjected to the same sequence of image preprocessing operations, the aim of which is to obtain a linguistic representation of the examined image. After receiving it, this representation is checked by the parser to confirm compliance with the patterns representing individual disease entities. The result of the syntactic analysis confirms the type of the detected abnormality or the correct bone structures of the hand.

The linguistic description of the medical data analyzed is presented in the form of a formal grammar. The description is based on the accepted nomenclature of the bones of the hand pre-presented in Table 1.

Cognitive analysis is possible using a graph grammar, for which the analysis of X-ray images of palm bone images was defined as follows [4]:

$$G_{hand}=(N_h,T_h,{\mathsf{\Gamma }}_h,S_h,P_h)$$

A description of the various defining elements for the introduced graph grammar is shown in Table 2.

Another important part of the description and analysis process is:

• Defining the graph of relations occurring between the individual bones of the fingers, metacarpus, and wrist;
• Defining the graph of relations occurring between the individual bones of the hand, taking into account the angles of inclination between the individual bones.

Graphs illustrating the solutions introduced are shown in Figure 6.

Correct formal linguistic descriptions using proper graph grammar allows UBIAS-class systems to conduct the semantic interpretation of image data. Examples of the results obtained are shown in Figure 7.

The use of graph grammar allows not only the creation of a universal and robust representation of the examined structure, but also the introduction of a hierarchical representation of semantic information. This is a result of the appropriate definition of the set of edge labels in the defined grammar, which define the spatial relationship taking into account some approximation (Table 2). The effect of using certain ranges in defining edge labels is to obtain a fuzzy and robust representation of the examined structures.

The presentation of the obtained results of cognitive systems is intended to highlight the broad applicability of the developed solutions, in terms of supporting diagnostic, disease, and treatment processes.

It is worth noting that the spatial relationships between the individual bones of the hand or wrist result from the anatomical structure and mutual arrangement of the bone structures of the hand. The introduced graph representation reflects the anatomical proximity of the individual hand bones and the existence of articular ligaments within individual bone elements. The advantage of basing the graph representation on the anatomical structure allows for the efficient further identification of bone abnormalities and injuries. However, a certain disadvantage is that in the case of analyzing other images, e.g., the foot, it is necessary to take into account other spatial relations that will reflect the anatomical structure of this structure. In order to introduce a grammar description for any medical images, preprocessing techniques can be used, which will allow for the identification of individual elements and define the relationships between them.

Experiments related to the use of cognitive systems to analyze wrist X-ray images were carried out on a set of several dozen images available in online medical databases (e.g., Kaggle, Bone Fracture Vision Project, etc.), which included cases of normal images as well as various disease entities, including hand fractures and carpal bones, bone displacement, and loss of small bones due to abnormal metabolic processes. In particular, the analyzed images showed the following pathological changes: finger fractures, hand bone fractures, degenerative changes, hand deformations, bone displacements, arthritis, and the tuberculosis of bones. The analyzed images were preprocessed using a number of operations, such as thresholding, skeletonization, edge detection, and the determination of the centers of gravity of the bone elements visible in the image. Information about other possible disease cases regarding the structure and morphology of the wrist bones was also obtained in cooperation with radiologists and orthopedists. This made it possible to introduce very representative grammatical descriptions, taking into account both the analyzed structures and other possible disease cases. The last step was the implementation of syntactic analyzers that served as procedures for recognizing and classifying the meaning of disease cases.

The effectiveness of developed cognitive systems in the process of diagnosing existing disease lesions was determined as a measure of the compliance of the diagnostic description of the disease with its automatic diagnosis. This means that each diagnosed disease case already had a diagnostic description prepared by a specialist. The results obtained by the cognitive systems were compared with these descriptions to confirm their agreement or discrepancy. Coefficients and statistical measures determining the effectiveness of the analysis are presented in Figure 8 and Figure 9,

Table 3. Types of lesions in the bones of the hand and wrist.

Types of Lesions Bones of the Hand and Wrist	The Effectiveness of Cognitive Analysis in the Process of Diagnosing Lesions [%]
finger fractures	90
hand bone fractures	93.3
degenerative changes	92.9
hand deformations	90
bone displacements	90.9
arthritis	88.9
tuberculosis of bones	87.5

.

Series 1–7 show the different types of diseases observed on X-ray images of the hand and wrist bones, presented in Table 3.

The effectiveness of applied analysis in percentage terms is shown in Figure 9.

4. Discussion

The results obtained for the described cognitive systems which operate on extended cognitive inference indicate a high efficiency of the developed methods. This efficiency is measured by the degree of correctness of the recognized disease entities and the compatibility of the diagnosis process with the knowledge held in the system against the analyzed medical images. This efficiency is within 90.9%, which is a very good result, and exceeds efficiency reached by previous generation systems i.e., above 80%.

By comparing the results of the analysis of wrist bone radiographs obtained using the described cognitive systems with other methods of analyzing such radiographs, it is easy to notice that the methodology presented in this work concerns the comprehensive analysis of hand images and their semantic classification. Research conducted by other researchers usually concerns selected aspects of such analysis, e.g., preprocessing of hand images [10], or morphological analysis of individual bones. For example, the works [17,18] describe approaches for the analysis of selected morphometric parameters of the bones of the hand and wrist, which was carried out using simple methods of calculating distances and shape coefficients, achieving an efficiency of just over 60%. The conclusion is that the analysis of multi-object bone structures is extremely difficult, and the cognitive systems presented in this work offer a holistic approach that significantly improves the effectiveness of the analysis of hand images.

It should be pointed out, however, that only a few dozen medical images were used for the study and increasing this number and checking the results of the study on a larger pool of data will probably result in a more thorough evaluation of the entire analysis process and the obtained solution.

The directions of further research will be determined by other opportunities to analyze other types of medical imaging, diversified both in terms of the type of diagnostic tests performed with medical image recorders and medical images depicting other areas of the human body.

The presented methodology for using cognitive systems to analyze medical images also has some limitations. Among the most important of these is the high complexity of the syntactic analyzer generation procedure for graph grammars. In order to reduce such complexity to the level of multi-omics complexity, it is necessary to use grammars of the appropriate classes in the analysis, for which efficient methods have been developed for the creation of parsers of a low complexity class [4]. A certain challenge related to this is the automatic generation of graph grammars, which is also being researched, and in future should facilitate the generation of grammars describing given structures and slowly and automatically create parser control arrays, which are used in the process of pattern classification [19].

Among the undoubted advantages of using the grammar-based approach are the high scalability and robustness that result from the high generalizability of the analysis process of medical imaging thanks to the definition of spatial relationships in graph grammars. This makes it possible to classify quite different cases and put them into a common class describing similar diagnostic cases.

The use of cognitive systems, like the rapid development of AI, may raise some social or ethical questions. However, it is worth noting that in this case the cognitive system is not completely autonomous in making the diagnosis but plays a supporting role and assists the diagnostic procedure. The analysis carried out is very advanced and allows not only the detection of pathological changes, but also the determination of the significance of progressive disease processes. However, the system presents the results of the analysis to medical personnel who will ultimately be responsible for making any therapeutic decisions.

5. Conclusions

This work presents the possibilities of applying new cognitive systems to the task of analyzing medical images. The high efficiency of such analysis is due to the fact that it is based on the implementation of recognition processes based on the methods of mathematical linguistics. Meaningful description allows for the unambiguous classification of the analyzed data due to the use of those elements that semantically unambiguously define the analyzed disease cases. This unambiguity in data description and interpretation becomes the most important aspect of the analysis of such data sets, where unambiguous interpretation is essential. The most sensitive images, due to the relevance of the data being description, are medical images. Therefore, this paper presents examples of the use of cognitive systems to analyze selected medical imaging using UBIAS systems. These systems are still being developed and have the advantage of being used in the processes of understanding medical image data. In this article, we have proposed and extended cognitive resonance-based systems that have achieved a higher performance in the semantic analysis of medical data and have the ability to self-learn and expand a knowledge base through the use of more efficient graph grammars.

The self-learning capabilities of cognitive systems of the E-UBIAS class stem from the ability to continuously expand the knowledge base used in resonance processes with new recognized diagnostic cases. Each new classified pattern, after verification, can be placed in the knowledge base and can be used as a reference pattern in further analysis. Continuous expansion of the knowledge base also makes it possible to increase the efficiency of diagnosing pathological cases. This is due to the fact that cognitive processes are based on previously acquired experiences, and as new patterns are acquired, these experiences will increase and allow for diagnosis to be carried out with significantly increased efficiency.