A Protocol to Reduce Interobserver Variability in the Computed Tomography Measurement of Orbital Floor Fractures

Abstract

Orbital fracture detection and size determination from computed tomography (CT) scans affect the decision to operate, the type of surgical implant used, and postoperative outcomes. However, the lack of standardization of radiological signs often leads to the false-positive detection of orbital fractures, while nonstandardized landmarks lead to inaccurate defect measurements. We aim to design a novel protocol for CT measurement of orbital floor fractures and evaluate the interobserver variability on CT scan images. Qualitative aspects of this protocol include identifying direct and indirect signs of orbital fractures on CT scan images. Quantitative aspects of this protocol include measuring the surface area of pure orbital floor fractures using computer software. In this study, 15 independent observers without clinical experience in orbital fracture detection and measurement measured the orbital floor fractures of three randomly selected patients following the protocol. The time required for each measurement was recorded. The intraclass correlation coefficient of the surface area measurements is 0.999 (0.997–1.000) with p-value < 0.001. This suggests that any observer measuring the surface area will obtain a similar estimation of the fractured surface area. The maximum error limit was 0.901 cm² which is less than the margin of error of 1 cm² in mesh trimming for orbital reconstruction. The average duration required for each measurement was 3 minutes 19 seconds (ranging from 1 minute 35 seconds to 5 minutes). Measurements performed with our novel protocol resulted in minimal interobserver variability. This protocol is effective and generated reproducible results, is easy to teach and utilize, and its findings can be interpreted easily.

Computed tomography (CT) scan is the gold standard imaging modality for patients with orbital trauma [1,2]. It allows for the detection of orbital fractures and the delineation of its extent, the detection of optic canal injuries, as well as the visualization of orbital contents, foreign bodies, and any associated soft tissue injuries around the eye.

The need for surgical repair of orbital floor fractures is commonly based on findings of enophthalmos, diplopia, or restricted ocular motility [3]. As larger fractures are associated with an increased frequency of these sequelae [4,5,6,7], which may not be apparent initially, various studies have recognized the importance of measuring orbital defect size from CT imaging to indicate the need for surgery [8,9,10,11]. In particular, Burnstine [12] has described extensive orbital floor fractures greater than or equal to 50% of the orbital floor as causing latent enophthalmos. Therefore, orbital floor fractures greater than or equal to 50% of the orbital floor are an indication for nonemergent repair within 2 weeks [12]. In addition, Ploder et al. [13] have determined that an enophthalmos of 2 mm can be expected with 3.38 cm² of fracture area. If surgery is indicated, orbital defect size measurements from CT scans determine the dimensions of orbital implants used for surgery, which can affect postoperative outcomes.

Nonstandardization of radiological signs and orbital landmarks may lead to considerable variability in preoperative measurement of orbital fracture defects. Therefore, a standardized protocol for measuring the size of orbital floor defects was developed to minimize discrepancy. The aim of our study is to design a novel standardized protocol for CT detection and measurement of orbital floor fractures and to evaluate the interobserver variability among novice readers.

Materials and Methods

Computed Tomography Orbits Image Acquisition

Digital Imaging and Communications in Medicine (DICOM) images were acquired using the Image Guidance Surgery Protocol. Briefly, the CT orbits images were taken using a CT scanner (Philips CT Scanner, Amsterdam, The Netherlands) with a gantry tilt of 0 degrees and slice thickness of 2 mm. A MacBookPro7,1 (Apple Computer, Inc., Cupertino, CA) with a 13-inch monitor, display resolution of 1,280 × 800 pixels and pixels per inch (PPI) of 113 pixels, and running on a Mac OS X v10.6.8 (Apple Computer, Inc.) was used to operate a free DICOM viewer software program—OsiriX v4.1.2 (Pixmeo, Geneva, Switzerland). This software is available as a free download by anyone using a computer running Mac OS (Apple Computer, Inc.). Viewing and measuring of CT orbits were done using OsiriX (Pixmeo) after opening the CT coronal images and selecting the bone-specific Hounsfield units.

Detection of Orbital Fractures

Orbital fracture detection was standardized using direct and indirect radiological signs [1,2] (

Table 1. Protocol for the detection of orbital floor fractures on computed tomography scan images [1,2].

). It was emphasized that the indirect signs, although not diagnostic of a fracture, should warrant a closer inspection of the osseous margins of the orbits. When the indirect signs were not apparent in the bone window setting, soft tissue window setting was used for visualization.

The normal anatomical landmarks for orbital fracture detection and the normal boundaries of the orbital walls were highlighted to facilitate the standardization of orbital defect size measurement. In severely comminuted unilateral fractures where there is disruption of orbital architecture, the intact contralateral orbit was used as the reference. In bilateral orbital fractures, normal anatomy from an atlas was used for comparison.

Common pitfalls in CT interpretation of orbital fractures were also highlighted. In particular, the normal anatomy of the infraorbital foramen and of the zygomaticosphenoid and frontozygomatic sutures was emphasized as these could be confused with orbital fractures [14]. The novice readers were taught that any interruption of bone continuity or increased gap of diastases in sutures or foramen, even without displacement as in greenstick fractures, could also indicate a fracture.

Measurement of Orbital Fractures

The quantitative aspect of this protocol (

Table 2. Protocol for the measurement of orbital floor fractures on computed tomography scan images.

^aThe various tools are found in OsiriX (Pixmeo, Geneva, Switzerland).

) was modified from a study performed by Schouman et al. [15]. Using a slice thickness of 2 mm, a total of at least 10 pairs of points of interests were marked in all the coronal images showing the fracture. Three-dimensional (3D) reconstructions of the orbital floor fracture using the 3D Surface Rendering tool are displayed in Figure 7, to illustrate how the surface area of the orbital fracture was obtained. This can also allow the surgeon to visualize the orbital defect and assist in the creation of a personalized preformed orbital implant if surgery is necessary to reconstruct the defect.

Statistical Analysis

In this study, 15 independent nonmedical observers without clinical experience in orbital fracture measurements detected and measured the orbital floor fractures of three patients after being briefed on the qualitative and quantitative aspects of this protocol. These patients were randomly selected from a database comprising patients who had undergone open reduction and internal fixation for orbital fractures in the National University Health System from January 2000 to February 2010. The intraclass correlation coefficient (ICC) was used to calculate the interobserver variability of the surface area measurements using SPSS statistical package (SPSS, Inc, New York, NY). It is derived by comparing how the quantitative measurements made by the different observers resemble each other. ICC ranges between — 1 and + 1, with + 1 signifying a perfect agreement between measurements. An ICC value greater than 0.7 is generally regarded as being the result of highly consistent measurements. The time taken for each measurement was recorded as well.

Figure 1. Direct signs of orbital fractures. (a) Right orbital floor fracture with an abrupt change in bone density. (b) Right orbital floor fracture with an interruption of cortical bone continuity. (c) Abnormal angulation between the right orbital floor and a displaced bone fragment.

Figure 1. Direct signs of orbital fractures. (a) Right orbital floor fracture with an abrupt change in bone density. (b) Right orbital floor fracture with an interruption of cortical bone continuity. (c) Abnormal angulation between the right orbital floor and a displaced bone fragment.

Results

The ICC for the orbital floor fracture surface area measured by the 15 independent observers was 0.999 (0.997–1.000) with p-value <0.001 (

Table 3. Results of OsiriX (Pixmeo, Geneva, Switzerland) measurements.

). This suggests good agreement among the measurements, and that an observer measuring the surface area will obtain a similar estimation as the other observers. Furthermore, the p-value indicates that the results obtained are reproducible if the same experiment is repeated. The maximum error limit was 0.901 cm², which is less than the margin of error of 1 cm² for mesh trimming in orbital reconstruction. The size of the mesh used for orbital reconstruction was larger than the measurements obtained on OsiriX (Pixmeo) (

Table 4. Comparison between OsiriX (Pixmeo, Geneva, Switzerland) measurements and size of mesh used.

). The average duration required for each measurement was 3 minutes 19 seconds (ranging from 1 minute 35 seconds to 5 minutes) (

Table 5. Duration taken for each measurement.

).

Discussion

Indications for Surgery in Orbital Floor Fractures

CT is recognized to be the gold standard imaging technique for assessing the severity of orbital fractures because of the complexity of the anatomy of the orbital and ethmoidal regions. Indications for surgical exploration include (1) presence of enophthalmos during the early stage, (2) diplopia for more than 6 weeks especially in the primary or downward gaze position, (3) restricted ocular motility, (4) large size of fracture, (5) substantial number of orbital walls involved, and (6) CT appearance of an extensive area of depressed and comminuted fracture of the orbital floor or entrapment of the inferior rectus muscle [16].

Advantages of Our Protocol

Other methods of measuring orbital floor defects using CT scans have been described, but they are not as comprehensive as our novel protocol. Most studies such as article by Hwang et al. [17] on the long-term epidemiological analysis of the natural history of orbital bone fractures do not elaborate on the method used to obtain the size of the orbital defects. Baumann et al. [18] estimated the fracture defect by the number of coronal slices where the fracture was visible on CT scan images, but this method was only a semiquantitative measurement. Manchio et al. [3] measured the maximum fracture width on the single coronal image that was felt to represent the greatest medial to lateral fracture extent [19] and the maximum fracture depth by counting the number of coronal slices where a fracture was observed and multiplying it by the known image spacing. However, the maximum width or depth may not be accurate if the slice thickness of the CT scan images is too wide. Furthermore, knowledge of the maximum fracture width and depth is not as useful clinically as knowledge of the surface area. Wang and Wang [20] calculated the fracture percentage by dividing the length of the fracture defect by the total length of the medial orbital wall in an axial section. It may be argued that the percentage of fractured orbital walls may be a more powerful parameter for decision making [15], instead of the absolute value of the defect’s surface area, due to the wide range of interindividual dimensional variations of orbital walls. However, our method can be easily expanded to consider the percentage of fractured orbital walls if one desires, by making an additional measurement of the entire length of the orbital wall. The disadvantage associated with all of the earlier methods is that the surface of the defect is extrapolated from a few linear measurements rather than from the precise measurement of the entire area of the defect.

Figure 2. Indirect signs of orbital fractures. (a) Coronal view of a left orbital floor fracture with opacification of the left maxillary sinus. (b) Presence of an air-fluid level in the right sphenoid sinus. (c) Intraorbital and periorbital subcutaneous emphysema in a bilateral orbital fracture. (d) Surrounding soft tissue edema in a patient with multiple facial fractures.

Figure 2. Indirect signs of orbital fractures. (a) Coronal view of a left orbital floor fracture with opacification of the left maxillary sinus. (b) Presence of an air-fluid level in the right sphenoid sinus. (c) Intraorbital and periorbital subcutaneous emphysema in a bilateral orbital fracture. (d) Surrounding soft tissue edema in a patient with multiple facial fractures.

Figure 3. Limits of the orbital floor anteriorly are bounded medially by the septum between the ethmoid and maxillary sinuses at point A, and laterally by the most inferior and lateral aspect of the zygomatic portion of the orbital floor at point B.

Figure 3. Limits of the orbital floor anteriorly are bounded medially by the septum between the ethmoid and maxillary sinuses at point A, and laterally by the most inferior and lateral aspect of the zygomatic portion of the orbital floor at point B.

Figure 4. Limits of the orbital floor posteriorly are bounded medially by the septum between the ethmoid and maxillary sinuses at point A, and laterally by the medial edge of the inferior orbital fissure at point C.

Figure 4. Limits of the orbital floor posteriorly are bounded medially by the septum between the ethmoid and maxillary sinuses at point A, and laterally by the medial edge of the inferior orbital fissure at point C.

Figure 5. Marking out the edges of the fracture. (a) Left orbital floor fracture with fracture size measured from the superior aspects of the fracture limits. (b) Stacking the reconstructed axial views onto each other by increasing the slab thickness, to view all the points of interest in a single layer. (c) Magnifying the area bounded by the points of interest by zooming in such that it fits the entire screen, to get a more accurate measurement by taking into account any irregular edges.

Figure 5. Marking out the edges of the fracture. (a) Left orbital floor fracture with fracture size measured from the superior aspects of the fracture limits. (b) Stacking the reconstructed axial views onto each other by increasing the slab thickness, to view all the points of interest in a single layer. (c) Magnifying the area bounded by the points of interest by zooming in such that it fits the entire screen, to get a more accurate measurement by taking into account any irregular edges.

Figure 6. Connecting the previously placed points of interest and measuring the area bounded by them to get the surface area of the defect.

Figure 6. Connecting the previously placed points of interest and measuring the area bounded by them to get the surface area of the defect.

Figure 7. Connecting all the points of interests allows the surface area of the orbital floor fracture to be measured.

Figure 7. Connecting all the points of interests allows the surface area of the orbital floor fracture to be measured.

There are advantages to our methodology. First, only basic clinical and computer knowledge is required to make the relevant measurements. As mentioned earlier, the quantitative portion of this protocol has been shared by Schouman et al, [15] who obtained consistent measurements by three specialists using the CT scan images of 10 patients. By introducing a qualitative portion for the detection of orbital fractures, as well as modifying the quantitative portion by adding additional details such as magnifying the area bound ed by the points of interest (Figure 5), we were able to obtain consistent measurements by 15 independent observers who had no prior experience in reading CT scan images. Second, one only needs a basic CT viewing software program that allows linear measurements to be made (OsiriX; Pixmeo), therefore minimizing costs as no additional software needs to be purchased. Third, our protocol facilitates the measurement of orbital wall defects with a lower degree of interobserver variability. Fourth, it takes less time compared with other methods [15,21], as all our observers took approximately 3 minutes for each measurement. This includes the time taken for identifying the fracture and for making the measurements. Fifth, one does not need a high-resolution monitor to make the surface area measurements. The display resolution and PPI of monitors generally range from 1,280 × 800 and 113 pixels, respectively, on a 13-inch monitor to 2,880 × 1,800 and 220 pixels, respectively, on a 15-inch monitor. We could obtain surface area measurements on a low-resolution monitor by placing relevant points of interest on the CT coronal images without encountering any difficulty. Therefore, one does not need a high-resolution monitor and can make with any low-resolution monitor that is available in the market.

Our analysis is based on the assumption that the orbital floor is a plane structure when calculating the surface area, although in reality, the orbital floor is a 3D structure with variability in the contours. Our main objective was to modify a previously published method [15], to create a more reproducible method of measuring orbital fractures which can be used by any novice readers. In selected cases where the orbital floor or medial wall fracture is exposed in its totality, a correlation of intraoperative findings with the findings of clinicians and radiologists was also proposed to ascertain the accuracy of orbital defect size measurement. These improvements can be considered in the future.

Measurement of Orbital Defect Area and Orbital Volume

Jin et al. [6] used a formula to calculate the area of medial wall fractures by using the height and length of the defect. The formula they used makes the assumption that all orbital floor fractures are elliptical, thus compromising the accuracy and reliability of their method. Jaquiéry et al. [22] described a semiquantitative assessment based on assigning orbital fractures into two-dimensional diagrams obtained by defolding a 3D orbital model, using information from axial and coronal CT scan images. Their method is time consuming [15] and was associated with greater interobserver variability.

There is no decisional pathway for conservative versus surgical treatment in orbital floor fractures, and the final decision is based on the clinical and radiological information available for the surgeon. Modern computer-based programs have been developed for calculating the area of bony defects and the volume of displaced tissue in orbital floor fractures based on CT scan results [15,21,23,24,25,26,27]. Ploder et al. [21] reported an accurate quantitative computational method based on the measurement of the defect’s width on every coronal section. The width of the defect on each slice was then multiplied by the slice thickness and summed up to get the total surface area of the defect. However, their method is too time consuming (3–12 minutes) to be used as part of a standard posttraumatic evaluation of orbital fractures [15].

Ploder et al. [13] went on to compare the calculated data with the amount of enophthalmos, the presence of diplopia, and the limitation of ocular motility, and found out that the fracture area of 3.38 cm² and 1.62 mL volume of displaced tissue was correlated with enophthalmos [6,13] and diplopia [13]. Charteris et al. [16] also noted retrospectively that there was a significant difference in orbital volume between patients managed conservatively or surgically. Orbital defect area can affect the decision for conservative versus surgical treatment [11], especially for patients without severe clinical signs and symptoms. It can provide an adjunct parameter alongside clinical signs in deciding the need for surgery, as well as to give an estimation of the dimensions and type [11] of the preformed orbital implant if surgery is required. Furthermore, it has sometimes been used as an independent quantitative criterion for surgical decision making [3,11]. Measurement of orbital volume can provide information in planning orbital volume [28], replacement in delayed surgery for orbital trauma and assessing postoperative outcomes, including the positioning of orbital implants.

Clinical judgment remains the mainstay in the evaluation of orbital fracture defects. Our novel protocol will be useful as an adjunct in detecting orbital fracture as well as minimizing interobserver variability in CT measurements of orbital floor fractures by novice readers.

Other Imaging Methods

Ultrasound has been used to detect orbital fractures because it is cheaper than CT scans, has no radiation, and is widely available. However, ultrasound does not yield the same diagnostic quality as CT scans, as it is operator dependent, is associated with high false positives, and is unnecessary in patients presenting with orbital trauma and other associated injuries which may require a CT investigation [29].

New multislice CT (MSCT) allows quick acquisition of coronal slices without the need for neck hyperextension, which is unsafe in patients with known or suspected cervical spinal injury [30]. The direct acquisition of thin-cut coronal imaging enhances diagnostic sensitivity and specificity of orbital fractures. Although MSCT is valuable for investigation of posttraumatic changes affecting the dimensions and volume of the orbit and its contents, it still involves extensive radiation to the lens, especially when multiple readings are taken. For example, when scanning both the maxilla and mandible, the effective dose is approximately 1,066.1 µSv [31] for an eight-slice MSCT.

A mobile CT device with improved resolution has recently been developed to allow intraoperative evaluation of the immediate results of orbital reconstruction and perhaps decrease the need for further operations [32]. Intraoperative CT visualization can be paired up with the forced duction test and clinical examination as intraoperative criteria for judging reduction results. A major disadvantage lies in the increased exposure of the patient to additional radiation [33].

Reformatted 3D-CT scan also images allow a quick overview and enhanced perception of orbital fractures and adjacent facial fractures, facilitating the surgeon in formulating a more efficient treatment plan [34,35]. The area and volume calculation of the orbit can be evaluated using 3D-CT software such as Analyze (Biomedical Imaging Resource of the Mayo Clinic, Rochester, MN). 3D-CT software is however limited by its need for further mathematical conversion to estimate the fracture area, by volume averaging and threshold errors that may lead to misinterpretation, by the lengthy amount of time needed to analyze the CT dataset, and by the 7 to 8% volume difference between the two orbits that occurs normally [36]. However, it may have a role in cases of corrections of posttraumatic enophthalmos, as the method of 3D reconstruction of the CT scan can support the treatment plan concerning surgical correction and loss of orbital volume [37]. The area and volume calculations offer reliable preoperative data to assist with diagnosis and guide presurgical planning.

3D magnetic resonance imaging (MRI) of the orbit is a superior aid to MSCT for decision making regarding secondary volume reduction in enophthalmos correction and evaluation of persistent diplopia [38]. Dynamic contrast-enhanced MRI [39] has been used to define the extent of morphological changes in the extraocular muscles after orbital fractures, as it relates to potential long-lasting paralysis. Limitation of muscle action is characterized by a lack of increase in the sectional area and volume in respective gaze intervals, and provides a quantitative analysis of the degree of muscle weakness, helping surgeons make an informed decision regarding further treatment. MRI is too expensive for usage in daily practice.

Conclusion

Our novel protocol in CT measurements of orbital fractures is easy to teach and utilize, and can be applied in clinical situations easily. This protocol can assist a novice reader in detecting an orbital floor fracture and quantifying it easily with a low degree of interobserver variability. It can be used routinely to standardize the posttraumatic evaluation of orbital floor fractures preoperatively. Orbital defect area can affect the decision for conservative versus surgical treatment, especially for patients without severe clinical signs and symptoms. It can also provide an adjunct parameter to estimate the dimensions of the orbital implant required if surgery is deemed to be necessary.