Acoustic Metadata Design on Object-Based Audio Using Estimated 3D-Position from Visual Image Toward Depth-Directional Sound Image Localization

Abstract

Multichannel audio is a sound field reproduction technology that uses multiple loudspeakers. Object-based audio is a playback method for multichannel audio that enables the construction of sound images at specified positions using coordinates within the playback space. However, the sound image positions must be manually specified by audio content creators, which increases the production workload, especially for works containing many sound images or feature films. We have previously proposed a method to reduce the workload of content creators by constructing sound images based on object positions in visual images. However, a significant challenge remains since depth localization of the sound image is not accurate enough. This paper aims to improve localization accuracy by changing the range of sound image movement along the depth direction. To confirm the localization accuracy of sound images constructed using the proposed method, we conducted a subjective evaluation experiment. The experiment identified the optimal movement range by presenting participants with visual images synchronized with sound images moving across varying spatial scales. Consequently, we were able to identify the range of sound image movement in the depth direction necessary for presenting sound images with high consistency with the visual images.

1. Introduction

In recent years, with the advances of video technologies, highly immersive audio technologies for reproducing realistic sound fields have attracted increasing attention [1]. Multichannel audio is a key technology for presenting highly immersive sound to listeners [2,3,4]. Multichannel audio reproduces sound arriving from various directions by arranging multiple loudspeakers around the listener. Object-based audio is a playback method for multichannel audio [5,6,7,8,9]. In object-based audio, sound image positions can be specified by using coordinates within the playback space. The sound image position is described in the acoustic metadata along with temporal information [10]. By treating these coordinates as time-series data, object-based audio enables flexible specification of sound image positioning and movement over time. Due to this characteristic, object-based audio has recently gained attention in film production as a technology for creating sound spaces that are highly consistent with the visual images [11].

However, there is a problem in film production using object-based audio. In film production employing object-based audio, content creators specify the sound image positions to match the objects within the visual images. The design of acoustic metadata poses a significant burden on content creators for film productions with many objects or for feature films.

To address these issues, various methods have been proposed ranging from specialized recording techniques to automatic generation of object-based audio from existing multichannel audio content. However, many approaches depend on specialized hardware, including RGB-D and panoramic cameras [12,13,14,15,16,17,18,19,20], which greatly limits their applicability. As a result, integration into filmmaking workflows remains difficult, and adoption in real-world production environments faces substantial barriers.

Therefore, we propose a method that reduces the creators’ workload by designing acoustic metadata based on object coordinates estimated from visual images [21,22]. Our method requires no additional hardware, relying only on standard video and audio sources. As a result, it avoids the equipment-specific implementation barriers noted in prior work. It is also highly compatible with existing production workflows, facilitating integration.

In our previous study, we have proposed a method that estimates the positions of moving objects in a visual image using the frame subtraction method [23] and utilizes the obtained positional information for the sound image construction of object-based audio [21]. However, object position estimation based on the frame subtraction method can only determine positions in the horizontal and vertical directions on a monitor placed in front of the viewer. Consequently, the sound image position cannot be controlled in the depth direction, leading to the problem of low localization accuracy along the depth axis.

To address this, we introduced object coordinate estimation using Dist-YOLO [24] for sound image position control in the depth direction [22]. Dist-YOLO is a machine learning model capable of estimating three-dimensional object coordinates, including depth, from two-dimensional visual images. The object coordinates estimated by this model are mapped from screen coordinate system to the coordinate system of the object-based audio reproduction space. This conversion is based on a linear mapping of Vector Base Amplitude Panning (VBAP) [25]. This mapping enables three-dimensional sound image control, including the depth dimension. In this method, the sound image is constructed by normalizing the estimated time-series depth data and setting a movement range where the farthest point is the monitor position and the nearest point is the listening position (the viewer’s head). However, evaluations revealed that this approach still suffers from lower sound image localization accuracy in the depth direction compared to the horizontal and vertical directions. This paper aims to further improve sound image localization accuracy in the depth direction. In our conventional method, the range of sound image movement was set from the position of the monitor in front of the viewer to the viewer’s head. In contrast, this paper proposes a method that restricts the nearest point of the sound image to a position closer to the monitor to improve localization accuracy. Specifically, we introduced a new coefficient to control the movement range into the definition formula for the sound image’s depth position.

In general, the ventriloquism effect [26,27,28], a well-known phenomenon in audio-visual integration, is also effective along the depth direction [29]. However, this effect diminishes as the physical distance between the visual and auditory stimuli increases. Therefore, an improvement in localization accuracy is expected by limiting the sound image movement to the vicinity of the monitor and suppressing the spatial discrepancy with the visual image. The proposed method considers only a single visible object within the camera’s field of view. It does not handle multiple objects, objects that enter or leave the frame, or occlusions. Finally, through evaluation experiments, we verified the consistency between the visual image and the sound image and identified the movement range that yields the highest localization accuracy.

The remainder of this paper is organized as follows. Section 2 presents the related works of the proposed method. Section 3 introduces conventional production method on object-based audio. Section 4 describes the proposed method, while Section 5 discusses objective evaluation experiment results. Section 6 discusses subjective evaluation experiment. Finally, Section 7 concludes the paper.

2. Related Works

Multichannel audio technology enables the reproduction of highly immersive sound fields. However, managing the large number of audio signals complicates the production workflow compared with traditional stereo audio [3]. In multichannel audio, a wide range of approaches has been investigated throughout the workflow from production to reproduction to enhance interactivity and presence [12,15,16,17,18,19,20]. This section focuses on object-based audio and reviews prior work on content-production methods that leverage this technology.

Coleman et al. (2018) [12] proposed an end-to-end pipeline for object-based audio. They introduced Objectification, a process that automatically estimates source-position metadata by integrating visual and auditory information. The system acquires environmental signals using RGB-D cameras [13] and binaural microphones [14]. Applying blind source separation and beamforming to these signals enables the extraction of specific sources as discrete objects that can be rendered dynamically as a function of listener position.

Arteaga et al. (2021) [15] proposed a deep-learning method for extracting individual audio objects from existing multichannel mixes. The key feature is multichannel-based learning, which uses pre-rendered multichannel material as training data rather than isolated object stems. This strategy enables efficient object extraction in complex acoustic scenes, such as film production, where many objects are intricately mixed.

Shi et al. (2024) [16] proposed a low-cost, efficient system for generating spatial audio from panoramic (360°) video to streamline VR content production. The method estimates three-dimensional sound source positions by applying object detection and depth estimation to video frames, while a scene-classification module selects appropriate background sounds. Using the inferred spatial information, the system upmixes existing stereo audio to a 5.1.4 ch. format via Vector Base Amplitude Panning (VBAP) [25]. This approach substantially reduces production time relative to conventional workflows.

However, these methods rely on specialized equipment such as RGB-D cameras, panoramic cameras, and binaural microphones, which are not standard in traditional film production. As a result, integrating them into established workflows remains challenging, and barriers to adoption remain significant in actual film production.

3. Conventional Production Method for Object-Based Audio

In recent years, object-based audio has been widely adopted in film production. Channel-based audio has traditionally relied on assigning sound to specific loudspeakers [2,3,4]. In contrast, object-based audio treats sound as an object, designing metadata for each object, such as positional information, movement information, and loudness. This feature allows content creators to freely control the positioning and movement of sound images, enabling them to construct their intended sound space with precision.

Figure 1 shows the workflow of film production using object-based audio. Content creators manually design the acoustic metadata by defining the positions and movements of sound images synchronized with the visual images.

The acoustic metadata is standardized as the Audio Definition Model under the ITU-R BS.2076, which is based on XML format [10,30,31,32,33,34]. In this paper, the Dolby Atmos Master File (DAMF) is used as the acoustic metadata format [35].

Table 1. Object setting parameters included in the DAMF metadata.

Setting	Explanation
samplePos	Sample position at which the setting is applied
pos	Sound image position
active	Object active state
snap	Force the sound image to localize to the nearest loudspeaker
elevation	Presence or absence of panning in the vertical direction
zones	Specify the loudspeaker zone used to render the sound image
size	Sound image size
decorr	Sound spreading
importance	Object importance
gain	Object volume
rampLength	Time required for a setting change to complete
trimBypass	Downmix setting for different formats
dialog	Dialog flag
music	Music flag
screenFactor	Pulls the lateral sound image position toward the screen
depthFactor	Depth factor for binaural rendering
headTrackMode	Head-tracking mode for headphone playback
binauralRenderMode	Binaural rendering mode

shows the object setting parameters included in the DAMF metadata [36].

The acoustic metadata is designed using a Digital Audio Workstation (DAW) [37,38,39]. Figure 2 illustrates the 3D Panner, an acoustic metadata design tool in Pro Tools [37]. The position of each sound image can be adjusted by moving the red sphere shown in Figure 2. Through the individual control of sound images, the creators’ intended acoustic space can be precisely reproduced.

However, the control of each individual sound images remain a manual process. As a result, the production workload increases in large-scale projects, such as feature films or works involving many sound images. Furthermore, acquiring the specialized knowledge required for object-based audio mixing is difficult, as it often demands certification and training through dedicated curricula [40].

Therefore, the proposed method automatically generates positional information within the acoustic metadata design process, based on object positions estimated from visual images, which has conventionally imposed a significant burden on creators. This approach aims to support acoustic metadata design and reduce the creators’ production workload.

4. Proposed Method

Figure 3 shows an overview of the proposed method. In the proposed method, sound images are constructed using the object positions estimated from the visual images. The proposed method consists of the following three steps:

• Step 1: Estimation of the object coordinates from the visual image
• Step 2: Conversion of the estimated coordinates
• Step 3: Design of acoustic metadata from the converted coordinates

Figure 3. Overview of the proposed method.

Figure 3. Overview of the proposed method.

4.1. Step 1: Estimation of the Object Coordinates from the Visual Image

In Step 1, the 3D coordinates $(x_l^{\prime },y_l^{\prime },z_l^{\prime })$ of an object in the l-th visual image frame are estimated. Dist-YOLO is employed for this estimation [24]. Dist-YOLO is a machine learning model that estimates object positions, along with depth information for each visual image frame. As shown in Figure 4, this model outputs bounding boxes for each detected object. In this study, we calculate the center of the bounding box and designated as the object’s coordinates $x_l^{\prime }$ and $y_l^{\prime }$. These coordinates are calculated using the following equations:

$$\begin{array}{cc}\hfill x_l^{\prime }& ={\displaystyle \frac{1}{2}}(x_{l,\mathrm{top}}^{\prime }+x_{l,\mathrm{bottom}}^{\prime }),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill y_l^{\prime }& ={\displaystyle \frac{1}{2}}(y_{l,\mathrm{top}}^{\prime }+y_{l,\mathrm{bottom}}^{\prime }),\hfill \end{array}$$

(2)

where $(x_{l,\mathrm{top}}^{\prime },y_{l,\mathrm{top}}^{\prime })$ are the top-left coordinates of the bounding box in the l-th visual image frame, and $(x_{l,\mathrm{bottom}}^{\prime },y_{l,\mathrm{bottom}}^{\prime })$ are the bottom-right coordinates of the bounding box in the l-th visual image frame. The 3D-coordinates $(x_l^{\prime },y_l^{\prime },z_l^{\prime })$ are formed by combining $x_l^{\prime }$ and $y_l^{\prime }$ with the depth value $z_l^{\prime }$, which Dist-YOLO also estimates. This method is designed for scenes containing a single object. For this reason, we do not define object classes, and all detections are handled as the same object regardless of the assigned category.

4.2. Step 2: Conversion of the Estimated Coordinates

In Step 2, the estimated object coordinates are transformed into a coordinate system suitable for object-based audio. As shown in Figure 3, the playback space is defined by a 3D Cartesian coordinate system, with the listener at the origin. The x-, y-, and z-axes represent the horizontal, vertical, and depth directions, respectively. The playback space coordinates $x_l$, $y_l$, and $z_l$ are normalized to the range $[-1,1]$. In this study, the playback environment assumes that the monitor is positioned in front of the listener at $z=1$. The coordinates $(x_l^{\prime },y_l^{\prime },z_l^{\prime })$ estimated in Step 1 are converted to the normalized playback space coordinates $(x_l,y_l,z_l)$ using the following equations:

$$\begin{array}{cc}\hfill x_l& ={\displaystyle \frac{W_{\mathrm{m}}}{W_{\mathrm{s}}}}·{\displaystyle \frac{2x_l^{\prime }-(N_{\mathrm{x}}-1)}{N_{\mathrm{x}}-1}},\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill y_l& ={\displaystyle \frac{H_{\mathrm{m}}}{H_{\mathrm{s}}}}·{\displaystyle \frac{2y_l^{\prime }-(N_{\mathrm{y}}-1)}{N_{\mathrm{y}}-1}},\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill z_l& =d+(1-d){\displaystyle \frac{z_l^{\prime }-\underset{k}{min}\left(z_k^{\prime }\right)}{\underset{k}{max}\left(z_k^{\prime }\right)-\underset{k}{min}\left(z_k^{\prime }\right)}},\hfill \end{array}$$

(5)

where $l,k=(1,2,\dots ,L)$ represents the visual image frame index, and L is the total number of visual image frames. $N_{\mathrm{x}}\left[\mathrm{pixel}\right]$ and $N_{\mathrm{y}}\left[\mathrm{pixel}\right]$ are the number of visual image pixels in the horizontal and vertical directions. $W_{\mathrm{m}}\left[\mathrm{m}\right]$ and $H_{\mathrm{m}}\left[\mathrm{m}\right]$ represent the width and height of the monitor. $W_{\mathrm{s}}\left[\mathrm{m}\right]$ and $H_{\mathrm{s}}\left[\mathrm{m}\right]$ represent the width and height of the playback space. This conversion maps the sound image to the same position as the corresponding object on the monitor in the $xy$-plane. Along the $xz$-plane, the system positions the sound image between the monitor and the listener. Furthermore, $d\in [0,1]$ in Equation (5) is a coefficient that constrains the range of sound image movement in the depth direction. The optimal value of d will be determined in a subsequent subjective evaluation experiment.

4.3. Step 3: Design of Acoustic Metadata from the Converted Coordinates

In Step 3, acoustic metadata is designed from the converted coordinates. The time resolution of the acoustic metadata is determined by the visual image frame rate and the sampling frequency of the sound signal. The time resolution $T\left[\mathrm{sample}/\mathrm{frame}\right]$ is given by the following equation:

$$T={\displaystyle \frac{F_{\mathrm{s}}}{F_{\mathrm{v}}}},$$

(6)

where $F_{\mathrm{v}}\left[\mathrm{frame}/\mathrm{s}\right]$ is the frame rate of the visual image and $F_{\mathrm{s}}\left[\mathrm{sample}/\mathrm{s}\right]$ is the sampling frequency of the sound signal. The time information $T_l\left[\mathrm{sample}\right]$ is calculated using the following equation:

$$T_l=(l-1)T.$$

(7)

In this method, we use the .atmos format, which is part of the Dolby Atmos Master File (DAMF) specification, as the acoustic metadata [35,36]. Figure 5 shows an example of description of coordinate and temporal information in the DAMF format. In Figure 5, sampleRate corresponds to the sampling frequency of the sound signal $F_{\mathrm{s}}$, samplePos corresponds to the time information for the l-th frame $T_l$ and pos corresponds to the coordinate information for the l-th frame $(x_l,y_l,z_l)$. Here, the estimated coordinates are defined such that the x, y, and z axes correspond to the horizontal, depth, and vertical directions, respectively. In contrast, in the .atmos acoustic metadata format, the x-axis is used for the horizontal direction, the y-axis for the depth direction, and the z-axis for the vertical direction. Therefore, the coordinates are described in the format pos: $(x_l,z_l,y_l)$.

5. Objective Evaluation Experiment

We conducted an objective evaluation to confirm the accuracy of object position estimation in the horizontal and vertical directions, as estimation errors directly affect the precision of acoustic metadata design. This evaluation is important because positional inaccuracies can cause inconsistencies between the visual images and the corresponding sound images. The object positions estimated by Dist-YOLO in Step 1 of the proposed method contain estimation errors. Therefore, this experiment was designed to evaluate the impact of these errors on the accuracy of sound image localization.

5.1. Experimental Conditions for Objective Evaluation

We created 3D CG animations using Unity to provide controlled visual images [41]. Each visual image featured a single object (a sphere) moving within the field of view. Seven different object movement patterns were used. These patterns were created by integrating three primary movement directions: horizontal, vertical, and depth, as illustrated in Figure 6.

Table 2. Object position for each movement pattern within Unity.

Object Movement Patterns	Start	End
Horizontal	$(-7.5,0.0,10.0)$	$(7.5,0.0,10.0)$
Vertical	$(0.0,-4.0,10.0)$	$(0.0,4.0,10.0)$
Depth	$(0.0,0.0,10.0)$	$(0.0,0.0,-5.0)$
Horizontal + Vertical	$(-7.5,-4.0,10.0)$	$(7.5,4.0,10.0)$
Horizontal + Depth	$(-7.5,0.0,10.0)$	$(1.5,0.0,-5.0)$
Vertical + Depth	$(0.0,-4.0,10.0)$	$(0.0,0.6,-5.0)$
Horizontal + Vertical + Depth	$(-7.5,-4.0,10.0)$	$(1.5,0.6,-5.0)$

lists the object movement patterns. Additionally, in Unity, the starting coordinates of the object’s movement were denoted as Start, and the ending coordinates were denoted as End, respectively. The camera coordinates were $(0.0,0.0,-10.0)$ utilizing a perspective projection, and the field of view was 27 degrees. Object in the visual images moved at an average speed of 643 pixels/s. The visual image resolution was $1920\times 1080$ pixels, and the frame rate was 30 fps. As shown in Figure 7, the estimation errors were evaluated as the angular differences, $\theta$ and $\phi$, between the ground truth and estimated object positions from the listening point’s perspective. The ground truth object positions were obtained from Unity, whereas the estimated positions were the output of Dist-YOLO.

5.2. Experimental Results for Objective Evaluation

Figure 8 shows estimation errors along the horizontal and vertical directions for each movement pattern. Objects moving horizontally show larger errors in the horizontal direction, whereas objects moving vertically show larger errors in the vertical direction. Conditions involving depth directional movement also yield larger errors than the other conditions, likely because scale changes induced by depth directional movement increase Dist-YOLO’s estimation error. For evaluation purposes, we considered the ventriloquism effect [26]. The ventriloquism effect refers to the perceptual phenomenon in which listeners tolerate spatial misalignment between visual and auditory stimuli within certain angular thresholds [28]. When visual and auditory stimuli are presented simultaneously, spatial misalignments up to approximately 11 degrees horizontally and 19 degrees vertically are perceived as consistent [27]. This effect is important for the proposed method, as it defines the acceptable range of positional estimation errors.

As indicated in Figure 8, the estimation errors measured in this evaluation fell within the acceptable range of 11 degrees horizontally and 19 degrees vertically for all movement patterns.

6. Subjective Evaluation Experiment

We confirmed the effectiveness of the proposed method through a subjective evaluation experiment of sound image localization accuracy. In this experiment, we simultaneously presented participants with visual images and sound images constructed using the proposed method, and evaluated the consistency between the visual images and sound images. Specifically, this experiment focused on sound image localization accuracy in the depth direction. We identified the condition that achieved the highest consistency with the visual images by varying the range of sound image movement for sound images constructed from the same visual images.

6.1. Experimental Conditions for Subjective Evaluation

In this experiment, sound images were constructed using Dolby Atmos, an object-based audio [8]. Figure 9 and Figure 10 show the loudspeaker and monitor arrangements used in the experiment [42].

Table 3. Experimental conditions of visual images, sound signals, and participants.

Sampling frequency of sound sources $\left(F_{\mathrm{s}}\right)$	48,000 Hz
Quantization of sound sources	24 bits
File type of visual images	mov
Codec of visual images	H.264/AVC
Bitrate of visual images	1.14 Mb/s
Bit depth of visual images	8 bits
Ambient noise level	$L_{\mathrm{A}}$ = 35.1 dB
Sound pressure level at the listening point	$L_{\mathrm{A}}$ = 75.2 dB
Frame rate of visual images $\left(F_{\mathrm{v}}\right)$	30 fps
Resolution of visual images $(N_{\mathrm{x}},N_{\mathrm{y}})$	1920 × 1080 pixels
Monitor width and height $(W_{\mathrm{m}},H_{\mathrm{m}})$	1.32 m, 0.71 m
Resolution of monitor	3840 × 2160 pixels
Playback space width and height $(W_{\mathrm{s}},H_{\mathrm{s}})$	2.82 m, 2.62 m
Number of participants	10 (1 woman, 9 men)

shows the conditions for the visual images and audio signals, and Figure 11 shows the spatial relationship between the participants and the monitor.

Table 4. Equipment used in the experiment.

Device	Type
Loudspeaker	YAMAHA (Hamamatsu, Japan), VXS5
Power amplifier	YAMAHA (Hamamatsu, Japan), XMV8280
D/A converter	RME (Haimhausen, Germany), M-32 DA
Audio interface	Avid (Burlington, VT, USA), Pro Tools|MTRX
Monitor	Panasonic (Kadoma, Japan), TH-60DX850

shows the equipment used in the experiment. Figure 12 shows examples of start and end visual image frames used in this experiment. These visual images feature a car or a helicopter moving in a straight line within the field of view. These were created using Blender [43]. Sound images were constructed using the proposed method, with sound effects (car driving and helicopter flying sounds) from the BBC Sound Effects corresponding to each visual image [44]. Five evaluation conditions with different ranges of sound image movement in the depth direction were implemented. These were set by varying the value of the coefficient d in Equation (5). The five evaluation conditions, Cond. AA to Cond. AE, correspond to different ranges of sound image movement, as shown in Figure 11. The baseline condition, Cond. AA, is defined by the absence of sound image movement in the depth direction, with movement restricted to the $xy$-plane of the monitor. This configuration conforms to the conventional method proposed by Kato et al. (2024) [21]. In this study, it serves as a reference for evaluating the effect of the presence and range of movement along the depth direction on sound image localization accuracy.

Ten participants (one woman, nine men) took part in the experiment [45,46]. All had normal hearing, confirmed during their annual medical checkups and by audiometric screening, and no hearing-related disorders that could affect the results were observed. Participants rated the consistency between the visual images and sound images for each condition on a four-point scale: “Strongly consistent”, “Slightly consistent”, “Slightly inconsistent”, and “Strongly inconsistent”. In this experiment, consistency was defined as the correspondence between the movement of a visible object (e.g., a car or helicopter) and the movement of the perceived auditory image. This definition explicitly excluded judgments based on any incongruity between the visual object and the timbre of the presented sound. Before the experiment, all participants attended a briefing in which the evaluation criteria were explained; they were instructed that the primary focus of their judgments was the consistency of the movement. The participants did not undergo any prior training for this experiment.

The experimental procedure was conducted as follows: first, five conditions (Cond. AA to Cond. AE) were presented in a randomized order once for a visual image of a car, followed by the same procedure for a visual image of a helicopter. A 30-s interval was provided after each condition, during which participants recorded their responses. Responses were collected using Google Forms, and the sessions were conducted individually for each participant. The total duration of the experiment was approximately 10 min per person.

6.2. Experimental Results for Subjective Evaluation

Figure 13 shows the Mean Opinion Score (MOS) for the consistency between the perceived sound image and the visual image. The vertical axis shows MOS on a four-point scale (4 = Strongly consistent, 3 = Slightly consistent, 2 = Slightly inconsistent, 1 = Strongly inconsistent). The horizontal axis lists the experimental conditions, Cond. AA through Cond. AE. Among the conditions, Cond. AD yielded the highest mean (3.7), followed by Cond. AC (3.5), Cond. AE (2.8), Cond. AB (2.3), and the baseline condition, Cond. AA (1.9).

To test for differences across conditions, we applied the nonparametric Friedman test. The test indicated a significant effect of condition, ${\chi }^2\left(4\right)=31.27$, $p<0.001$. We then conducted Wilcoxon signed-rank tests comparing the top-rated condition, Cond. AD, with each of the other conditions; p-values were Bonferroni correction for four comparisons ($m=4$). The results are summarized in

Table 5. Comparison results of sound image consistency between the Cond. AD and other conditions.

Comparison Target	p-Value (Unajusted)	p-Value (Ajusted)	Significance
Cond. AA	0.0010	0.0039	Significant (**)
Cond. AB	0.0042	0.0167	Significant (*)
Cond. AC	0.3458	1.0000	Not significant
Cond. AE	0.0003	0.0012	Significant (**)

* : $p<0.05$, ** : $p<0.01$.

. Post hoc analyses showed that Cond. AD yielded significantly higher consistency than the conventional baseline (Cond. AA) as well as Cond. AB and Cond. AE. The marked improvement over Cond. AA suggests that introducing depth-wise motion of the sound image is a key factor in enhancing consistency with the visual image. Conversely, the comparisons with Cond. AB and Cond. AE indicate that consistency tends to decrease when the amount of depth movement is too restricted or overly exaggerated.

Using Cond. AC, which did not differ significantly from Cond. AD, as the reference, we conducted the same post hoc Wilcoxon signed-rank tests (Bonferroni correction; $m=4$). The results, summarized in

Table 6. Comparison results of sound image consistency between the Cond. AC and other conditions.

Comparison Target	p-Value (Unajusted)	p-Value (Ajusted)	Significance
Cond. AA	0.0020	0.0079	Significant (**)
Cond. AB	0.0078	0.0312	Significant (*)
Cond. AD	0.5000	1.0000	Not significant
Cond. AE	0.0078	0.0312	Significant (*)

* : $p<0.05$, ** : $p<0.01$.

, show that Cond. AC yielded significantly higher consistency than Cond. AA, Cond. AB, and Cond. AE. Taken together with the analyses for Cond. AD, these findings indicate that the sound image achieves the highest level of consistency with the visual image under either Cond. AC or Cond. AD. Furthermore, the visual images of both cars and helicopters yielded comparable results, with no significant differences observed between the visual images. This is thought to have enhanced the ventriloquism effect and improved sound image localization accuracy by limiting the movement of the sound image to the vicinity of the monitor and suppressing the spatial discrepancy with visual information. In particular, setting the parameter d in Equation (5) to values between $0.3$ and $0.5$ produces a sound image that closely aligns the perceived auditory movement with the visual movement.

7. Conclusions

In this paper, we proposed a method for designing acoustic metadata for object-based audio. The sound image positions were determined by estimating the coordinates of objects in the visual images. Then, acoustic metadata was designed based on the estimated coordinates and the frame rate of the visual images, and the sampling frequency of the sound signal. The objective evaluation confirmed that the object position estimation errors fell within the acceptable range of the ventriloquism effect. The subjective evaluation confirmed the range of sound image movement in the depth direction necessary for constructing sound images that are highly consistent with the visual images.

In the proposed method, control of the sound image was driven by coordinate information in the acoustic metadata, so distance was effectively cued only by sound pressure level. However, human judgments of auditory distance rely on multiple cues, including spectral variations and the Direct-to-Reverberant Ratio [47,48,49,50]. We anticipate that incorporating these additional cues will further improve sound image localization accuracy. In future work, we plan to predict and compensate for object coordinates when the object moves out of the field of view in the visual images based on object’s past movement information.