Optimal View Estimation Algorithm and Evaluation with Deviation Angle Analysis

Abstract

Image-based viewpoint estimation is one of the tasks in image analysis, and another is the inverse problem of selecting the best viewpoint for displaying a three-dimensional object. Currently, two issues need further exploration in image-based viewpoint estimation research: insufficient labeled data and a limited number of evaluation methods for estimation results. To address the first issue, this paper proposes a spherical viewpoint sampling method based on a combination of analytical methods and motion adjustment, and designs a viewpoint-based projection image acquisition algorithm. Considering the difference between viewpoint inference and image classification, we propose an accuracy evaluation method with deviation angle tolerance for viewpoint estimation. Based on constructing a new dataset with viewpoint labels, the new accuracy evaluation method has been validated through experiments. The experimental results show that its estimation accuracy can reach 89% according to the new estimation evaluation indicators. Additionally, we applied our method to estimate the viewpoints of images from a furniture website and analyzed the viewpoint preferences in its furniture displays.

1. Introduction

Viewpoint estimation and analysis have been popular and important subjects in computer vision, three-dimensional (3D) scene understanding [1,2], object detection and image analysis [3], and so on. According to different application data and research purposes, viewpoint estimation problems can be roughly divided into two categories: 3D model-based viewpoint estimation and image-based viewpoint estimation.

The 3D model-based viewpoint estimation method aims to find the best view of a 3D model which serves as the input data. The best view can be defined based on requirements from geometric shape features to aesthetic needs. In 2012, Kucerova et al. [4] presented three best view selection approaches: (1) based on geometry and using the maximum amount of visible vertices, (2) employing the visual attention model, and (3) calculating the amount of information in a scene after projecting the 3D model to the 2D image using entropy. In 2004, Sweet et al. [5] found that textures constructed from planes more nearly orthogonal to the line of sight tend to be better at revealing surface shape, and that viewing surfaces from an oblique view is significantly more effective for revealing surface shape than viewing them from directly above. For bas-relief generation and synthesis, Li et al. [6] divided this problem into two parts: selecting the best view and arranging the relief layout. In 2020, Zhou et al. [7] projected a 3D shape onto its best-view images to develop a 3D shape retrieval system.

From another perspective, the problem of selecting the best view is mainly considered a task of capturing the most information about the 3D shape, which is highly intuitive and closely related to human perception. In 2011, Liu et al. [8] proposed an approach to automatically select the best view for a 3D shape by internet image voting (IIV), assuming that in a network of similar object images, the viewpoint with the highest number of images from the same viewpoint reflects the aesthetic preferences of the public. In addition, in 2021, Collander et al. [9] introduced a reinforcement learning approach utilizing a novel topology-based information gain metric to determine the next best view of a noisy 3D sensor.

The image-based viewpoint estimation method aims to find the viewpoint at which the camera captures the input image. In 2016, Ghezelghieh et al. [10] estimated a 3D human pose from a single RGB image. For the problem of selecting the optimal viewpoint for taking good photographs for architecture, He et al. [11] addressed this by learning from professional photographs of world-famous landmarks available on the internet. In addition, viewpoint estimation can be used to construct a depth map of an image [12]. In 2020, Sun et al. [13] used a specially designed GAN model to map a given natural image to a geometry image, from which the corresponding 3D mesh can be reconstructed with accurate viewpoint information. Given that viewpoint estimation from 2D-rendered images helps understand how users select viewpoints for 3D model (point cloud, mesh, or voxel) visualization and guides users to select better viewpoints based on previous visualizations [14], this paper focuses on this topic, i.e., image-based viewpoint estimation.

To evaluate the accuracy of the image-based viewpoint estimation method, it is generally necessary to have images with known viewpoints. This type of data is paired data of images and corresponding viewpoint labels, which can be generated using the 3D model-based method by projecting to a plane specified by the viewpoint. As is well known, training deep neural networks to estimate the viewpoint of objects requires large labeled training datasets [15]. Therefore, building such a dataset is a valuable task. Traditionally, based on such a dataset, the accuracy of viewpoint estimation can be calculated by the deviation and cosine distance to measure the error. For machine learning methods, classification accuracy can be defined by considering the viewpoint position as a category, and using classification accuracy to define the accuracy of viewpoint estimation [10]. Viewpoint estimation is not entirely equivalent to classification [16]; for example, in Figure 1, Figure 1a,b belongs to the same class in the view of image classification. However, if inferred based on the viewpoint of the image, the viewpoints of Figure 1a,c are the same and differ greatly from the viewpoint of Figure 1b.

In addition, in engineering applications, such as taking photos or installing surveillance cameras, even if there is a slight angular deviation between the actual viewpoint and the optimal viewpoint it is often acceptable, because adjacent viewpoints still contain some accuracy information when characterized by cosine distance. Another example is three-dimensional reconstruction, where 15-degree angle perturbations to the estimated viewpoint of the input image are acceptable [17].

Considering the differences between image-based viewpoint estimation and image classification, it is necessary to establish a publicly available image dataset with viewpoint direction annotation for viewpoint estimation problems and new accuracy evaluation metrics. In this paper, we contribute to addressing these two issues. The highlights of this paper are as follows:

(1)

We propose an image-based viewpoint estimation method that demonstrates the complete process from creating a dataset to evaluating the results.

(2)

In the viewpoint position selection module, we integrate the initial viewpoint moving strategy into the spherical viewpoint uniform sampling based on analytical methods to enhance the effectiveness of spherical uniform random sampling.

(3)

A new dataset has been established and made available on the GitHub website. Based on the new dataset learning, we implemented image-based viewpoint estimation using a simple CNN. The viewpoint estimation results are evaluated based on the new accuracy calculation formula.

(4)

We use the viewpoint estimation method to analyze the chair display angles on a furniture website, illustrating the display angle preferences of merchants displaying items.

The remainder of the paper is organized as follows. Section 2 reviews related work. Section 3 describes the methods. Section 4 introduces the dataset and the implementation details. Section 5 concludes the paper by summarizing our study and future works.

2. Related Work

The existing literature on image-based viewpoint estimation can roughly be classified into three aspects: rule-based methods, machine learning-based methods, and evaluation.

2.1. Rule-Based Methods

The main idea of the rule-based methods for estimating viewpoints from object images is to analyze image features, such as statistical features and geometric features, and establish a mapping between these statistical features and the corresponding viewpoints. For example, in 1997, Takamatsu et al. [18] proposed a viewpoint estimation method based on moment features representing non-local image properties, including gravity center and breadth. This method demonstrated a lower computational cost and notable robustness to noise. It was found to be a connection between the 3D shape contour and the viewpoint [19]. To improve the viewpoint estimation accuracy, in 2018, Busto Pau et al. [2] proposed a domain adaptation approach that aligns the domain of the synthesized images with the domain of the actual images. In 2000, Sanchiz et al. [20] presented a method for estimating the viewpoint by identifying the minimum distance point to the set of step rays. In 2024, Sandro et al. used a Gaussian viewpoint estimator and an extended Kalman filter to predict the three-dimensional (3D) object’s position [21].

In summary, some typical features are listed in

Table 1. The examples of statistical features and geometric features used in references.

Reference	Statistical Feature	Geometric Feature
[2]	HOG, CNN feature vector	--
[18]	Moments	Gravity center and breadth
[19]	Histogram of features	Contour
[20]	Step ray; unbiased distance estimator	Central-projection analysis
[21]	A statistical optimization	Rotation matrix

HOG is the Histogram of Oriented Gradient.

as examples of these rule-based methods. Rule-based viewpoint estimation methods often perform feature analysis based on image content, so these methods are generally class-dependent. We did not adopt this viewpoint estimation method based on image feature analysis to make the construction method more versatile.

2.2. Machine Learning-Based Methods

In recent years, due to the widespread availability of online images and the establishment of labeled databases, machine learning-based methods have been widely explored. For example, in 2020, Mariotti et al. [22] proposed a semi-supervised viewpoint estimation method that can learn to infer viewpoint information from unlabeled image pairs, where two images differ by viewpoint change. Divon et al. [23] presented five key insights that should be taken into consideration when designing a CNN that solves the problem. Based on these insights, the paper proposes a network in which (i) the architecture jointly solves detection, classification, and viewpoint estimation; (ii) new types of data are added and trained on; And (iii) a novel loss function, which takes into account both the geometry of the problem and the new types of data, is proposed. In 2019, Shi et al. [14] proposed a viewpoint estimation method based on Convolutional Neural Networks (CNNs) for volume visualization. In 2021, Dutta et al. [24] proposed a self-supervised viewpoint estimation method that infers viewpoints from live camera images. This approach relies on a 3D model of the object and does not require manual data annotation or real images as input. In 2020, Joung et al. [25] developed a cylindrical convolutional network to estimate viewpoint.

A commendable viewpoint is that 3D models have the potential to generate a large number of images of high variation, which can be well exploited by deep CNNs with a high learning capacity [26]. Some work builds an image database with viewpoint labels before performing image-based viewpoint estimation on a given object. In 2015, Su H et al. [26] proposed a scalable and overfit-resistant image synthesis pipeline, together with a novel CNN specifically tailored for the viewpoint estimation task. In 2016, Wang Y et al. [27] built a CNN model to estimate object viewpoint from a single RGB image after generating a synthetic RGB image dataset with viewpoint annotations and extracting powerful features for the estimation. Several representative CNN models are listed in

Table 2. The examples of CNN models used in references.

Reference	Method	Gen. Data	Model
[14]	19-layer VGGNet, based on AlexNet	Y	Fish, head, tree, etc.
[22]	Semi-supervised, simple CNN	Y	Airplane, car, and chair
[24]	Self-supervised, Rectification Network	Y	Duck, cat, iron, etc.
[26]	A CNN with shared lower layers	Y	Bus, car, chair, etc.
[27]	R-CNN model	Y	Airplane, boat, and car

Gen. Data means that new image data are generated.

as examples of these machine learning-based methods. Table 2 shows that these methods have established datasets, but unfortunately, these data have not been made public.

We have established a new dataset and made it public in this paper. In addition, unlike these references, we also provide a viewpoint sampling method to facilitate readers in establishing new datasets.

2.3. Evaluation

The quality evaluation of estimated viewpoints depends on labeled images and is primarily conducted through visual observation or quantitative indicator measurement. Regarding the accuracy of quantitative evaluation estimation, the method based on minimizing the empirical loss proposed in [22] is representative. Minimizing empirical loss means minimizing the total error between the estimated viewpoint direction and the actual viewpoint direction. In 2018, Meng M et al. [28] proposed a novel method of viewpoint quality evaluation (VQE), which utilizes texture distortion as an evaluation metric. The texture stretch and object fragment are taken as the main distortion factors. In 2010, Helin Dutagaci et al. [29] established a benchmark for the evaluation of best view selection algorithms. The benchmark consists of the preferred views of 68 3D models provided by 26 human subjects. In 2019, Zhang F et al. [30] presented a novel viewpoint evaluation method combining geometric and visual perception information. In 2022, Stewart E et al. [31] predicted the viewpoint similarity for rotated novel objects based on the projected position shifts of surface points as the object rotates.

Considering the outstanding performance of deep learning methods in image classification problems and their early exploration in viewpoint estimation problems, we adopt a new evaluation method combining a CNN and viewpoint estimation to achieve image-based viewpoint estimation.

3. Methods

Although our goal is to develop an image-based viewpoint estimation method, this issue also involves obtaining an image dataset with accurate viewpoint information and establishing a reasonable evaluation metric. Therefore, the proposed method consists of four modules: obtaining viewpoints, generating an image dataset with viewpoint annotations, estimating a viewpoint using a CNN, and evaluating the accuracy of the estimated viewpoint. The flow chart of the proposed method is shown in Figure 2.

In our flowchart, the first two modules, the obtaining viewpoints and the generating the image dataset with viewpoint annotations modules, are the same steps as the existing CNN-based methods listed in Table 2. The difference is that our method analyzes the selection of viewpoints, but these listed methods overlooked this point. The third module is the estimating viewpoint using a CNN without a rule-based module, given that the effectiveness of rule-based methods is trained in a class-dependent manner [26], with a comparison in Table 2. In the last module, viewpoint evaluation, an accuracy evaluation method with deviation angle tolerance is proposed.

3.1. Uniform Sampling on a Unit Sphere

Similar to [16,32], a viewpoint for observing an object in three-dimensional (3D) space represents a viewpoint direction, which can be labeled as a point on the sphere surrounding the observed object. Therefore, generating random viewpoints on the sphere corresponds to the statistical problem of sampling on the sphere. The sphere is a curved space and the uniform sampling in this space is not an easy task. Traditionally, the implementation of uniform sampling on a sphere can be based on analytical formulas [33] or particle motion simulation. To improve the effect of sampling, we propose a new algorithm called the Fibonacci spherical uniform sampling algorithm based on repulsion and random optimization (FSUS-RR), combining the particle motion simulation with analytical formulas. The new sampling algorithm consists of two stages: generating sampling points through analytical methods, and moving sample points as particles. Through random experiments and verification, we chose the analytical method, the canonical Fibonacci lattice sampling method (CFLS) [33], to generate uniform sampling points on the unit sphere. Then, the initial sampling points are optimized by particle moving using two strategies: repulsion moving and random moving.

Letting $d_i^{nei}$ be the distance from viewpoint $p_i$ to its closest neighbor, the moving step $m_{step}$ is directly proportional to the difference ($d_{max}^{nei}-d_{min}^{nei}$), that is

$$m_{step}=\tau (d_{max}^{nei}-d_{min}^{nei})/2,$$

(1)

where

$$d_{max}^{nei}=\underset{j}{\max }\left\{d_j^{nei}\right\},d_{min}^{nei}=\underset{j}{\min }\left\{d_j^{nei}\right\}.$$

(2)

Letting $v_j^{nei}$ be the unit vector from the closest neighbor of viewpoint $p_j$ to $p_j$, the moving direction $v_{step}$ is staggered, that is, the direction of repulsive force $v_i^{nei}$ and a random direction are alternately selected. The detail of our FSUS-RR is shown in Algorithm 1. This algorithm combines the sampling of analytical methods with a random movement strategy, achieving a balance between sampling efficiency and the uniform distribution effect; reasoning from this, the analytical method sampling has high efficiency in generating viewpoints, but the uniformity of sampling points is insufficient. The random movement method can improve the uniform distribution effect, but the sampling efficiency is low.

Algorithm 1: FSUS-RR

Input: N, //The number of sampling points,    $I_{ite}$ //The number of iterations Output: $\mathrm{V}=\left\{{\mathrm{p}}_i\right(x_i,y_i,z_i\left)\right\}$ //the coordinates of N points on the unit sphere 1: $\mathsf{\eta }=(1+\sqrt{5})/2$  //for steps 1 through 6, use the CFLS method [33] 2: for $i \mathrm{f}\mathrm{r}\mathrm{o}\mathrm{m} 0 \mathrm{t}\mathrm{o} N-1$ 3:    $\mathsf{\vartheta }=2\mathsf{\pi }\mathrm{i}/\mathsf{\eta }$ 4:    $\mathsf{\phi }={\cos }^{-1}(1-2(i+0.5)/N)$ 5:    ${\mathrm{p}}_i=(x_i,y_i,z_i)=(\cos \theta \sin \phi ,\sin \theta \sin \phi ,\cos \phi )$ 6: end 7: flag = 1; 8: for $i \mathrm{f}\mathrm{r}\mathrm{o}\mathrm{m} 1 \mathrm{t}\mathrm{o} I_{ite}$ 9:    Build $kd-\mathrm{t}\mathrm{r}\mathrm{e}\mathrm{e}$ using all sampling points $\left\{{\mathrm{p}}_i\right(x_i,y_i,z_i\left)\right\}$ 10:   Calculating $m_{step}$ using Equation (1) 11:  for j from 1 to N 12:    if flag == 1, then 13:     $v_{step}=v_j^{nei}$ 14:    else 15:     $v_{step}$ is a random unit vector 16:    end 17:    ${\mathrm{p}}_j={\mathrm{p}}_j+m_{step}\times v_{step}$ //Update the position of ${\mathrm{p}}_j$ 18:    flag = 1-flag 19:    ${\mathrm{p}}_j={\mathrm{p}}_j/‖{\mathrm{p}}_j‖$ //In terms of vector units 20:   end 21: end 22: Output the sampling points $\mathrm{V}=\left\{{\mathrm{p}}_i\right(x_i,y_i,z_i\left)\right\}$

To verify the effectiveness of our method, we conducted comparative experiments. Several representative methods include the CFLS [33], Continuous Stochastic Dynamics (CSDyn) [34], Inverse Transform sampling (https://github.com/benroberts999/inverseTransformSample, access date: 10 March 2025) (InveT), analytically exact spiral scheme (Spiral) [35], sampling of concentric spherical shells in k-space with the new strategy (Kspace) [36], the traditional Spherical Coordinate Sampling using Azimuthal rotation and the Polar rotation (SCS-AP), and ours. The spherical sampling results of these methods are shown in the first row of Figure 3.

The differences between these results are easily noticeable when the number of points is small. When the number of points is relatively large, the difference is not obvious. We design a method to visualize the uniform distribution effect here. For any sampling point $p_m$, the average distance of $p_m$ to its k nearest neighbors $q_j(j=1,2,\dots ,k)$ can be calculated (Figure 4a) as

$${\overline{d}}_m={\displaystyle \frac{1}{k}}{\sum }_{j=1}^k{‖p_m-q_j‖}_2.$$

(3)

where ${‖p_m-q_j‖}_2$ is the 2-norm distance between point $p_m$ and its neighbor point $q_j$.

Then, we let $p_m$ move to a new position $p_i^{’}$ using the bellow formula.

$$p_m^{’}=p_m+{\overline{d}}_m\times {\overrightarrow{O{p}_m}}^o.$$

(4)

where ${\overrightarrow{O{p}_m}}^o$ is the unit vector parallel to $\overrightarrow{O{p}_m}$. In addition, to enhance the visualization effect, we also design a coloring scheme (Figure 4b) for the moved sampling points. The sampling results are visualized in Figure 5. If the uniform sampling of the sphere is uniform, then the colored sphere is a perfectly spherical shape and the coloring is also evenly distributed. The best result is that the colors are the same within the sphere. Based on these two visual indicators, our method demonstrates the best performance among these seven methods.

In addition to comparing visual effects, we also quantitatively analyze and compare the spherical sampling results of various methods. Quantitative analysis is mainly conducted on the average distance between neighbors, calculated by Formula (3). An assumption is that, in an ideal uniform sampling, the average distance ${\overline{d}}_m(m=1,2,\dots ,N)$ between the neighbors of each point should be the same. Therefore, we calculate the minimum (minAveD), maximum (maxAveD), and standard deviation (stdAveD) of the average distance ${\{\overline{d}}_m,m=1,2,\dots ,N\}$ between the nearest neighbors of all points, and perform a Kolmogorov–Smirnov (KS) test for uniform distribution and normal distribution. N is the number of sampling points, and we test three sampling densities, namely N = 25, 250, and 900, corresponding to the cases of a few, a medium number, and many points, respectively. The quantitative index calculation results of the sampling points obtained by seven methods are shown in

Table 3. Calculation results of the sampling points (N = 25). Bold numbers indicate the best results; The underlined number is the second-best result.

Methods	minAveD	maxAveD	Range	stdAveD	UniHks	UniP	NrmHks	NrmP	Time
CFLS	1.0441	1.1311	0.0869	0.0261	0	0.7773	0	0.8557	0.0010
CSDyn	0.8286	1.2514	0.4228	0.1235	0	0.0683	0	0.3685	0.0159
InveT	0.8720	1.2764	0.4043	0.1019	0	0.2367	0	0.8364	0.0008
Spiral	0.8517	1.2752	0.4236	0.1319	0	0.2589	0	0.3715	0.0014
Kspace	0.9118	1.2297	0.3178	0.0922	1	0.0334	0	0.7959	0.0152
SCS-AP	0.7784	1.3215	0.5431	0.1883	1	0.0004	0	0.0981	0.0009
Ours	1.0149	1.0439	0.0290	0.0073	0	0.4686	0	0.9375	0.0693

UniHks represents the KS test result of uniform distribution: 0 means passing the test, and 1 rejecting the hypothesis. NrmHks represents the KS test result of normal distribution: 0 means passing the test, and 1, rejecting the hypothesis. UniP and NrmP are the p-values corresponding to the KS test results of a uniform distribution and normal distribution, respectively. The number marked in bold is the best value, and the number marked with an underline is the second best value.

,

Table 4. Calculation results of the sampling points (N = 250). Bold numbers indicate the best results; The underlined number is the second-best result.

Method	minAveD	maxAveD	Range	stdAveD	UniHks	UniP	NrmHks	NrmP	Time
CFLS	0.3297	0.3585	0.0287	0.0029	1	0.0000	1	0.0000	0.0010
CSDyn	0.2480	0.4877	0.2397	0.0464	1	0.0000	0	0.2444	0.8326
InveT	0.1933	0.5313	0.3380	0.0562	1	0.0000	0	0.2507	0.0008
Spiral	0.2039	0.4757	0.2719	0.0522	1	0.0000	0	0.9607	0.0044
Kspace	0.2323	0.4702	0.2379	0.0419	1	0.0000	0	0.6081	0.0003
SCS-AP	0.0000	0.4435	0.4435	0.1392	1	0.0000	1	0.0000	0.0009
Ours	0.3207	0.3381	0.0174	0.0033	1	0.0000	0	0.9953	2.1910

and

Table 5. Calculation results of the sampling points (N = 900). Bold numbers indicate the best results; The underlined number is the second-best result.

Method	minAveD	maxAveD	Range	stdAveD	UniHks	UniP	NrmHks	NrmP	Time
CFLS	0.1738	0.1891	0.0154	0.0011	1	0.0000	1	0.0000	0.0014
CSDyn	0.1220	0.2449	0.1229	0.0210	1	0.0000	0	0.8633	10.3144
InveT	0.0799	0.2852	0.2053	0.0301	1	0.0000	0	0.2412	0.0010
Spiral	0.0984	0.2966	0.1983	0.0296	1	0.0000	1	0.0027	0.0015
Kspace	0.1024	0.2589	0.1565	0.0272	1	0.0000	0	0.8009	0.0013
SCS-AP	0.0000	0.2329	0.2329	0.0643	1	0.0000	1	0.0000	0.0011
Ours	0.1680	0.1840	0.0160	0.0027	1	0.0000	0	0.3879	3.7868

.

According to these quantitative index calculation results of the sampling points, our method generally demonstrates the best performance according to the four evaluation metrics: maxAveD, Range, stdAveD, and NrmHks. The proposed method is similar to other methods for the remaining five indicators. For example, our minimum distance is relatively large, ranking second among all methods, which is also quite good. Because we use analytical methods to calculate the initial viewpoint position and then use random movement to improve uniformity, the calculation time among these five indicators is more prolonged than using analytical methods alone, but it is more efficient than using the random movement method alone.

3.2. Image Generated According to Viewpoints

After selecting viewpoints using Algorithm 1, the dataset for image viewpoint estimation can be constructed. We utilize 3D models to perform parallel projection based on the specified viewpoint directions, thereby generating 2D projection images. This process is illustrated in Figure 6.

Due to the different sizes of the models, it is necessary to standardize them into a fixed size. Let $N_v$ be the number of vertices of the 3D object model, and $V_i\left(x_i,y_i,z_i\right)(i=1,2,\dots ,N_v)$ the ith vertex of the model. Then, this model can be standardized using the following formula:

$$Q_i^{new}={\displaystyle \frac{{\gamma }_o}{R_o}}{(V}_i-O_{cnt}),i=1,2,\dots ,N_v.$$

(5)

where ${\gamma }_o$ is a scale factor, and we use ${\gamma }_o=0.45$ as the default value; $R_o$ is the diameter of the object; and $O_{cnt}$ is the geometric center of the object. That is,

$$O_{cnt}={\displaystyle \frac{1}{N_v}}\sum _{i=1}^{N_v}{V}_i$$

(6)

After standardizing the model using Equation (5), it can be completely enclosed within the unit sphere.

With the standardized model and the viewpoints obtained according to Section 3.1, projection images with viewpoint labels can be obtained through parallel projection based on the viewpoint direction. The mathematical formulas for parallel projection based on the direction of the viewpoint $p_{view}$ ($‖p_{view}‖=1$) can be formulated as

$$Q_n=Q_o-(p_{view}·\overrightarrow{O{Q}_o})p_{view}.$$

(7)

where $Q_o$ is any vertex of the 3D object of which the center has been translated to the origin, and $Q_n$ is the projection point of $Q_o$ on the projected image. The algorithm for generating images according to specified viewpoints (Algorithm 2) is described as follows. In the process of generating this image with a viewpoint label, Algorithm 1 is the first step in Algorithm 2.

Algorithm 2: Generating images according to specified viewpoints

Input: ${\mathsf{\vartheta }}_L,{\mathsf{\vartheta }}_U$, //rang ${\mathsf{\vartheta }}_L\le \mathsf{\vartheta }{\le \mathsf{\vartheta }}_U$,    ${\mathsf{\phi }}_L,{\mathsf{\phi }}_U,$ //rang ${\mathsf{\phi }}_L\le \mathsf{\phi }\le {\mathsf{\phi }}_U$,    $d_{vp},$ //The minimum distance between neighboring viewpoints    $I_r,I_c$.//The size of the output image Output: $\mathsf{\Omega }=\{I_i,{\mathsf{\vartheta }}_i,{\mathsf{\phi }}_i\}$ //images with labeled viewpoints 1: Generate viewpoints using Algorithm 1; 2: Remove viewpoints that are not within the specified ranges; 3: Remove neighboring viewpoints with distances less than $d_{vp}$; 4: Generate a projection image for each viewpoint using Equation (7); 5: Construct the AABB (Axially Aligned Bounding Box) of the object projection image, and cut off the external redundant margins which are out of the AABB; 6: Enlarge or reduce the image to the specified size (e.g., $32\times 32$); 7: Output $\mathrm{the}\mathrm{images}\mathsf{\Omega }=\{I_i,{\mathsf{\vartheta }}_i,{\mathsf{\phi }}_i\}$ to a folder.

In the labeled image dataset, the viewpoints we have chosen are in front of the upper hemisphere, considering that the viewpoint is often higher than the horizontal line of sight in most applications, such as autonomous driving or furniture exhibitions.

3.3. Viewpoint Estimated Using CNN

Deep learning-based image classification has achieved remarkable success since 2006. Because of the difference between viewpoint estimation and image classification, we utilize a deep learning model for image classification with viewpoint labels after developing a new soft accuracy function. The accuracy function allows for the perspective estimation bias, which is defined as follows.

$$A_{cc}={\displaystyle \frac{1}{N}}\sum _{m=1}^N{\delta }_m,$$

(8)

where N is the number of images, and

$${\delta }_m=\left\{\begin{array}{cc}\overrightarrow{O{P}_m^{est}}·\overrightarrow{O{P}_m^{gt}},\hfill & \hspace{1em}if\overrightarrow{O{P}_m^{est}}·\overrightarrow{O{P}_m^{gt}}\ge {\mathrm{c}\mathrm{o}\mathrm{s}(\delta }_0)\hfill \\ \hfill 0,& \hspace{1em}otherwise\hfill \end{array}\right..$$

(9)

In Equation (9), m is the $m^{th}$ image, and ${\delta }_0$ the threshold which is the maximum tolerable viewpoint deviation angle. Since our study utilizes a unit vector to represent a viewpoint direction, the inner product $\overrightarrow{O{P}_m^{est}}·\overrightarrow{O{P}_m^{gt}}$ of the vectors $\overrightarrow{O{P}_m^{est}}$ and $\overrightarrow{O{P}_m^{gt}}$ directly corresponds to the cosine value of the angle between these two vectors.

According to the algorithm in Section 3.2, we have constructed an image dataset with viewpoint position labels. These images are classified according to the viewpoint positions. Using Equation (9), the loss function can be rewritten. After completing the dataset construction and loss function design, a simple CNN framework [37] is established to achieve image-based viewpoint estimation. This CNN framework consists of seven layers, and its architecture and parameters are illustrated in Figure 7.

3.4. Accuracy Evaluation of Estimated Viewpoint

During the model testing phase, or when comparing image-based viewpoint estimation methods, the accuracy of image-based viewpoint estimation is also evaluated using Equations (8) and (9). The value of the parameter ${\delta }_0$ can be specified by the user based on the number of viewpoints and error tolerance. For example, ${\delta }_0=\pi /6$.

4. Experiment

The experiment aims to evaluate the proposed method (Figure 2) and assess its effectiveness through three aspects: viewpoint image dataset construction, parameter sensitivity analysis, and image-based viewpoint estimation performance evaluation. Our algorithms are implemented in MATLAB (2018). Each run is executed on a single thread on a personal desktop computer, and the computer configuration is Intel (R) Core (TM) i9-9700 CPU @ 3.00 GHz processor and 16 GB RAM per run. The runtime for each parameter setting is measured as the average execution time (in seconds) of five repeated experiments.

4.1. Dataset

Considering furniture sale cases, furniture is displayed to customers through images captured from optimal viewpoints. Our experiment aims to estimate the viewpoint based on these furniture images, primarily featuring chairs, stools, and sofas.

First, we downloaded an open dataset [38]. This dataset consists of over a hundred 3D furniture models. Among them, 29 objects, including chairs, stools, and sofas, were selected for our viewpoint screenshot experiment.

Then, we applied Algorithm 1 to generate viewpoints on the unit sphere. In this experiment, we selected only 31 viewpoints in front of the object in the upper hemisphere, as illustrated in Figure 8a.

Finally, we utilized Algorithm 2 to generate images based on these specified viewpoints. The dataset used in our experiments consists of 31 viewpoint classes, each with 29 images. Therefore, the generated dataset consists of 899 images divided into 31 categories, with 29 images per category. Figure 9 presents 31 images of the same object projected from 31 different viewpoints.

4.2. Parameter Analysis and Comparisons

In Algorithm 1, the parameters are defined based on a random distance and do not require user specification. Parameter $I_{ite}$, the number of iterations, according to our experiment, has a minor impact on the distribution when the number of points is relatively small, e.g., less than 30. If $N>30$, we set $I_{ite}=2500$ times as the default value.

In Section 3.3 and Section 3.4, parameter ${\delta }_0$, the maximum tolerable viewpoint deviation angle, significantly affects the accuracy value. If ${\delta }_0=0$, an estimated viewpoint is considered correct only if it perfectly coincides with the ground truth viewpoint; otherwise, it is considered incorrect. In this case, the formula for calculating the classification accuracy follows the standard accuracy formula used in image classification. The accuracy calculation method for viewpoint estimation proposed in this study is an extension of the traditional accuracy calculation method for image classification. The smaller the value, the stricter the accuracy criterion, allowing for less error tolerance.

We experimented to demonstrate the effectiveness of using our method for viewpoint estimation. Using ${\delta }_0=0,\pi /6,\pi /3$, the nearest neighbor relationships of the 31 viewpoints are represented by a distance matrix, as shown in Figure 10.

4.3. Viewpoint Estimation Results

From the generated dataset, we randomly select $⌊75\%\times N⌋$ images for training, where $⌊x⌋$ rounds number $x$ to the lower integer. Since each class consists of 29 images, there are 21 images in each class in the training set, and the remaining eight images are used for testing. The maximum number of iterations is set to 20,000, with other parameters set to their default values. The training process takes 10,719.46 s, and the training accuracy is 100% due to the small training set. The confusion matrix of the test set is shown as a heat map in Figure 11. By summing the diagonal elements and taking the total number of images, the classification accuracy can be calculated to be 0.6855, corresponding to ${\delta }_0=0$.

If the viewpoint evaluation method with the angle deviation metric proposed in this article is adopted, with a tolerance angle of 30 degrees (${\delta }_0=\pi /6$), as shown in Figure 8b, the accuracy of the viewpoint estimation is 0.8105. The number of images with deviation angles within $\pi /6$ is marked as a red number in the confusion matrix in Figure 12.

Similarly, if we increase the deviation angle to 60 degrees (${\delta }_0=\pi /3$), as shown in Figure 8c, the accuracy of the viewpoint estimation increases to 0.8911, improved by approximately 9.94% compared to ${\delta }_0=\pi /6$. The number of images with deviation angles within $\pi /3$ is also marked as a red number in the confusion matrix in Figure 13.

4.4. Application to Display Image Analysis

One application of viewpoint estimation is the analysis of the display angle of objects. Among various object displays, the chair display is the most closely related to our training set. Therefore, we use it as an example to illustrate the application of our method. We downloaded chair images from the Nitori company website (https://www.nitori-net.jp/ec/cat/Chair/, access date: 12 February 2025). We randomly chose chair products across different categories and used the first image of the product profile page. A total of 105 chair images were analyzed using viewpoint estimation with our method. The estimated viewpoints were then visualized using a frequency distribution histogram, as shown in Figure 14. From the figure, we observe that only 13 viewpoints are adopted by the company, and 44% of the images were captured from the same viewpoint, the 28th viewpoint. This perspective corresponds to a right-side, slightly upward-angled shot of the chair. This application illustrates that we can infer the preference of the website display angles for items using our proposed viewpoint estimation method.

5. Conclusions

In this paper, we propose an image-based method for viewpoint estimation; verify that using a sampling point movement strategy to improve the uniform distribution of initial spherical viewpoints generated by analytical methods reduces the average distance of the maximum nearest neighbors; generate a small image dataset with viewpoint markers; and develop a calculation method for the accuracy of viewpoint estimation that integrates deviation angles. The viewpoint estimation experiments show that the accuracy of CNN-based estimation can be improved from 68.55% to 81.05% when the viewpoint angle deviation is $\mathsf{\pi }/6$; the proposed viewpoint estimation method can be used for analyzing the display angle of chairs on furniture websites.

The main limitations of our method are that the CNN model in the algorithm framework is simple and lacks a comprehensive selection of various viewpoint estimation models, and the newly established dataset is too small. Although comprehensive selection is not the focus of this paper, it is a worthwhile direction for future research. If a comprehensive model selection is conducted and a more suitable machine learning model is found, the accuracy of viewpoint estimation can be improved. The scale of image datasets with viewpoint annotations provided in existing papers, including this paper, is still too small, so it is worth looking forward to generating large datasets and providing them to the public for use.