Single-Shot, Monochrome, Spatial Pixel-Encoded, Structured Light System for Determining Surface Orientations

Abstract

This study introduces a technique for determining surface orientations by projecting a monochrome, spatial pixel-encoded pattern and calculating the surface normals from single-shot measurement. Our method differs from traditional methods, such as shape from shading and shape from texture, in that it does not require relating the local surface orientations of adjacent points. We propose a multi-resolution system incorporating symbols varying in sizes from 8 × 8, 10 × 10, 12 × 12, 14 × 14, and 16 × 16 pixels. Compared to previous methods, we have achieved a denser reconstruction and obtained a 5.2 mm resolution using an 8 × 8 pattern at a depth of 110 cm. Unlike previous methods, which used local point orientations of grid intersection and multiple colors, we have used the monochrome pattern and deterministic centroid positions to compute the unit vector or direction vector between the neighboring symbols. The light plane intersections are used to calculate the tangent vectors on the surface. Surface normals are determined by the cross-product of two tangent vectors on the surface. A real experiment was conducted to measure simple plane surfaces, circular surfaces, and complex sculptures. The results show that the process of calculating surface normals is fast and reliable, and we have computed 1654 surface normals in 29.4 milliseconds for complex surfaces such as sculptures.

1. Introduction

Structured light (SL) is a useful and low-cost method for 3D measurements. Various SL methods have evolved during the past few decades [1,2]. SL can provide high accuracy and dense reconstruction; however, it is sensitive to object color and texture [3,4]. The structured light-based systems work well for measuring objects in the close range. They cannot be employed for far-distance measurement. SL techniques are mainly classified into temporal and spatial encoding schemes [5]. Temporal coding schemes include binary and gray coding, phase shift, and fringe patterns [6,7]. The spatial encoding schemes include 1D and 2D grid-indexed-based methods [8]. The advantage of spatial encoding schemes is being fast and single-shot [9]. These include the De Bruijn Sequence [10], pseudo-random sequence or M-arrays [11,12,13,14,15,16], non-formal coding schemes [17,18], and speckle projection techniques [19].

The two principal descriptors for defining shapes or objects in 3D are their position and orientation profiles. The orientation profile of the shapes is generally represented by surface normals, which describe the orientation of the neighborhood points on the surface. A surface normal at a certain point is a vector perpendicular to the tangential plane of the surface at that point. It defines the orientation of the surface at that point. Orientation can be realized by correlating the neighboring position data. However, this may not be independent information and depends on a particular abstraction of position data. Different densities of the same position data may result in various orientations, necessitating an independent determination of orientation.

The orientation profile has been determined using several methods in the computer vision community, including photometric stereo [20], texture [21,22], shading [23], specular flow [24], and polarization [25]. Traditional methods, such as texture and shading, require specific object properties to determine the surface orientation. These properties include consistent albedo or uniform texture distribution on the surface. This may restrict the application of these methods to a range of objects that lack such attributes. This work employs a monochrome spatial pixel-encoded symbol-based pattern to determine the orientation profile regardless of the position on the surface. Thus, it overcomes the limitations of traditional methods, as spatial SL encoding can produce artificial and uniform textures on the object surface regardless of position. The monochrome pattern also allows us to measure colored surfaces with non-uniform albedo.

The computation of surface normals is essential for many visualization and object registration [26], 3D geometry [27], semantic segmentation [28], classification [29], recognition [30], and representation [31] problems in artificial intelligence (AI) and robotics applications. Fast and accurate estimation of surface normals is required in many automotive applications [32].

The existing literature on SL is mainly related to determining the position profiles of a surface. Fewer works calculate surface normals through SL. For instance, a grid pattern is employed in [33] by using two orthogonal stripe patterns for static and smooth planar surfaces to find the orientation and structure of a surface. The grid pattern has the advantage of both simple point and line patterns. However, sharp discontinuities may appear as abrupt changes on the surface. So, grid coding implies weak constrictions on the surface of 3D objects [34]. In another approach, the target object is illuminated with a grid pattern [35]. The surface normals at the grid points (where the intersections of the grid lines occur) are determined by analyzing the variations in the lengths of the grid edges. Significant inaccuracies happen in some regions, mainly when the surface orientation differs significantly from the reference plane or where there is non-zero surface curvature.

In [36], surface normals were determined by projecting stripe patterns by estimating the slope and intervals between the stripes for a smooth planar surface. Labeling intersecting points of the grid is also time consuming, considerably if some parts of the object are occluded. In [37], a pseudo-random array of color dots was used as an illumination pattern. It was assumed that each circular dot from the surface would appear as an ellipse in the image. The center points of such ellipse were regarded as feature points, and a method was used to calculate the surface normals by using the shear and scaling factor of the ellipse along the epipolar lines. The approach assumes that the object surface is planar, not only in the intermediate vicinity of elliptical pattern elements, but also in the far neighborhood. So, the method could not be helpful for the measurement of objects with large curvatures. Multiple same-color dots may appear on the same epipolar line, causing ambiguous decoding.

In [38], two strip patterns were projected at different angles to compute two surface tangents for the same position. In this approach, the angle of the strips depends upon the orientation of the 3D object and gradient operation, thus resulting in outliers and erroneous noisy measurements. A method for determining surface normals using the slope and widths of the stripes was introduced in [39]. This approach relies on the assumption that the surface patches between the two stripe edges are either planar or highly smooth, and the inclination angle can be estimated from the deformed width of the stripe by comparing it to the measured width on the reference plane. In this approach, it was assumed that the pattern lighting and imaging system has a parallel orientation. Therefore, the intrinsic parameters of the camera and projector are neglected. Hence, errors due to overly simplified projection and image models are unavoidable.

Among all the mentioned attempts, the most important is [40], which utilized grid point intersections of rhombic-shaped color objects spread through the robust pseudo-random sequence as grid line feature points. Due to the surface curvature or texture, the neighboring rhombic-shaped symbols forming grid intersections may be disconnected. The accurate location of the grid-intersection points of the rhombic-shaped color symbols cannot be determined due to the presence of image noise. The grid intersection points in the image are determined by calculating the differentials of raw intensities between neighboring locations through a mask. This process results in increased image noise and decreased accuracy. Due to the use of multiple colors, the projected pattern will be immune to noise and segmentation errors influenced by the surface color. With highly saturated surface colors, the projected illumination can easily warp through the intrinsic shades of the measuring surface. This method utilized local point orientation angles to compute the normals to the intersecting planes to determine the surface tangents. When the surface curvature modulates the projected pattern due to which the neighboring rhombic-shaped objects are disconnected, hence the exact location of grid point intersections and the accurate estimation of local point orientations is impeded.

In contrast, we utilized the deterministic centroid positions of adjacent symbols to compute the surface normals. We propose improvement in the method presented in [40] by implementing it through a monochromatic, multi-resolution system. The monochromatic geometric symbol-based method is an alternative to the color-encoding technique. The monochrome geometric symbol-based methods are more resilient compared to color encoding in terms of the effect of ambient light, uneven albedos, and surface colors [41]. Instead of using local point orientation angles in computation, we used the deterministic centroid position to determine the unit vector or direction vector between the neighboring symbols. These unit vectors are utilized in calculating the normals of the intersecting planes, which in turn determine the surface tangents. These unit vector calculations will simplify the process of determining the normals to the projection-side light planes as all the unit vectors on the projection-side will be parallel to the X and Y axes. Our previous works [14,42], introduced pixel-encoded patterns for encoding SL to find the position information. We presented a flexible method to design patterns according to the surface area requirements and measurement precision by using the multi-resolution system through single-shot measurement. This was achieved by employing robust pseudo-random sequences of any required size depending upon the projector’s resolution. In this work, we suggested a grid-indexed-based SL method to determine the orientation regardless of the surface position.

2. Materials and Methods

2.1. Designing of a Pattern

This section will explain how to form a projection pattern to determine the surface orientation using spatially encoded SL patterns. These Patterns use pixel-encoded monochrome symbols varying in size from 8 × 8, 10 × 10, 12 × 12, 14 × 14, and 16 × 16 pixels and spread through corresponding robust pseudo-random sequences or M-array.

2.1.1. Defining Symbols

We propose a multi-resolution system in which the symbol sizes vary from 8 × 8, 10 × 10, 12 × 12, 14 × 14, and 16 × 16 pixels in symbol space. The symbols used to form the projection pattern are depicted in Figure 1.

These symbols have already been proposed in our previous work, and their physical properties have been analyzed and recorded [14]. To implement the algorithm for surface normal computation, we selected fourteen symbols whose centroid positions are exactly at the center of the symbol spaces. The reason for choosing symbols with a centroid at the center of the symbol space is to simplify the process of computing surface normals since the neighborhood symbols in the projection pattern will align in horizontal rows or vertical columns, and it will be explained in the subsequent Section 2.2.

2.1.2. Robust Pseudo-Random Sequences or M-Arrays

To spread the symbols in a projection pattern in a controllable manner, we generated five M-arrays of different sizes. We formed the M-arrays using the method described in our previous work [14]. We used a projector with a resolution of 800 × 1280 and proposed a multi-resolution system with a pattern designed employing fourteen symbols or symbols. We generated five M-arrays corresponding to each pixel size. Each M-array consisted of four alphanumeric bases. Therefore, four symbols were used simultaneously out of fourteen symbols in a projection pattern.

The robustness of the codewords in all the M-arrays was ensured by computing the Hamming distances between each codeword. The dimensions, number of symbols, Hamming distances, percentile of the robust codeword, and number of feature points contributed by each M-array are shown in

Table 1. Properties of M-arrays for various resolutions and number of feature points contributed in pattern.

Symbol Size	Spacing Between Consecutive Symbols	No. of Symbols Used in M-Array	M-Array Dimensions (m × n)	Average Hamming Distance	Robust Codewords (%)	No. of Feature Points in the Projected Pattern
8 × 8	1	4	90 × 144	6.7517	99.8683	12,496
10 × 10	1	4	75 × 117	6.7524	99.8667	8352
12 × 12	2	4	60 × 93	6.7503	99.8713	5187
14 × 14	2	4	51 × 81	6.7489	99.8734	4000
16 × 16	2	4	45 × 72	6.7495	99.8452	3124

.

The Hamming distance profiles of each M-array formed to use in the projection pattern are represented in Figure 2. The profiles of M-arrays reflected that most of the codewords have Hamming distances greater than 3, and about 60% of codewords have Hamming-distance spaces of 7, 8, and 9, which proves that the generated M-arrays are robust.

2.1.3. Formation of a Projection Pattern

We used five M-arrays with four symbols to generate five projection patterns using the symbols or symbols proposed in [14]. The number of feature points in the pattern increased with the use of smaller symbol resolution as M-arrays for smaller symbols have bigger dimensions while keeping the projector resolution constant, which is 800 × 1280. Conversely, as the symbol sizes in the pattern increased, both the number of feature points and the M-array dimensions reduced. The identical phenomenon is illustrated in the seventh column of Table 1. The number of feature points for an 8 × 8 symbol resolution is 12,496, while for a 10 × 10 resolution, it decreases to 8352, and for a 12 × 12 resolution, it is further reduced to 5187. Similarly, the pattern resolution of 14 × 14 comprises 4000 feature points, while the symbol resolution of 16 × 16 possesses the fewest feature points of 3124. So, the number of feature points has decreased from 12,496 for the 8 × 8 pattern to almost one-fourth, specifically 3124 for the 16 × 16 pattern resolution. Consequently, the employment of smaller symbol sizes enables denser reconstruction of 3D points and results in the computation of a greater number of surface normals on the measured object. The parts of five generated projection patterns using single-centroid symbols are shown in Figure 3.

2.2. Computation of Surface Normals

To compute surface normals, we used the principle described in [40] with necessary modifications. Figure 4 explains the mechanism of calculating surface normals.

2.2.1. Defining of Points Geometry

Let us consider three points, P₁ (X₁, Y₁, Z₁), P₂ (X₂, Y₂, Z₂), and P₃ (X₃, Y₃, Z₃) on the surface of a 3D object in the world coordinate system. The corresponding points on the camera image plane are P_c1 (x_c1, y_c1, f_c), P_c2 (x_c2, y_c2, f_c), and P_c3 (x_c3, y_c3, f_c). Similarly, the corresponding points on the projector pattern plane are P_p1 (x_p1, y_p1, f_p), P_p2 (x_p2, y_p2, f_p), and P_p3 (x_p3, y_p3, f_p). The projector induced points on the surface of the 3D object while the camera recorded the induced points of the projector on the surface. P₁ is the point where we compute the surface normal or, more specifically, the centroid position of the symbol in consideration induced by the projected pattern. So, P₁ is a projection of P_p1, and it is observed in the camera image as P_c1. P₂ is the right-side neighborhood point or, more specifically, the centroid of the right-side neighborhood symbol, which lies on the surface induced through the projected pattern. So, P₂ is a projection of P_p2, and it is observed in the camera image as P_c2. P₃ is the down-side neighborhood point or, more specifically, the centroid of the down-side neighborhood symbol that lies on the surface induced by the projected pattern. So, P₃ is a projection of P_p3, and it is observed in the camera image as P_c3. The points on the projected patterns are fully accessible since they are the entities under system design. Similarly, the points in the camera image are the entities under observation. Point P₁ is selected where the correspondence between the projected pattern and camera image matches exactly.

2.2.2. Defining Device Parameters, Projection, and Image Planes

C is the camera’s optical center, and P is the projector’s optical center. The optical centers of the projector and camera are at the distance of focal lengths from the midpoints of the projector pattern plane and camera image plane. The camera focal length is denoted by f_c, and the projector focal length is denoted by f_p. X_c and Y_c are the dimensions of the camera image plane, and X_p and Y_p are the dimensions of the projector pattern plane. O_c (x_oc = X_c/2, y_oc = Y_c/2, f_c) and O_p (x_op = X_p/2, y_op = Y_p/2, f_p) are the midpoints of the camera image plane and projector pattern plane, respectively, and they are exactly half of the camera and projector X and Y dimensions.

2.2.3. Defining Light Planes and Their Normals

n_px is normal to the light plane defined by the projector optical center, P, to the world coordinates, P₁ and P₂, i.e., ∏PP₁P₂. The corresponding light plane to the camera side from the camera’s optical center, C, to the point, P₁, and P₂ is ∏CP₁P₂, and its normal is denoted by n_cx. The two common vertices of these two light planes are formed by the current symbol considered for surface orientation calculation and its right-side neighborhood symbols, which are induced on the surface by the projected pattern. The edges of these light planes are passed through the centroids of current symbols and its right-side neighborhood symbol both in the projected pattern and camera image. The two light planes, i.e., ∏PP₁P₂, which is passing from the projected pattern, and ∏CP₁P₂, which is passing from the camera image, intersect each other on the curve between the two points on the surface, i.e., P₁ and P₂. Similarly, n_py is the normal to the light plane defined by the projector’s optical center, P, to the world coordinates, P₁ and P₃, i.e., ∏PP₁P₃. The corresponding light planes to the camera side from the camera’s optical center, C, to the points, P₁ and P₃, is ∏CP₁P₃, and its normal is denoted by n_cy. The two common vertices of these two light planes are formed by the current symbol considered for surface orientation calculation and its down-side neighborhood symbols, which are induced on the surface by the projected pattern. The edges of these light planes are passed through the centroids of current symbols and its down-side neighborhood symbol both in the projected pattern and camera image. The two light planes, i.e., ∏PP₁P₃, which is passing from the projected pattern, and ∏CP₁P₃, which is passing from the camera image, intersect each other on the curve between the two points on the surface, i.e., P₁ and P₃.

2.2.4. Underlying Principle

The smaller planes on the projector and camera sides, i.e., light planes from the optical centers (C or P) of the projector or camera to the coordinate points on the projector (P_p1, P_p2) or (P_p1, P_p3) and camera (P_c1, P_c2) or (P_c1, P_c3), are part of the bigger plane, i.e., light plane from the optical centers (C or P) of the projector and camera to the 3D world coordinate points (P₁, P₂) or (P₁, P₃) on the measuring object. The smaller light plane ∏PP_p1P_p2 is part of the bigger light plane ∏PP₁P₂. Similarly, the smaller light plane ∏PP_p1P_p3 is part of the bigger light plane ∏PP_p1P_p3. On the camera side, the smaller light plane ∏CP_c1P_c2 is part of the bigger light plane ∏CP₁P₂. Similarly, the smaller light plane ∏CP_c1P_c3 is part of the bigger light plane ∏CP₁P₃. Thus, any normal on the smaller plane will be the normal on the bigger plane. From the smaller planes of the projector and camera, we can easily determine the normal to the bigger planes that intersect on the 3D object since all the parameters of the smaller light planes are known.

Computation of Light Planes Normals to the Projector Side

On the projector side, the normal to the light plane ∏PP₁P₂ can be calculated from the smaller plane ∏PP_p1P_p2. The normal to this plane is simply the vector cross-product of the position vector at point, P_p1, and the unit vector between points, P₁ and P₂. Similarly, the normal to the light plane ∏PP₁P₃ can be calculated from the smaller plane ∏PP_p1P_p3. The normal to this plane is also the vector cross-product of the position vector at point, P_p1, and the unit vector between the points, P_p1 and P_p3.

$${\mathrm{n}}_{\mathrm{p}}=\left[\begin{array}{c}{\mathrm{x}}_{\mathrm{p}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{p}}\\ {\mathrm{y}}_{\mathrm{p}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{p}}\\ f_p\end{array}\right]X\left[\begin{array}{c}{\mathrm{ĉ}}_{\mathrm{p}\mathrm{x}}\\ {\mathrm{ĉ}}_{\mathrm{p}\mathrm{y}}\\ 0\end{array}\right]=\left|\begin{array}{ccc}i& j& k\\ {\mathrm{x}}_{\mathrm{p}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{p}}& {\mathrm{y}}_{\mathrm{p}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{p}}& f_p\\ {\mathrm{ĉ}}_{\mathrm{p}\mathrm{x}}& {\mathrm{ĉ}}_{\mathrm{p}\mathrm{y}}& 0\end{array}\right|$$

(1)

where $i$, j, and k are the unit vectors along the X, Y, and Z axes. ĉ_px and ĉ_py are the components of the unit vector between the points (P_p1 to P_p2) or (P_p1 to P_p3). So, if the unit vector is calculated between the right-side neighborhood points between P_p1 and P_p2, it can be written as:

$${\mathrm{ĉ}}_{\mathrm{p}}={\displaystyle \frac{\left({\mathrm{x}}_{\mathrm{p}2}-{\mathrm{x}}_{\mathrm{p}1}\right) i+\left({\mathrm{y}}_{\mathrm{p}2}-{\mathrm{y}}_{\mathrm{p}1}\right) j}{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{p}2}-{\mathrm{x}}_{\mathrm{p}1}\right)}^2+{({\mathrm{y}}_{\mathrm{p}2}-{\mathrm{y}}_{\mathrm{p}1})}^2} }}={\mathrm{ĉ}}_{\mathrm{p}\mathrm{x}} i+{\mathrm{ĉ}}_{\mathrm{p}\mathrm{y}} j$$

(2)

Similarly, if the unit vector is calculated between the down-side neighborhood points between P_p1 and P_p3, it can be written as:

$${\mathrm{ĉ}}_{\mathrm{p}}={\displaystyle \frac{\left({\mathrm{x}}_{\mathrm{p}3}-{\mathrm{x}}_{\mathrm{p}1}\right) i+\left({\mathrm{y}}_{\mathrm{p}3}-{\mathrm{y}}_{\mathrm{p}1}\right) j}{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{p}3}-{\mathrm{x}}_{\mathrm{p}1}\right)}^2+{({\mathrm{y}}_{\mathrm{p}3}-{\mathrm{y}}_{\mathrm{p}1})}^2} }}={\mathrm{ĉ}}_{\mathrm{p}\mathrm{x}} i+{\mathrm{ĉ}}_{\mathrm{p}\mathrm{y}} j$$

(3)

The advantage of using symbols in the projection pattern with a centroid exactly at the center of the symbol space will be evident here. Since the two neighborhood symbols in the projection pattern are horizontally aligned for the right-side neighborhood and vertically aligned for the down-side neighborhood. Thus, unit vector computation simplifies the process of determining the projection-side planes as all the unit vectors on the projection side are either parallel to the X- or Y-axis. Thus, for the light plane parallel to the X-axis, (x_p2 − x_p1) = 0, and ĉ_py = 1. Similarly, for the light plane parallel to the Y-axis, (y_p3 − y_p1) = 0 and ĉ_px = 1. Therefore, normals to the light plane parallel to the X-axis and Y-axis can be easily determined as:

$${\mathrm{n}}_{\mathrm{p}\mathrm{x}}=\left|\begin{array}{ccc}i& j& k\\ {\mathrm{x}}_{\mathrm{p}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{p}}& {\mathrm{y}}_{\mathrm{p}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{p}}& f_p\\ 1& 0& 0\end{array}\right|=f_p j-\left({\mathrm{y}}_{\mathrm{p}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{p}}\right)k=\left[\begin{array}{c}0\\ f_p\\ -\left({\mathrm{y}}_{\mathrm{p}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{p}}\right)\end{array}\right]$$

(4)

$${\mathrm{n}}_{\mathrm{p}\mathrm{y}}=\left|\begin{array}{ccc}i& j& k\\ {\mathrm{x}}_{\mathrm{p}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{p}}& {\mathrm{y}}_{\mathrm{p}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{p}}& f_p\\ 0& 1& 0\end{array}\right|=-f_p i+\left({\mathrm{x}}_{\mathrm{p}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{p}}\right) k=\left[\begin{array}{c}-f_p\\ 0\\ \left({\mathrm{x}}_{\mathrm{p}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{p}}\right)\end{array}\right]$$

(5)

Computation of Light Plane Normals to the Camera Side

On the camera side, the normal to the light plane ∏CP₁P₂ can be calculated from the smaller plane ∏CP_c1P_c2. The normal to this plane is simply the vector cross-product of the position vector at point, P_c1, and the unit vector between points, P_c1 and P_c2. Similarly, the normal to the light plane ∏CP₁P₃ can be calculated from the smaller plane ∏CP_c1P_c3. The normal to this plane is also the vector cross-product of the position vector at point, P_c1, and the unit vector between the points, P_c1 and P_c3.

$${\mathrm{n}}_{\mathrm{c}}=\left[\begin{array}{c}{\mathrm{x}}_{\mathrm{c}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{c}}\\ {\mathrm{y}}_{\mathrm{c}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{c}}\\ f_c\end{array}\right]X\left[\begin{array}{c}{\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}}\\ {\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}}\\ 0\end{array}\right]=\left|\begin{array}{ccc}i& j& k\\ {\mathrm{x}}_{\mathrm{c}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{c}}& {\mathrm{y}}_{\mathrm{c}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{c}}& f_c\\ {\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}}& {\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}}& 0\end{array}\right|$$

(6)

where ĉ_cx and ĉ_cy are the components of the unit vector between the points (P_c1 to P_c2) or (P_c1 to P_c3). The arc between the real-world points, P₁ to P₂ and P₁ to P₃, on the 3D object surface will appear as the curves in the camera image points, P_c1 to P_c2 and P_c1 to P_c3. So, if the unit vector is calculated between P_c1 and right-side neighborhood point, P_c2, it can be computed as:

$${\mathrm{ĉ}}_{\mathrm{c}}={\displaystyle \frac{\left({\mathrm{x}}_{\mathrm{c}2}-{\mathrm{x}}_{\mathrm{c}1}\right) i+\left({\mathrm{y}}_{\mathrm{c}2}-{\mathrm{y}}_{\mathrm{c}1}\right) j}{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{c}2}-{\mathrm{x}}_{\mathrm{c}1}\right)}^2+{({\mathrm{y}}_{\mathrm{c}2}-{\mathrm{y}}_{\mathrm{c}1})}^2} }}={\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}} i+{\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}} j$$

(7)

where ${\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}}={\displaystyle \frac{\left({\mathrm{x}}_{\mathrm{c}2}-{\mathrm{x}}_{\mathrm{c}1}\right)}{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{c}2}-{\mathrm{x}}_{\mathrm{c}1}\right)}^2+{({\mathrm{y}}_{\mathrm{c}2}-{\mathrm{y}}_{\mathrm{c}1})}^2} }}$, and ${\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}}={\displaystyle \frac{\left({\mathrm{y}}_{\mathrm{c}2}-{\mathrm{y}}_{\mathrm{c}1}\right) }{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{c}2}-{\mathrm{x}}_{\mathrm{c}1}\right)}^2+{({\mathrm{y}}_{\mathrm{c}2}-{\mathrm{y}}_{\mathrm{c}1})}^2}}}$

Since the unit vector of the curve between P_c1 and the right-side neighborhood point, P_c2 is calculated. So (x_c2, y_c2) can be written as (x_RNB, y_RNB) while the initial point (x_c1, y_c1) is the point where the surface normal is computed. Therefore, Equation (7) will become:

$${\mathrm{ĉ}}_{\mathrm{c}\mathrm{R}\mathrm{N}\mathrm{B}}={\displaystyle \frac{\left({\mathrm{x}}_{\mathrm{R}\mathrm{N}\mathrm{B}}-{\mathrm{x}}_{\mathrm{c}1}\right) i+\left({\mathrm{y}}_{\mathrm{R}\mathrm{N}\mathrm{B}}-{\mathrm{y}}_{\mathrm{c}1}\right) j}{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{R}\mathrm{N}\mathrm{B}}-{\mathrm{x}}_{\mathrm{c}1}\right)}^2+{({\mathrm{y}}_{\mathrm{R}\mathrm{N}\mathrm{B}}-{\mathrm{y}}_{\mathrm{c}1})}^2} }}={\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{R}\mathrm{N}\mathrm{B}} i+{\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{R}\mathrm{N}\mathrm{B}} j$$

(8)

where ${\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{R}\mathrm{N}\mathrm{B}}={\displaystyle \frac{\left({\mathrm{x}}_{\mathrm{R}\mathrm{N}\mathrm{B}}-{\mathrm{x}}_{\mathrm{c}1}\right)}{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{R}\mathrm{N}\mathrm{B}}-{\mathrm{x}}_{\mathrm{c}1}\right)}^2+{({\mathrm{y}}_{\mathrm{R}\mathrm{N}\mathrm{B}}-{\mathrm{y}}_{\mathrm{c}1})}^2} }}$, and ${\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{R}\mathrm{N}\mathrm{B}}={\displaystyle \frac{\left({\mathrm{y}}_{\mathrm{R}\mathrm{N}\mathrm{B}}-{\mathrm{y}}_{\mathrm{c}1}\right) }{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{R}\mathrm{N}\mathrm{B}}-{\mathrm{x}}_{\mathrm{c}1}\right)}^2+{({\mathrm{y}}_{\mathrm{R}\mathrm{N}\mathrm{B}}-{\mathrm{y}}_{\mathrm{c}1})}^2}}}$

Therefore, the normal to the light plane ∏CP₁P₂ can be calculated using Equation (6):

$$\begin{gathered} {\mathrm{n}}_{\mathrm{c}\mathrm{x}}=\left[\begin{array}{c}{\mathrm{x}}_{\mathrm{c}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{c}}\\ {\mathrm{y}}_{\mathrm{c}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{c}}\\ f_c\end{array}\right]X\left[\begin{array}{c}{\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{R}\mathrm{N}\mathrm{B}}\\ {\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{R}\mathrm{N}\mathrm{B}}\\ 0\end{array}\right]=\left|\begin{array}{ccc}i& j& k\\ {\mathrm{x}}_{\mathrm{c}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{c}}& {\mathrm{y}}_{\mathrm{c}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{c}}& f_c\\ {\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{R}\mathrm{N}\mathrm{B}}& {\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{R}\mathrm{N}\mathrm{B}}& 0\end{array}\right|=-{{\mathrm{ĉ}}_{cyRNB}f}_c i+{\mathrm{ĉ}}_{cxRNB}{f}_{c }j+ \\ \left\{\left({\mathrm{x}}_{\mathrm{c}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{c}}\right){\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{R}\mathrm{N}\mathrm{B}}-{\left({\mathrm{y}}_{\mathrm{c}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{c}}\right)\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{R}\mathrm{N}\mathrm{B}}\right\} k=\left[\begin{array}{c}-{\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{R}\mathrm{N}\mathrm{B}} f_c\\ {\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{R}\mathrm{N}\mathrm{B}} f_{c }\\ \left({\mathrm{x}}_{\mathrm{c}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{c}}\right){\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{R}\mathrm{N}\mathrm{B}}-{\left({\mathrm{y}}_{\mathrm{c}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{c}}\right)\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{R}\mathrm{N}\mathrm{B}}\end{array}\right] \end{gathered}$$

(9)

Similarly, if the unit vector is calculated between P_c1 and down-side neighborhood point, P_c3, it can be computed as:

$${\mathrm{ĉ}}_{\mathrm{c}}={\displaystyle \frac{\left({\mathrm{x}}_{\mathrm{c}3}-{\mathrm{x}}_{\mathrm{c}1}\right) i+\left({\mathrm{y}}_{\mathrm{c}3}-{\mathrm{y}}_{\mathrm{c}1}\right) j}{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{c}3}-{\mathrm{x}}_{\mathrm{c}1}\right)}^2+{({\mathrm{y}}_{\mathrm{c}3}-{\mathrm{y}}_{\mathrm{c}1})}^2} }}={\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}} i+{\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}} j$$

(10)

where ${\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}}={\displaystyle \frac{\left({\mathrm{x}}_{\mathrm{c}3}-{\mathrm{x}}_{\mathrm{c}1}\right)}{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{c}3}-{\mathrm{x}}_{\mathrm{c}1}\right)}^2+{({\mathrm{y}}_{\mathrm{c}3}-{\mathrm{y}}_{\mathrm{c}1})}^2} }}$, and ${\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}}={\displaystyle \frac{\left({\mathrm{y}}_{\mathrm{c}3}-{\mathrm{y}}_{\mathrm{c}1}\right) }{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{c}3}-{\mathrm{x}}_{\mathrm{c}1}\right)}^2+{({\mathrm{y}}_{\mathrm{c}3}-{\mathrm{y}}_{\mathrm{c}1})}^2}}}$

Since the unit vector of the curve between P_c1 and the right-side neighborhood point, P_c3 is calculated. So (x_c3, y_c3) can be written as (x_DNB, y_DNB) while the initial point (x_c1, y_c1) is the point where the surface normal is computed. Therefore, Equation (10) will become:

$${\mathrm{ĉ}}_{\mathrm{c}\mathrm{D}\mathrm{N}\mathrm{B}}={\displaystyle \frac{\left({\mathrm{x}}_{\mathrm{D}\mathrm{N}\mathrm{B}}-{\mathrm{x}}_{\mathrm{c}1}\right) i+\left({\mathrm{y}}_{\mathrm{D}\mathrm{N}\mathrm{B}}-{\mathrm{y}}_{\mathrm{c}1}\right) j}{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{D}\mathrm{N}\mathrm{B}}-{\mathrm{x}}_{\mathrm{c}1}\right)}^2+{({\mathrm{y}}_{\mathrm{D}\mathrm{N}\mathrm{B}}-{\mathrm{y}}_{\mathrm{c}1})}^2} }}={\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{D}\mathrm{N}\mathrm{B}} i+{\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{D}\mathrm{N}\mathrm{B}} j$$

(11)

where ${\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{D}\mathrm{N}\mathrm{B}}={\displaystyle \frac{\left({\mathrm{x}}_{\mathrm{D}\mathrm{N}\mathrm{B}}-{\mathrm{x}}_{\mathrm{c}1}\right)}{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{D}\mathrm{N}\mathrm{B}}-{\mathrm{x}}_{\mathrm{c}1}\right)}^2+{({\mathrm{y}}_{\mathrm{D}\mathrm{N}\mathrm{B}}-{\mathrm{y}}_{\mathrm{c}1})}^2} }}$, and ${\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{D}\mathrm{N}\mathrm{B}}={\displaystyle \frac{\left({\mathrm{y}}_{\mathrm{D}\mathrm{N}\mathrm{B}}-{\mathrm{y}}_{\mathrm{c}1}\right) }{\sqrt[2]{{\left({\mathrm{x}}_{\mathrm{D}\mathrm{N}\mathrm{B}}-{\mathrm{x}}_{\mathrm{c}1}\right)}^2+{({\mathrm{y}}_{\mathrm{D}\mathrm{N}\mathrm{B}}-{\mathrm{y}}_{\mathrm{c}1})}^2}}}$

Therefore, the normal to the light plane ∏CP₁P₃ can be calculated using Equation (6):

$$\begin{gathered} {\mathrm{n}}_{\mathrm{c}\mathrm{y}}=\left[\begin{array}{c}{\mathrm{x}}_{\mathrm{c}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{c}}\\ {\mathrm{y}}_{\mathrm{c}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{c}}\\ f_c\end{array}\right]X\left[\begin{array}{c}{\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{D}\mathrm{N}\mathrm{B}}\\ {\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{D}\mathrm{N}\mathrm{B}}\\ 0\end{array}\right]=\left|\begin{array}{ccc}i& j& k\\ {\mathrm{x}}_{\mathrm{c}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{c}}& {\mathrm{y}}_{\mathrm{c}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{c}}& f_c\\ {\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{D}\mathrm{N}\mathrm{B}}& {\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{D}\mathrm{N}\mathrm{B}}& 0\end{array}\right|=-{{\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{D}\mathrm{N}\mathrm{B}}f}_c i+{\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{D}\mathrm{N}\mathrm{B}}{f}_{c }j+ \\ \left\{\left({\mathrm{x}}_{\mathrm{c}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{c}}\right){\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{D}\mathrm{N}\mathrm{B}}-{\left({\mathrm{y}}_{\mathrm{c}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{c}}\right)\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{D}\mathrm{N}\mathrm{B}}\right\} k=\left[\begin{array}{c}-{\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{D}\mathrm{N}\mathrm{B}} f_c\\ {\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{D}\mathrm{N}\mathrm{B}} f_{c }\\ \left({\mathrm{x}}_{\mathrm{c}1}-{\mathrm{x}}_{\mathrm{o}\mathrm{c}}\right){\mathrm{ĉ}}_{\mathrm{c}\mathrm{y}\mathrm{D}\mathrm{N}\mathrm{B}}-{\left({\mathrm{y}}_{\mathrm{c}1}-{\mathrm{y}}_{\mathrm{o}\mathrm{c}}\right)\mathrm{ĉ}}_{\mathrm{c}\mathrm{x}\mathrm{D}\mathrm{N}\mathrm{B}}\end{array}\right] \end{gathered}$$

(12)

Computation of Surface Tangents

The two light planes ∏PP₁P₂ (with the normal denoted as ‘n_px’), which is passing from the projected pattern plane, and ∏CP₁P₂ (with the normal denoted as ‘n_cx’), which is passing from the camera image plane, intersect on the 3D object surface on the arc forming between P₁ and P₂, which are the centroids of the current symbol and the right-side neighborhood symbol in a horizontal location. These light planes are called the x-planes of the projector and camera sides. Similarly, the two light planes ∏PP₁P₃ (with the normal denoted as ‘n_py’), which is passing from the projected pattern plane, and ∏CP₁P₃ (with the normal denoted as ‘n_cy’), which is passing from the camera image plane, intersect on the 3D object surface on the arc between P₁ and P₃, which are the centroids of the current symbol and the down-side neighborhood symbol in a vertical location. These light planes are called the y-planes of the projector and camera sides. After accommodating translation and rotation on both the camera and the projector side, the surface tangents will be calculated. The vectors, n_cx, n_cy, n_px, and n_py, after rotation and translation, will take the following form:

$${{\mathrm{T}}_{\mathrm{c}}{\mathrm{R}}_{\mathrm{c}}\mathrm{n}}_{\mathrm{c}\mathrm{x}}=\left[\begin{array}{c}{\mathrm{n}}_{\mathrm{c}\mathrm{x}1}\\ {\mathrm{n}}_{\mathrm{c}\mathrm{x}2}\\ {\mathrm{n}}_{\mathrm{c}\mathrm{x}3}\end{array}\right],\mathrm{a}\mathrm{n}\mathrm{d} {{\mathrm{T}}_{\mathrm{c}}{\mathrm{R}}_{\mathrm{c}}\mathrm{n}}_{\mathrm{c}\mathrm{y}}=\left[\begin{array}{c}{\mathrm{n}}_{\mathrm{c}\mathrm{y}1}\\ {\mathrm{n}}_{\mathrm{c}\mathrm{y}2}\\ {\mathrm{n}}_{\mathrm{c}\mathrm{y}3}\end{array}\right]$$

(13)

$${\mathrm{T}}_{\mathrm{p}}{\mathrm{R}}_{\mathrm{p}}{\mathrm{n}}_{\mathrm{p}\mathrm{x}}=\left[\begin{array}{c}{\mathrm{n}}_{\mathrm{p}\mathrm{x}1}\\ {\mathrm{n}}_{\mathrm{p}\mathrm{x}2}\\ {\mathrm{n}}_{\mathrm{p}\mathrm{x}3}\end{array}\right],\mathrm{a}\mathrm{n}\mathrm{d} {{\mathrm{T}}_{\mathrm{p}}{\mathrm{R}}_{\mathrm{p}}\mathrm{n}}_{\mathrm{p}\mathrm{y}}=\left[\begin{array}{c}{\mathrm{n}}_{\mathrm{p}\mathrm{y}1}\\ {\mathrm{n}}_{\mathrm{p}\mathrm{y}2}\\ {\mathrm{n}}_{\mathrm{p}\mathrm{y}3}\end{array}\right]$$

(14)

where R_p and R_c are the 3 × 3 rotation matrices, T_p and T_c are the 3 × 1 translation matrices with respect to world coordinates, or 3D object coordinates for the projector and camera, respectively. The rotation and translation are known as external or extrinsic calibration parameters for the camera and projector.

The tangent along the X-direction on the object surface will be the vector cross-product of the normals of the two corresponding intersecting x-planes:

$${\mathrm{t}}_{\mathrm{x}}={{(\mathrm{T}}_{\mathrm{c}}{\mathrm{R}}_{\mathrm{c}}\mathrm{n}}_{\mathrm{c}\mathrm{x}}\left)X\right({\mathrm{T}}_{\mathrm{p}}{\mathrm{R}}_{\mathrm{p}}{\mathrm{n}}_{\mathrm{p}\mathrm{x}})$$

(15)

The tangent along the Y-direction on the object surface, will be the vector cross-product of the normal of the two corresponding intersecting y-planes:

$${\mathrm{t}}_{\mathrm{y}}=\left({{\mathrm{T}}_{\mathrm{c}}{\mathrm{R}}_{\mathrm{c}}\mathrm{n}}_{\mathrm{c}\mathrm{y}}\right)X\left({\mathrm{T}}_{\mathrm{p}}{\mathrm{R}}_{\mathrm{p}}{\mathrm{n}}_{\mathrm{p}\mathrm{y}}\right)$$

(16)

The tangents on the object surface, t_x and t_y, will then be calculated as follows:

$$\begin{gathered} {\mathrm{t}}_{\mathrm{x}}=\left[\begin{array}{c}{\mathrm{n}}_{\mathrm{c}\mathrm{x}1}\\ {\mathrm{n}}_{\mathrm{c}\mathrm{x}2}\\ {\mathrm{n}}_{\mathrm{c}\mathrm{x}3}\end{array}\right]X\left[\begin{array}{c}{\mathrm{n}}_{\mathrm{p}\mathrm{x}1}\\ {\mathrm{n}}_{\mathrm{p}\mathrm{x}2}\\ {\mathrm{n}}_{\mathrm{p}\mathrm{x}3}\end{array}\right]=\left|\begin{array}{ccc}i& j& k\\ {\mathrm{n}}_{\mathrm{c}\mathrm{x}1}& {\mathrm{n}}_{\mathrm{c}\mathrm{x}2}& {\mathrm{n}}_{\mathrm{c}\mathrm{x}3}\\ {\mathrm{n}}_{\mathrm{p}\mathrm{x}1}& {\mathrm{n}}_{\mathrm{p}\mathrm{x}2}& {\mathrm{n}}_{\mathrm{p}\mathrm{x}3}\end{array}\right|={(\mathrm{n}}_{\mathrm{c}\mathrm{x}2}{\mathrm{n}}_{\mathrm{p}\mathrm{x}3}-{\mathrm{n}}_{\mathrm{c}\mathrm{x}3}{\mathrm{n}}_{\mathrm{p}\mathrm{x}2})i+({\mathrm{n}}_{\mathrm{c}\mathrm{x}3}{\mathrm{n}}_{\mathrm{p}\mathrm{x}1} \\ -{\mathrm{n}}_{\mathrm{c}\mathrm{x}1}{\mathrm{n}}_{\mathrm{p}\mathrm{x}3})j+({\mathrm{n}}_{\mathrm{c}\mathrm{x}1}{\mathrm{n}}_{\mathrm{p}\mathrm{x}2}-{\mathrm{n}}_{\mathrm{c}\mathrm{x}2}{\mathrm{n}}_{\mathrm{p}\mathrm{x}1})k=\left[\begin{array}{c}{(\mathrm{n}}_{\mathrm{c}\mathrm{x}2}{\mathrm{n}}_{\mathrm{p}\mathrm{x}3}-{\mathrm{n}}_{\mathrm{c}\mathrm{x}3}{\mathrm{n}}_{\mathrm{p}\mathrm{x}2})\\ ({\mathrm{n}}_{\mathrm{c}\mathrm{x}3}{\mathrm{n}}_{\mathrm{p}\mathrm{x}1}-{\mathrm{n}}_{\mathrm{c}\mathrm{x}1}{\mathrm{n}}_{\mathrm{p}\mathrm{x}3})\\ ({\mathrm{n}}_{\mathrm{c}\mathrm{x}1}{\mathrm{n}}_{\mathrm{p}\mathrm{x}2}-{\mathrm{n}}_{\mathrm{c}\mathrm{x}2}{\mathrm{n}}_{\mathrm{p}\mathrm{x}1})\end{array}\right]=\left[\begin{array}{c}{\mathrm{t}}_{\mathrm{x}1}\\ {\mathrm{t}}_{\mathrm{x}2}\\ {\mathrm{t}}_{\mathrm{x}3}\end{array}\right] \end{gathered}$$

(17)

$$\begin{gathered} {\mathrm{t}}_{\mathrm{y}}=\left[\begin{array}{c}{\mathrm{n}}_{\mathrm{c}\mathrm{y}1}\\ {\mathrm{n}}_{\mathrm{c}\mathrm{y}2}\\ {\mathrm{n}}_{\mathrm{c}\mathrm{y}3}\end{array}\right]X\left[\begin{array}{c}{\mathrm{n}}_{\mathrm{p}\mathrm{y}1}\\ {\mathrm{n}}_{\mathrm{p}\mathrm{y}2}\\ {\mathrm{n}}_{\mathrm{p}\mathrm{y}3}\end{array}\right]=\left|\begin{array}{ccc}i& j& k\\ {\mathrm{n}}_{\mathrm{c}\mathrm{y}1}& {\mathrm{n}}_{\mathrm{c}\mathrm{y}2}& {\mathrm{n}}_{\mathrm{c}\mathrm{y}3}\\ {\mathrm{n}}_{\mathrm{p}\mathrm{y}1}& {\mathrm{n}}_{\mathrm{p}\mathrm{y}2}& {\mathrm{n}}_{\mathrm{p}\mathrm{y}3}\end{array}\right|={(\mathrm{n}}_{\mathrm{c}\mathrm{y}2}{\mathrm{n}}_{\mathrm{p}\mathrm{y}3}-{\mathrm{n}}_{\mathrm{c}\mathrm{y}3}{\mathrm{n}}_{\mathrm{p}\mathrm{y}2}) i+({\mathrm{n}}_{\mathrm{c}\mathrm{y}3}{\mathrm{n}}_{\mathrm{p}\mathrm{y}1}- \\ {\mathrm{n}}_{\mathrm{c}\mathrm{y}1}{\mathrm{n}}_{\mathrm{p}\mathrm{y}3}) j+({\mathrm{n}}_{\mathrm{c}\mathrm{y}1}{\mathrm{n}}_{\mathrm{p}\mathrm{y}2}-{\mathrm{n}}_{\mathrm{c}\mathrm{y}2}{\mathrm{n}}_{\mathrm{p}\mathrm{y}1}) k=\left[\begin{array}{c}{(\mathrm{n}}_{\mathrm{c}\mathrm{y}2}{\mathrm{n}}_{\mathrm{p}\mathrm{y}3}-{\mathrm{n}}_{\mathrm{c}\mathrm{y}3}{\mathrm{n}}_{\mathrm{p}\mathrm{y}2})\\ \left({\mathrm{n}}_{\mathrm{c}\mathrm{y}3}{\mathrm{n}}_{\mathrm{p}\mathrm{y}1}-{\mathrm{n}}_{\mathrm{c}\mathrm{y}1}{\mathrm{n}}_{\mathrm{p}\mathrm{y}3}\right)\\ \left({\mathrm{n}}_{\mathrm{c}\mathrm{y}1}{\mathrm{n}}_{\mathrm{p}\mathrm{y}2}-{\mathrm{n}}_{\mathrm{c}\mathrm{y}2}{\mathrm{n}}_{\mathrm{p}\mathrm{y}1}\right)\end{array}\right]=\left[\begin{array}{c}{\mathrm{t}}_{\mathrm{y}1}\\ {\mathrm{t}}_{\mathrm{y}2}\\ {\mathrm{t}}_{\mathrm{y}3}\end{array}\right] \end{gathered}$$

(18)

The surface normal will be the vector cross-product of the two tangent vectors and will be computed as:

$$\mathrm{n}\left(\mathrm{X},\mathrm{Y},\mathrm{Z}\right)={\mathrm{t}}_{\mathrm{x}}X{\mathrm{t}}_{\mathrm{y}}=\left[\begin{array}{c}{\mathrm{t}}_{\mathrm{x}1}\\ {\mathrm{t}}_{\mathrm{x}2}\\ {\mathrm{t}}_{\mathrm{x}3}\end{array}\right]X\left[\begin{array}{c}{\mathrm{t}}_{\mathrm{y}1}\\ {\mathrm{t}}_{\mathrm{y}2}\\ {\mathrm{t}}_{\mathrm{y}3}\end{array}\right]=\left|\begin{array}{ccc}i& j& k\\ {\mathrm{t}}_{\mathrm{x}1}& {\mathrm{t}}_{\mathrm{x}2}& {\mathrm{t}}_{\mathrm{x}3}\\ {\mathrm{t}}_{\mathrm{y}1}& {\mathrm{t}}_{\mathrm{y}2}& {\mathrm{t}}_{\mathrm{y}3}\end{array}\right|=\left({\mathrm{t}}_{\mathrm{y}3}{\mathrm{t}}_{\mathrm{x}2}-{\mathrm{t}}_{\mathrm{y}2}{\mathrm{t}}_{\mathrm{x}3}\right)i+\left({\mathrm{t}}_{\mathrm{y}1}{\mathrm{t}}_{\mathrm{x}3}-{\mathrm{t}}_{\mathrm{y}3}{\mathrm{t}}_{\mathrm{x}1}\right)j+\left({\mathrm{t}}_{\mathrm{y}2}{\mathrm{t}}_{\mathrm{x}1}-{\mathrm{t}}_{\mathrm{y}1}{\mathrm{t}}_{\mathrm{x}2}\right)k=\left[\begin{array}{c}{\mathrm{t}}_{\mathrm{y}3}{\mathrm{t}}_{\mathrm{x}2}-{\mathrm{t}}_{\mathrm{y}2} {\mathrm{t}}_{\mathrm{x}3}\\ {\mathrm{t}}_{\mathrm{y}1}{\mathrm{t}}_{\mathrm{x}3}-{\mathrm{t}}_{\mathrm{y}3}{\mathrm{t}}_{\mathrm{x}1}\\ {\mathrm{t}}_{\mathrm{y}2}{\mathrm{t}}_{\mathrm{x}1}-{\mathrm{t}}_{\mathrm{y}1}{\mathrm{t}}_{\mathrm{x}2}\end{array}\right]=\left[\begin{array}{c}{\mathrm{n}}_{\mathrm{x}}\\ {\mathrm{n}}_{\mathrm{y}}\\ {\mathrm{n}}_{\mathrm{z}}\end{array}\right]$$

(19)

The computation of surface normals using Equation (19) is a totally deterministic process that requires only image and projection information local to a particular point. Unlike traditional methods, such as shape from shading or texture, it does not necessitate the assumptions of the relationship between the orientations of neighborhood points. Computed surface normals are calculated from the finitely small light planes. Their values are normalized by computing the unit vectors as follows:

$$\widehat{\mathrm{n}}\left(\mathrm{X},\mathrm{Y},\mathrm{Z}\right)={\displaystyle \frac{{\mathrm{n}}_{\mathrm{x}}i+{\mathrm{n}}_{\mathrm{y}}j+{\mathrm{n}}_{\mathrm{z}}k}{\sqrt[2]{{{\mathrm{n}}_{\mathrm{x}}}^2+{{\mathrm{n}}_{\mathrm{y}}}^2+{{\mathrm{n}}_{\mathrm{z}}}^2}}}$$

(20)

2.3. Computation of 3D World Coordinates [43,44,45,46]

The projective transformation from 3D world coordinates to the camera and projector coordinates can be written as:

$$\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{C}}\\ {\mathrm{Y}}_{\mathrm{C}}\\ {\mathrm{Z}}_{\mathrm{C}}\\ 1\end{array}\right]=\left[\begin{array}{c}{\mathrm{R}}_{\mathrm{c}} {\mathrm{T}}_{\mathrm{c}}\\ 0 1\end{array}\right]\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{w}}\\ {\mathrm{Y}}_{\mathrm{w}}\\ {\mathrm{Z}}_{\mathrm{w}}\\ 1\end{array}\right]=M_c\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{w}}\\ {\mathrm{Y}}_{\mathrm{w}}\\ {\mathrm{Z}}_{\mathrm{w}}\\ 1\end{array}\right]$$

(21)

$$\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{P}}\\ {\mathrm{Y}}_{\mathrm{P}}\\ {\mathrm{Z}}_{\mathrm{P}}\\ 1\end{array}\right]=\left[\begin{array}{c}{\mathrm{R}}_{\mathrm{P}} {\mathrm{T}}_{\mathrm{P}}\\ 0 1\end{array}\right]\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{w}}\\ {\mathrm{Y}}_{\mathrm{w}}\\ {\mathrm{Z}}_{\mathrm{w}}\\ 1\end{array}\right]=M_p\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{w}}\\ {\mathrm{Y}}_{\mathrm{w}}\\ {\mathrm{Z}}_{\mathrm{w}}\\ 1\end{array}\right]$$

(22)

After including the internal or intrinsic camera parameters, the following transformation will be obtained:

$${\mathrm{Z}}_{\mathrm{c}}\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{c}}\\ {\mathrm{v}}_{\mathrm{c}}\end{array}\right]=\left[\begin{array}{ccc}{f}_c& 0& {\mathrm{x}}_{\mathrm{o}\mathrm{c}}\\ 0& f_c& {\mathrm{y}}_{\mathrm{o}\mathrm{c}}\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cc}{\mathrm{R}}_{\mathrm{c}}& {\mathrm{T}}_{\mathrm{c}}\\ 0& 1\end{array}\right]\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{w}}\\ {\mathrm{Y}}_{\mathrm{W}}\\ {\mathrm{Z}}_{\mathrm{W}}\\ 1\end{array}\right]$$

(23)

$${\mathrm{Z}}_{\mathrm{p}}\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{p}}\\ {\mathrm{v}}_{\mathrm{p}}\end{array}\right]=\left[\begin{array}{ccc}{f}_p& 0& {\mathrm{x}}_{\mathrm{o}\mathrm{p}}\\ 0& f_p& {\mathrm{y}}_{\mathrm{o}\mathrm{p}}\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cc}{\mathrm{R}}_{\mathrm{p}}& {\mathrm{T}}_{\mathrm{p}}\\ 0& 1\end{array}\right]\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{w}}\\ {\mathrm{Y}}_{\mathrm{W}}\\ {\mathrm{Z}}_{\mathrm{W}}\\ 1\end{array}\right]$$

(24)

The camera and projector coordinates are transformed into a 2D image and pattern plane (u_n, v_n) with the following relationship:

$$\mathrm{Z}\left[\begin{array}{c}{\displaystyle \frac{\mathrm{X}}{\mathrm{Z}}}\\ {\displaystyle \frac{\mathrm{Y}}{\mathrm{Z}}}\end{array}\right]=\mathrm{Z}\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{n}}\\ {\mathrm{v}}_{\mathrm{n}}\end{array}\right]=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}\mathrm{X}\\ \mathrm{Y}\\ \mathrm{Z}\\ 1\end{array}\right]$$

(25)

The camera and projector lens distortions, including radial distortion and tangential distortion, can be defined as:

$${\mathrm{m}}_{\mathrm{c}\mathrm{d}}=\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{c}\mathrm{d}}\\ {\mathrm{v}}_{\mathrm{c}\mathrm{d}}\end{array}\right]=\left[1+{\mathrm{k}}_{\mathrm{c}1}{\mathrm{r}}_{\mathrm{c}}^2+{\mathrm{k}}_{\mathrm{c}2}{\mathrm{r}}_{\mathrm{c}}^4\right]\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{c}\mathrm{n}}\\ {\mathrm{v}}_{\mathrm{c}\mathrm{n}}\end{array}\right]+{\mathrm{d}\mathrm{x}}_{\mathrm{c}}$$

(26)

$${\mathrm{m}}_{\mathrm{p}\mathrm{d}}=\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{p}\mathrm{d}}\\ {\mathrm{v}}_{\mathrm{p}\mathrm{d}}\end{array}\right]=\left[1+{\mathrm{k}}_{\mathrm{p}1}{\mathrm{r}}_{\mathrm{p}}^2+{\mathrm{k}}_{\mathrm{p}2}{\mathrm{r}}_{\mathrm{p}}^4\right]\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{p}\mathrm{n}}\\ {\mathrm{v}}_{\mathrm{p}\mathrm{n}}\end{array}\right]+{\mathrm{d}\mathrm{x}}_{\mathrm{p}}$$

(27)

where (u_cd, v_cd) and (u_pd, v_pd) are the camera and projector digital image coordinates after accommodating distortion effects. (u_cn, v_cn) and (u_pn, v_pn) are the coordinates of the camera and projector image plane after transformation from 3D to 2D. r_c² = u_cn² + v_cn² and r_p² = u_pn² + v_pn². k_c1, k_c2, and k_p1, k_p2 are the radial distortion parameters of the camera and projector. The tangential distortion for the camera and projector will be defined as:

$${\mathrm{d}\mathrm{x}}_{\mathrm{c}}=\left[\begin{array}{c}2{\mathrm{p}}_{1\mathrm{c}}{\mathrm{u}}_{\mathrm{c}\mathrm{n}}{\mathrm{v}}_{\mathrm{c}\mathrm{n}}+{\mathrm{p}}_{2\mathrm{c}}\left({{\mathrm{r}}_{\mathrm{c}}}^2+2{{\mathrm{u}}_{\mathrm{c}\mathrm{n}}}^2\right)\\ {\mathrm{p}}_{1\mathrm{c}}\left({\mathrm{r}}_{\mathrm{c}}^2+2{{\mathrm{v}}_{\mathrm{c}\mathrm{n}}}^2\right)+2{\mathrm{p}}_{2\mathrm{c}}{\mathrm{u}}_{\mathrm{c}\mathrm{n}}{\mathrm{v}}_{\mathrm{c}\mathrm{n}}\end{array}\right]$$

(28)

$${\mathrm{d}\mathrm{x}}_{\mathrm{p}}=\left[\begin{array}{c}2{\mathrm{p}}_{1\mathrm{p}}{\mathrm{u}}_{\mathrm{p}\mathrm{n}}{\mathrm{v}}_{\mathrm{p}\mathrm{n}}+{\mathrm{p}}_{2\mathrm{p}}\left({{\mathrm{r}}_{\mathrm{p}}}^2+2{{\mathrm{u}}_{\mathrm{p}\mathrm{n}}}^2\right)\\ {\mathrm{p}}_{1\mathrm{p}}\left({\mathrm{r}}_{\mathrm{p}}^2+2{{\mathrm{v}}_{\mathrm{p}\mathrm{n}}}^2\right)+2{\mathrm{p}}_{2\mathrm{p}}{\mathrm{u}}_{\mathrm{p}\mathrm{n}}{\mathrm{v}}_{\mathrm{p}\mathrm{n}}\end{array}\right]$$

(29)

where P_1c, P_2c, and P_1p, P_2p are the camera and projector’s tangential distortion parameters.

The complete transformation from 3D world coordinates to the digital image and pattern after accommodation of the camera’s extrinsic and intrinsic parameters and lens distortion for the camera and projector will be:

$${\mathrm{Z}}_{\mathrm{p}}\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{p}}\\ {\mathrm{v}}_{\mathrm{p}}\\ 1\\ 1\end{array}\right]={\mathrm{m}}_{\mathrm{p}\mathrm{d}}\left[\begin{array}{ccc}{f}_p& 0& {\mathrm{x}}_{\mathrm{o}\mathrm{p}}\\ 0& f_p& {\mathrm{y}}_{\mathrm{o}\mathrm{p}}\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cc}{R}_p& T_p\\ 0& 1\end{array}\right]\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{w}}\\ {\mathrm{Y}}_{\mathrm{W}}\\ {\mathrm{Z}}_{\mathrm{W}}\\ 1\end{array}\right]=\left[\begin{array}{cccc}{n}_{11}& n_{12}& n_{13}& n_{14}\\ n_{21}& n_{22}& n_{23}& n_{24}\\ n_{31}& n_{32}& n_{33}& n_{34}\end{array}\right]\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{w}}\\ {\mathrm{Y}}_{\mathrm{W}}\\ {\mathrm{Z}}_{\mathrm{W}}\\ 1\end{array}\right]$$

(30)

$${\mathrm{Z}}_{\mathrm{c}}\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{c}}\\ {\mathrm{v}}_{\mathrm{c}}\\ 1\\ 1\end{array}\right]={\mathrm{m}}_{\mathrm{c}\mathrm{d}}\left[\begin{array}{ccc}{f}_c& 0& {\mathrm{x}}_{\mathrm{o}\mathrm{c}}\\ 0& f_c& {\mathrm{y}}_{\mathrm{o}\mathrm{c}}\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cc}{\mathrm{R}}_{\mathrm{c}}& {\mathrm{T}}_{\mathrm{c}}\\ 0& 1\end{array}\right]\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{w}}\\ {\mathrm{Y}}_{\mathrm{W}}\\ {\mathrm{Z}}_{\mathrm{W}}\\ 1\end{array}\right]=\left[\begin{array}{cccc}{m}_{11}& m_{12}& m_{13}& m_{14}\\ m_{21}& m_{22}& m_{23}& m_{24}\\ m_{31}& m_{32}& m_{33}& m_{34}\end{array}\right]\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{w}}\\ {\mathrm{Y}}_{\mathrm{W}}\\ {\mathrm{Z}}_{\mathrm{W}}\\ 1\end{array}\right]$$

(31)

Eliminating Z_p and Z_c using the plane line triangulation method [47], the 3D world coordinates can be calculated as:

$$\left[\begin{array}{ccc}{m}_{31}{\mathrm{u}}_{\mathrm{c}}-m_{11}& m_{32}{\mathrm{u}}_{\mathrm{c}}-m_{12}& m_{33}{\mathrm{u}}_{\mathrm{c}}-m_{13}\\ m_{31}{\mathrm{v}}_{\mathrm{c}}-m_{21}& m_{32}{\mathrm{v}}_{\mathrm{c}}-m_{22}& m_{33}{\mathrm{v}}_{\mathrm{c}}-m_{23}\\ n_{31}{\mathrm{u}}_{\mathrm{p}}-n_{11}& n_{32}{\mathrm{u}}_{\mathrm{p}}-n_{12}& n_{33}{\mathrm{u}}_{\mathrm{p}}-n_{13}\end{array}\right]\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{w}}\\ {\mathrm{Y}}_{\mathrm{W}}\\ {\mathrm{Z}}_{\mathrm{W}}\end{array}\right]=\left[\begin{array}{c}{m}_{14}-m_{34}{\mathrm{u}}_{\mathrm{c}}\\ m_{24}-m_{34}{\mathrm{v}}_{\mathrm{c}}\\ n_{14}-n_{34}{\mathrm{u}}_{\mathrm{p}}\end{array}\right]$$

(32)

2.4. Decoding of Patterns

2.4.1. Preprocessing, Segmentation, and Labeling

Preprocessing is carried out as described in our previous works [14,42] by using thresholding and segmentation by using the Otsu methods [48], and segmentation and labeling are carried out by applying the algorithm specified by Haralick [49].

2.4.2. Decoding, Classification, and Computation of Parameters

The decoding and classification of the symbols are carried out using techniques described in our previous works [14,42], using area functions and shape description parameters. The only difference here in this work is that we use four symbols in the pattern compared to the three symbols and stripes previously. The four symbols we employed in the experiment are butterfly-shaped diagonally arranged square symbols, square-shaped symbols, horizontal strips, and plus-shaped (+) symbols, as shown in Figure 5.

The square-shaped symbol and plus (+)-shaped symbols are differentiated from the horizontal stripe and diagonally arranged circles based on areas, eccentricity ratio, and aspect ratio. The square has a maximum area, and the plus sign also has a greater area than horizontal strips and diagonally arranged circles. The aspect ratio and eccentricity of the square and plus signs are lower, while stripe and diagonally arranged circles have higher values. The squares were then differentiated from the plus sign based on the solidity ratio and rectangularity ratio. The square-shaped symbols have higher solidity and rectangularity ratios compared to the plus sign, which has lower values. The horizontal strip and diagonally arranged squares are then differentiated based on their orientations, as horizontal stripes have an orientation around zero while diagonally arranged squares have an orientation around forty-five degrees.

2.4.3. Calibration

Before applying the projection patterns to the surface to be measured, the projector and the camera must be calibrated by using any of the techniques available in [50,51,52,53] by computing the intrinsic and extrinsic parameters.

2.5. Experiment and Devices

2.5.1. Camera and Projector Devices

The experimental setup comprised of a digital camera (DH-HV2051UC, Daheng Imaging, Beijing, China) and a DLP projector (DELL M110, Dell Technologies, Beijing, China). The camera has a pixel resolution of 1600 × 1200. It is a progressive scan CMOS device with an 8-bit pixel depth and 10-bit analog-to-digital conversion accuracy. The pixel size is 4.2 um × 4.2 um. It can acquire ten frames per second. The projector has a pixel resolution of 800 × 1280, with a projection or throw ratio of 1.5:1 and a contrast ratio of 10,000:1 (typical at full on or off).

2.5.2. Target Surfaces

We experimented on three surfaces: (1) a simple plane surface that is approximately 800 mm wide and 600 mm long, (2) a cylindrical surface that has a radius of 150 mm and height of 406 mm, and (3) a sculpture that has a height of 223 mm, width of 190 mm, and depth of 227 mm. The standard deviation of the cylindrical surface is equal to its radius of 150 mm. The estimated standard deviation of the textured surface, i.e., sculpture, is approximately 200 to 225 mm. The surfaces to be measured are white to avoid the influence of spatial varying reflectivity.

2.5.3. Experiment Setup

Our method was endorsed through a real experiment, and we appraised the system’s performance. The surface to be measured was 110 cm apart from the projector and camera, whereas the camera and projector were separated by 18 cm. We put the target surface in the mid-position of the projector and camera. The projector and camera were tilted at 85.3 degrees to throw a pattern and record the observation. The angle of incidence from the projector to the camera was 9.4 degrees. The ambient light was firmly controlled during the recording of images so that errors could be minimized.

Pattern Employed in the Experiment

We tested and proved our method using five projection patterns with resolutions of 8 × 8, 10 × 10, 12 × 12, 14 × 14, and 16 × 16 pixels. The five projection patterns differed only in resolution (8 × 8, 10 × 10, 12 × 12, 14 × 14, and 16 × 16) and their corresponding M-arrays. Table 1 shows that all the M-arrays had four symbols and a 3 × 3 window feature, with the only difference being their dimensions. The 8 × 8 resolution projection pattern employed a 90 × 144 M-array. The 10 × 10 pattern resolution used a 75 × 117 M-array. The 12 × 12 pattern resolution utilized a 60 × 93 M-array. The 14 × 14 pattern resolution used a 51 × 81 M-array. For the 16 × 16 resolution, a 45 × 72 M-array was deployed. For the 8 × 8 and 10 × 10 resolutions, 1-pixel spacing between the symbols was used, whereas for the 12 × 12, 14 × 14, and 16 × 16 resolutions 2-pixel spacing was used. Figure 5 shows the texture of the five projection patterns used in the experiment.

3. Results

3.1. Comparison of Measured Resolution

The system is designed to perform in between the depth ranges of 40 to 250 cm.

Table 2. Comparison of measured projector resolutions and the area covered.

Pattern Resolution	Depth (z) cm	Area (cm2)	Proposed Method	Zhou (2023) [16]	Yin (2021) [19]	Nguyen (2020) [7]	Song (2010) [40]	Winkelbach (2001, 2002) [38,39]	Davies (1998) [37]
			Position and Orientation	Position Based Methods			Position and Orientation	Orientation	Position and Orientation
			Resolution (mm)	Resolution (mm)	Resolution (mm)	Resolution (mm)	Resolution (mm)	Resolution (mm)	Resolution (mm)
8 × 8	250	103.8 × 166	11.7	26.7	25.9	41.4	25.7	87.4	57.6 (Area reduced to 63.0 × 166)
10 × 10			14.3
12 × 12			18.3
14 × 14			20.9
16 × 16			23.5
8 × 8	200	83 × 132.8	9.4	21.3	20.7	33.1	20.6	69.9	46.1 (Area reduced to 50.4 × 132.8)
10 × 10			11.5
12 × 12			14.6
14 × 14			16.7
16 × 16			18.8
8 × 8	150	62.3 × 99.6	7.0	16.1	15.6	24.9	15.4	52.4	34.5 (Area reduced to 37.8 × 99.6)
10 × 10			8.6
12 × 12			11.0
14 × 14			12.5
16 × 16			14.1
8 × 8	120	49.8 × 79.7	5.6	12.7	12.3	19.7	12.3	41.9	27.6 (Area reduced to 30.2 × 79.7)
10 × 10			6.9
12 × 12			8.8
14 × 14			10.0
16 × 16			11.3
8 × 8	110	45.7 × 73.0	5.2	11.8	11.3	18.2	11.3	38.5	25.3 (Area reduced to 27.7 × 73.0)
10 × 10			6.3
12 × 12			8
14 × 14			9.2
16 × 16			10.3
8 × 8	100	41.5 × 66.4	4.7	10.7	10.3	16.5	10.3	35.0	23.0 (Area reduced to 25.2 × 66.4)
10 × 10			5.7
12 × 12			7.3
14 × 14			8.3
16 × 16			9.4
8 × 8	80	33.2 × 53.1	3.8	8.5	8.2	13.2	8.2	28.0	18.4 (Area reduced to 20.1 × 53.1)
10 × 10			4.6
12 × 12			5.8
14 × 14			6.7
16 × 16			7.5
8 × 8	60	24.9 × 39.8	2.8	6.4	6.2	10.0	6.2	21.0	13.8 (Area reduced to 15.1 × 39.8)
10 × 10			3.4
12 × 12			4.4
14 × 14			5.0
16 × 16			5.6
8 × 8	40	16.6 × 26.6	1.9	4.2	4.1	7.5	4.1	14.0	9.2 (Area reduced to 10.1 × 26.6)
10 × 10			2.3
12 × 12			2.9
14 × 14			3.3
16 × 16			3.8

shows the comparison of the XY-plane of the projector’s resolution and the corresponding area covered for the depth ranges of 40 to 250 cm on a flat surface. In the experiment and measurement, objects or surfaces were 110 cm from the projector and the camera. The XY-plane projector resolutions were 5.2 mm for 8 × 8, 6.3 mm for 10 × 10, 8.0 mm for 12 × 12, 9.2 mm for 14 × 14, and 10.3 mm for 16 × 16. This work achieved the exact resolution of our previous work [14]. See Table 2. We have compared our technique with position-based methodologies and orientation-based methodologies. The position-based methods were Zhou (2023) [16], Yin (2021) [19], and Nguyen (2020) [7], whereas Song (2010) [40], Davies (1998) [37], and our proposed method determine both position and orientation, while Winkelbach (2001, 2002) [38,39] specifically used it only to determine the orientation. Despite achieving the exact resolution that we presented in our previous work [14], our method has surpassed other approaches in terms of resolution, and the data presented in Table 2 confirm this assertion. More precisely, we conducted measurements with a resolution of 5.2 mm using an 8 × 8 pattern at a depth of 110 cm. In contrast, the earlier position-based methods, such as that of Zhou [16], which used a large-sized M-array, achieved 11.8 mm. Yin [19], who employed speckle and fringe projection, achieved 11.3 mm. Nguyen [7], who used a fringe-based method, had a resolution of 18.2 mm. The earlier orientation-based methods, such as Song [40], which utilized a pseudo-random sequence of diamond-shaped colored elements, had a resolution of 11.3 mm. Meanwhile, Winkelbach [38,39] employed a stripe pattern that has a measured resolution of 38.5 mm. Davies [37], who used color circles as a projection, had a measurement resolution of 25.3 mm.

3.2. Classification or Decoding of Symbols or Feature Points in a Pattern

We applied the methods described in the preceding section and [14] to classify and decode the patterns used in the experiment. The application of our process to the original projection resulted in the successful decoding of 100% of the pattern symbols, thereby proving the reliability of the applied method. Once we had verified the algorithm’s accuracy on the original pattern, we used the technique for the pattern obtained from the planar surface, circular surface, and sculpture.

Table 3. Detected and decoded primitives and comparison.

Pattern Type & Depth			Surface Types
	Primitives	Original Pattern	Plane	Cylinder	Sculpture
Pattern 1 (8 × 8 resolution) Depth 110 cm	Detected	12,496	4205	2739	2127
	Decoded	12,496	4205	2724	2103
	%	100	100	99.5	98.9
Pattern 2 (10 × 10 resolution) Depth 110 cm	Detected	8352	2810	1967	1449
	Decoded	8352	2810	1954	1426
	%	100	100	99.3	98.4
Pattern 3 (12 × 12 resolution) Depth 110 cm	Detected	5187	1717	1159	901
	Decoded	5187	1717	1148	879
	%	100	100	99.0	97.6
Pattern 4 (14 × 14 resolution) Depth 110 cm	Detected	4000	1281	951	723
	Decoded	4000	1281	939	702
	%	100	100	98.7	97.1
Pattern 5 (16 × 16 resolution) Depth 110 cm	Detected	3124	1031	749	558
	Decoded	3124	1031	737	537
	%	100	100	98.4	96.2
Ahsan (2020) [14] (16 × 16 resolution) Depth 200 cm	Detected	3124	1650	1161	689
	Decoded	3124	1617	1128	585
	%	100	98.0	97.1	84.9

displays the number of symbols or feature points detected, decoded, and classified when the experiment utilized five different patterns on the measured surfaces. Our method successfully interpreted 100% of the symbols on a planar or flat surface for all five resolutions, which further confirms its reliability. In contrast, the previous work [14] achieved a classification of 98% of primitives. This indicates an improvement in the decoding process. Our classification method demonstrates better performance than other approaches. For instance, Petriu [11] was able to decode 59% of primitives, Albitar [12] was able to decode 95%, and Ahsan [14] was able to decode 97% of primitives on circular surfaces and 85% of primitives on a sculpture. We can accurately classify 98.4% of primitives for a 16 × 16 size, 98.7% of feature points on a 14 × 14 size, 99.0% for a 12 × 12 size, 99.3% for a 10 × 10 size, and 99.5% of primitives for an 8 × 8 size when dealing with circular surfaces. Moreover, we can classify 96.2% of primitives for a 16 × 16 size, 97.1% for a 14 × 14 size, 97.6% for a 12 × 12 size, 98.4% for a 10 × 10 size, and 98.9% for an 8 × 8 size when our method is applied to a sculpture. Therefore, smaller symbol sizes exhibit a noticeable and progressive improvement in classification. The classification of smaller symbols improved due to the reduced exposure to distortion caused by surface curvature. Since more shape descriptor parameters are utilized for the classification of symbols, there are no instances of wrong classification or false detection.

Figure 6 displays the parts of decoded patterns for the original projection and their results from captured images when applied to a flat planar surface. Figure 7 illustrates the decoded patterns of the acquired images from cylindrical and sculpture surfaces. To distinguish the symbols from each other, the centroid positions of each symbol type were highlighted using distinct colors. This was performed on all five patterns used in the experiment, i.e., the 8 × 8, 10 × 10, 12 × 12, 14 × 14, and 16 × 16 resolutions. The centroid positions of the square-shaped symbol are indicated with a black star (*). The centroid positions of the butterfly-shaped diagonally arranged squares are marked with a blue star (*). The centroid positions of a horizontal strip are reflected through a green star (*). Similarly, the centroid positions of the plus sign symbol are marked with a pink star (*).

3.3. Point Clouds and Surface Normals of the Measured Objects

After applying the method explained in the previous section, the surface normals and point clouds of the measured surfaces, i.e., cylinder and sculpture, were obtained for the five patterns used in the experiment. The measured resolution for the point clouds was 5.2 mm for an 8 × 8, 6.3 mm for a 10 × 10, 8.0 mm for a 12 × 12, 9.2 mm for a 14 × 14, and 10.3 mm for a 16 × 16 pattern. Figure 8 shows the point clouds obtained from the cylinder and sculpture for the five patterns employed in the experiment. Figure 9 shows the surface normals plot of the measured surfaces for the five projection patterns.

In our previous work [14], we implemented a 16 × 16 resolution, whereas in [42], we implemented two resolutions, 14 × 14 and 16 × 16. In this work, we implemented five resolutions, i.e., 8 × 8, 10 × 10, 12 × 12, 14 × 14, and 16 × 16. We obtained a denser point cloud and more surface normals with a smaller resolution of 8 × 8. The density of the point cloud decreased with an increase in the pattern size resolution of the point cloud from 8 × 8 to 16 × 16. All the points in the point clouds lay in between the depth or Z-axis of 1040 mm to 1120 mm.

The points in the point clouds were calculated on the basis of the corresponding matching points in the M-array. We may miss at least two rows or two columns near the boundaries of the measuring surface due to the window feature (3 × 3) of the M-array.

Table 4. Percentage of corresponding points in the M-arrays with respect to decoded primitives.

Pattern Type & Depth			Surface Types
	Primitives	Original Pattern	Plane	Cylinder	Sculpture
Pattern 1 (8 × 8 resolution) Depth 110 cm	Correspondence	12,040	3937	2517	1654
	Decoded	12,496	4205	2724	2103
	%	96.4	93.6	92.4	78.7
Pattern 2 (10 × 10 resolution) Depth 110 cm	Correspondence	7980	2596	1780	1076
	Decoded	8352	2810	1954	1426
	%	95.6	92.4	91.1	75.5
Pattern 3 (12 × 12 resolution) Depth 110 cm	Correspondence	4895	1550	1014	662
	Decoded	5187	1717	1148	879
	%	94.4	90.3	88.3	75.3
Pattern 4 (14 × 14 resolution) Depth 110 cm	Correspondence	3744	1139	819	521
	Decoded	4000	1281	939	702
	%	93.6	88.9	87.2	74.2
Pattern 5 (16 × 16 resolution) Depth 110 cm	Correspondence	2898	903	631	400
	Decoded	3124	1031	737	537
	%	92.8	87.6	85.6	74.5
Ahsan (2020) [14] (16 × 16 resolution) Depth 200 cm	Correspondence	2898	1329	859	397
	Decoded	3124	1617	1128	585
	%	92.8	82.2	76.1	67.9

shows the number of points for which the correspondence has been established with respect to the total number of points decoded for each pattern and surface.

The corresponding points were compared with our previous work [14], which shows an improvement in the process of attaining the corresponding points. In the process of matching corresponding points for the original patterns, the result remained the same for the exact resolution, i.e., 16 × 16. As the symbol sizes or pattern resolution reduced, the percentage of the corresponding points increased and reached up to 96.4% for 8 × 8, compared to 92.8% for 16 × 16. A smaller symbol size or resolution resulted in a greater number of corresponding points, indicating that a smaller symbol size increases the possibility of matching points in close proximity to the boundaries. This phenomenon was witnessed in all three types of surfaces: plane or flat, cylinder, and sculpture. We have made significant progress in the process of matching points compared to our previous work [14]. In the previous work on a plane surface, we achieved 82.2% of matching points. However, with the exact resolution of 16 × 16, we have now achieved 87.6% of the corresponding points. The matching point increases as the resolution decreases and rises to 93.6% for an 8 × 8 resolution. When considering a cylindrical surface, the matching points increased from the previous 76.1% to 85.5% for a 16 × 16 resolution and rose to 92.4% for an 8 × 8 resolution. For a sculpture, the matching points increased from the previous 67.9% to 74.5% for a 16 × 16 resolution and reached up to 78.7% for an 8 × 8 resolution.

3.4. Time Durations of the Different Processes

Table 5. The time calculation for different processes.

Surface Type	Method	Resolution	Preprocessing (Filtering + Thresholding)	Labeling	Parameter Calculation	Classification	Correspondence	Rate of Correspondence	Computation of Surface Normals	Computation of 3D Point Cloud
Original Pattern	Ahsan [14] (2020) depth: 200 cm	16 × 16	566	42	587	3.3	485	0.19	-	-
	Proposed Method depth: 110 cm	16 × 16	307.6	18.9	1105.4	1.2	836.9	0.27	-	-
		14 × 14	322.3	21.9	1366.4	1.4	1278.1	0.32	-	-
		12 × 12	342.4	26.3	1644.6	1.6	1969.6	0.38	-	-
		10 × 10	352.6	34.4	2651.8	2.6	4604.0	0.55	-	-
		8 × 8	363.7	59.2	4011.4	4.4	8437.8	0.68	-	-
Plane Surface	Ahsan (2020) [14] depth: 200 cm	16 × 16	611	53	365.6	2.2	480	0.3	-	24.7
	Proposed Method depth: 110 cm	16 × 16	380.9	17.7	472.6	0.52	265.3	0.26	18.5	22.4
		14 × 14	390.4	24.0	594.1	0.63	388.9	0.30	19.3	25.6
		12 × 12	401.3	25.3	739.9	0.75	587.0	0.34	28.5	38.2
		10 × 10	412.3	33.2	1048.1	0.92	1403.1	0.50	69.3	89.6
		8 × 8	422.6	36.5	1492.5	1.4	2301.1	0.55	174.1	207.7
Cylinder	Ahsan (2020) [14] depth: 200 cm	16 × 16	649	41.5	361	2.7	331.1	0.29	-	24.3
	Proposed Method depth: 110 cm	16 × 16	360.8	17.8	326.1	0.25	178.7	0.25	10.6	14.4
		14 × 14	371.0	19.1	415.8	0.31	292.5	0.31	15.0	21.2
		12 × 12	379.6	20.8	463.4	0.38	372.5	0.33	16.4	21.8
		10 × 10	386.7	23.2	746.8	0.62	904.9	0.46	29.3	39.4
		8 × 8	397.3	26.9	977.6	0.92	1505.2	0.55	68.9	89.4
Sculpture	Ahsan (2020) [14] depth: 200 cm	16 × 16	644	38	271	2.7	318	0.5	-	15.4
	Proposed Method depth: 110 cm	16 × 16	372.0	12.2	260.1	0.18	196.2	0.37	7.0	10.4
		14 × 14	384.5	16.2	307.2	0.25	321.1	0.46	7.4	11.4
		12 × 12	393.6	17.5	326.7	0.29	474.1	0.54	11.4	16.2
		10 × 10	401.0	18.6	524.1	0.46	934.5	0.66	19.3	26.2
		8 × 8	411.9	20.4	764.6	0.68	2097.8	1.00	29.4	42.1

displays the time duration of the different processes in the decoding, computation of the surface normals, and computation of the 3D point clouds and compares these values with those in [14]. Each process has a specific time duration. The time durations were measured on an average core i5 computer. The average time for each process was calculated and repeated many times for optimization to obtain the optimum values. The time measurements are specified in milliseconds. All the processes, such as filtering and thresholding, labeling, parameter calculation, classification, correspondence matching, computation of surface normals, and computation of point clouds, indicate an increase in the time duration with the complexity and texture of the surface and the number of detected primitives. With the decrease in resolution for each type of surface, the number of primitives increases, which increases the time durations. Compared to [14], the preprocessing, labeling, classification, correspondence matching, and computation of point cloud time decreased. This may be due to the better specification of the computing machine compared to that used in [14]. However, the time for parameter calculation has increased since four symbols were employed in the pattern compared to the three in [14], and more shape descriptor parameters are required to be calculated for symbol classification. The computation time of surface normals is a few milliseconds, which signifies that the method is fast and can be applied to real-time applications. For instance, for sculptures using an 8 × 8 resolution, 1654 surface normals were calculated in 29.4 milliseconds. Similarly, for cylindrical surfaces using an 8 × 8 resolution, 2517 surface normals were computed in 68.9 milliseconds.

4. Conclusions

We have described a method to find the surface orientations through a single-shot, spatially encoded monochromatic symbol-based pixel-level design. Our technique is more robust against image noise and surface color variations due to the monochromatic design. We presented multi-resolution patterns to determine the dense reconstruction for calculating the surface orientation. We have calculated orientations through surface normals and position profiles for different resolutions and compared these results quantitatively and qualitatively with previous methods. Our method differs from shape to shading and shape to texture in that it does not necessitate an assumption of the relationship between local surface orientations at neighborhood points. Therefore, we introduced a position-agnostic approach to identify the orientation independently. We applied the technique on real objects such as a plane surface, circular surface, and complex sculpture. We obtained a higher resolution compared to previous methods and computed the surface normals in a matter of milliseconds. We improved the correspondence matching and estimation of feature points in a pattern. A higher percentage of feature points has been decoded, resulting in an increased number of matching points in the correspondence. Additionally, the duration of the processing has been decreased. By employing a monochromatic pattern and a straightforward decoding approach, we can obtain a greater number of decoded primitives and points that match correspondences.