The large baseline stereovision system is composed of two identical observation stations, each station having the following components (see Figure 1): -

“Wide-angle” objective with a focal distance of 20 mm, sigma EX 20 mm F1.8 DG RF aspherical type [9].

Each camera captures images of the night sky and sends captured images to a local computer. The stereoscope's base-line (the distance between the camera systems) is 37 km, a compromise between simultaneous detection of low altitude objects from two locations and triangulation accuracy.

The theoretical visual FOV covered by this system configuration has a diagonal of 68 degrees. We have tested different exposure times for maximum sensibility and maximum aperture of the objective (ISO 3200, F1.8), and we found that the optimum exposure time was 5 s. The acquisition cycle is comprised of: the exposure time, the analog/digital conversion time and the transfer/storage time (computer and/or camera memory card). The analog/digital converter afferent to the DSLR camera works on 14 bits. To satisfy the resolution and speed requirements previously mentioned we have worked in binning mode—“sRAW2” (2,376 × 1,584 pixels) or “small JPEG” (2,352 × 1,568 pixels). The acquisition cycle for small resolution mode took a minimum of 7 seconds (5 seconds exposure time), and in the case of direct transfer to the computer it reached 8–9 s. Consequently, this method allowed us to obtain an average of 10 images for a complete passing of a LEO satellite.

For the small JPEG image size, the intrinsic calibration process has computed a focal distance, measured in pixels, of 2,187. For this focal distance, and the baseline of 37 km, the expected stereo measurement accuracy can be computed if we assume the equivalent rectified (canonical) stereoscope having the same parameters. The actual stereo performance, for an epipolar geometry system such as the one described in this paper, depends on multiple factors, such as the orientation of the observed object with respect to the epipolar line, but the canonical equivalent error estimation is still a good predictor. For the canonical case, the distance Z, the baseline b, the focal distance f and the disparity d are related as Z = bf/d. For our baseline and our focal length, if we assume plus or minus one pixel of error in disparity estimation, we get error values ranging from 4 km at 400 km of distance (1% error), to 55 km at 1,500 km (3.7%), that is, from less than 1% to 4% for the whole LEO range. The error decreases linearly with the baseline. A double baseline would mean half the error, but it can also mean that we get less overlapping field of view between the cameras. A double focal length would mean a similar halving of the error, but with the cost of decreasing the field of view.

Stereovision requires synchronized image acquisition from the two cameras. In a classical stereo system the synchronization is achieved by triggering the cameras by a common signal. However, this solution cannot be applied when the cameras are 37 km apart. Thus, we need two independent units that can generate triggering signals at the same time, without a physical connection between them (see Figure 2). Each unit has a schedule list of triggering times, and if the units have a common schedule, the triggering signals will be simultaneous. Unfortunately, working with a schedule means assuming that the two synchronization units have a precise internal time, which is an assumption that for a common laptop PC is not true. Even if the internal clocks are precisely synchronized, they will quickly fall out of sync.

In the absence of a precise time base, we needed some external signal to keep the two units in sync. This signal is the GPS radio signal, which is received simultaneously, once a second, all over the world. Each unit receives this signal with the help of a GPS receiver, connected to the PC. The following algorithm shows the operation of a unit, based on GPS reception.

The unit reads the current triggering time from the ScheduleList. The current global time is polled through the method GPSReceiver.GetTimeNonBlocking(), which will deliver the most recent timestamp of a GPS reception, without blocking the execution of the program. If the global time is close to the triggering time, the program goes in the blocking mode. The GPS receiver is queried by GPSReceiver.GetTimeBlocking(), a function that will exit only when a fresh signal is received. In this way, there will be a minimum delay between the reception of the signal and the triggering of the camera, if the received time is the one matching the schedule list.

In order to represent the position of the detected objects, we have chosen a coordinate system whose origin is located in the centre of the Earth, and its coordinate axes are fixed in relation to our planet. Such a coordinate system is the Earth Centered, Earth Fixed (ECEF) one, whose OZ axis is the Earth's axis of rotation, pointing to the North Pole, the OX axis joins the centre of the Earth with the point on the surface that has zero latitude and longitude, and OY is perpendicular to the XOZ plane. The equatorial plane is identical to plane XOY, and the zero meridian plane is XOZ (see Figure 3).

In stereovision terms, we shall refer to the ECEF coordinate system as World Coordinate System, W_C. The two cameras that will observe the sky will have their own coordinate systems, the left camera system centered in C_L and having the axes X_LY_LZ_L, and the right camera system centered in C_R, and having the axes X_RY_RZ_R. The left image plane will be parallel to the plane X_LC_LY_L, and the right image plane will be parallel to X_RC_RY_R.

The stereovision process relies on accurate triangulation of corresponding features from the left and right camera images. For this reason, the correspondence between the pixel position and the 3D world must be accurately described by the cameras' parameters. There are two types of parameters, intrinsic and extrinsic.

The extrinsic parameters describe the relation between the coordinate systems of the cameras and the world coordinate system. For each camera, we have a translation vector, which describes the position of the camera's optical center in the world coordinate system. We shall denote these vectors by T_CL and T_CR. Conversely, one can express the position of the world coordinate system's origin in the cameras' coordinate systems. These translation vectors are denoted T_WL for the left camera, and T_WR for the right camera.

The orientations of the camera coordinate systems with respect to the world coordinate system are described by the rotation matrices of the two cameras, R_CL and R_CR.

Using the translation vectors and the rotation matrices, one can perform coordinate transformations between the world and camera coordinate systems, for any 3D point.

The specific characteristics of a camera plus lens optical system are described by the intrinsic camera parameters (in what follows, we'll denote by the word “camera” the whole camera-lens assembly). For each camera, we'll have the following intrinsic parameters: -

Focal length, measured in pixels, which is the distance from the optical centre to the image plane. Each camera has its own focal length: f_L for the left camera, and f_R for the right camera.

Another type of distortion appears when the curvature centers of the lenses that are assembled in the optical system are not collinear. This distortion has, besides the radial component, a tangential one [13]. The tangential distortion is modeled by the following equation: (3) [ ∂ x t ∂ x t ] = [ 2 p 1 ⋅ x y + p 2 ( r 2 + 2 x 2 ) p 1 ( r 2 + 2 y 2 ) + 2 p 2 ⋅ x y ]where p₁, p₂ are the tangential distortion parameters.

In order to apply the stereovision process, we have to know all the parameters described in this section, for both cameras. The process of computing these parameters is the process of camera calibration, which has two major steps: intrinsic calibration and extrinsic calibration.

The intrinsic calibration is performed once, before the cameras are set up in their working positions. The calibration process is a time consuming one, requiring that multiple images containing a chess-like pattern be acquired. The pattern must cover all image regions, and must be photographed from multiple angles and multiple distances from the camera.

The intrinsic calibration is performed using the Bouguet method, available through the Caltech Camera Calibration Toolbox [14]. The method is semi-automatic, relying on the user for selecting the rectangle containing the relevant corners in the image, the individual corners of the chess squares being detected automatically afterwards. The intrinsic calibration process provides the position of the principal point, the focal length, and the distortion coefficients for each camera.

Once the intrinsic calibration process is completed, we know the distortion coefficients that affect the way a scene is represented in the image space. These coefficients cannot be ignored, because a wide angle lens, such as the one used for our night sky observation system, presents considerable distortion towards the image's margins. For this reason, we have to apply a distortion removal algorithm for each image we acquire.

The distortion removal process is applied before extrinsic calibration, and before the process of satellite detection can begin. This way, we can assume that in the next processing steps the images are acquired by an ideal perspective camera (pinhole camera).

The distortion removal algorithm is the following:

For each pixel (x,y) from the corrected image D (destination): -

Compute the distorted coordinates (x′, y′) using Equations (1) and (3).

In the next sections, we'll assume that all the images are corrected for distortions.

The classical extrinsic calibration methodology relies on the following assumptions: -

The intrinsic parameters are known.

The calibration process relies on the projection equations, which map a 3D point in the World Coordinate System with its 2D correspondent in the image space. The unknowns of this relation are the extrinsic parameters of the camera: the translation vector and the rotation matrix. Having enough points, an iterative optimization process can extract the unknown values. The precision of the calibration process depends on the number of calibration points, on the accuracy of the measurement of their 3D coordinates with respect to the World Coordinate System, and on the precision of localization of their image correspondents.

The extrinsic calibration must be performed with the cameras in their final position for stereovision-based measurements, and the size of the calibration scene has to be comparable with the distances that we want to measure. In our case, the cameras will be positioned roughly 37 km apart, pointed towards the sky, while the distances we want to measure are in the range of hundreds of kilometers. In these conditions, finding a good calibration scene for a classical calibration approach is very difficult.

To further complicate the matter, the cameras are fixed on equatorial mounts with star tracking capabilities. This mount simplifies the detection process, by making the stars appear static in the image sequence, but in the same time it complicates the calibration process, as it constantly changes the orientation of the cameras with respect to the World Coordinate System.

In this situation, the only calibration objects that we can rely on are the stars, whose position in the World Coordinate System is predictable. Unfortunately, the stars cannot be considered to be objects of known 3D coordinates X, Y and Z, as they are so far away that for all practical purposes their distance from the World Coordinate System's origin can be considered infinite. The only reliable information about the stars is composed of their angular coordinates, and thus they can be used only for computing the camera orientation angles, not for computing the translation vectors.

However, the translation vectors can be determined from the GPS coordinates of the camera's locations. Even if the commercial GPS devices have a low precision, with errors in the range of meters, this error is not critical for our situation, as the distance between cameras (the baseline) is several orders of magnitude greater.

The GPS coordinates of the two observation locations are the following: Feleacu Latitude: 46°42′36.50″N Longitude: 23°35′36.74″E Elevation: 743 m Marisel Latitude: 46°40′34.362″N Longitude: 23°07′8.904″E Elevation: 1,130 m

In order to extract the translation vectors (in the ECEF coordinate system) from the GPS parameters, we'll use the World Geodetic System standard, in its latest revision, WGS 84 [15]. This standard defines the Earth's surface as an ellipsoid with the major radius (equatorial radius) a = 6,378,137 m and a flattening factor f = 1/298.257223563. The minor (polar) radius can thus be computed as b = a(1 − f), being approximately 6,356,752.3142 m.

Knowing the latitude φ, the longitude λ and the elevation h, we can extract the translation vector's components X, Y and Z using the following equations (the latitude and longitude angles are converted to radians): (4) c = 1 cos 2 φ + ( 1 − f ) 2 sin 2 φ (5) s = ( 1 − f ) 2 c (6) r = ( a c + h ) cos φ (7) X = r cos λ (8) Y = r sin λ (9) Z = ( as + h ) sin φ

At this point, we know the translation vectors T_CL and T_CR, and the intrinsic parameters of the cameras. All these parameters are fixed, as the camera properties do not change, and the positions of observation remain fixed. The parameters that are still unknown are the rotation matrices R_CL and R_CR, and these parameters change for each pair of acquired images, as the star tracking system continuously re-orients the cameras. Fortunately, for rotation matrix computation we can use the stars as calibration targets. For a precise calibration, we have chosen 18 stars of high brightness, easily identifiable in the images, which cover as much image surface as possible (see Figure 4).

Table 1 shows the equatorial coordinates [16], for the stars used in calibration process (shown in Figure 4). The Right Ascension (RA) and Declination (DEC) are updated (precession, nutation and proper motion) for the epoch of the observations, 9 July 2011 [17].

As our World Coordinate System is fixed with respect to the Earth, we cannot use, in the process of calibration, the Right Ascension and the Declination directly. We need stellar coordinates that depict the instantaneous positions of the stars, in a specific time instant, with respect to the Earth. The declination angle is related to the equatorial plane, and therefore it remains the same as the Earth rotates, but the Right Ascension is relative to a fixed point in space, and this point changes its position with respect to the World Coordinate System constantly.

The equivalent of longitude for the celestial sphere is the Hour Angle. This angle is formed by the local meridian plane of a place on Earth with the plane passing through the Earth's axis and the sky object to be observed. The Hour Angle is positive towards the West, and is traditionally expressed in time units. The Hour Angle of a star is computed as the difference between the Local Sidereal Time (LST) of the place and the Right Ascension (RA) of the star. The local sidereal time can be computed from the local time [16].

As the World Coordinate System relates to the Equator and to the Zero Meridian, we find convenient to relate the Hour Angle depicting the instantaneous position of a star to the same Zero Meridian, instead of relating it to the meridian of the place of observation (see Figure 5). In order to find the Hour Angle relative to meridian zero, HA₀, the following steps must be executed: -

Conversion of the local time of frame acquisition to GMT

For a distance R, the polar coordinates can be converted into Cartesian coordinates. The X, Y and Z coordinates of the calibration stars will be computed using the following equations: (11) X = R cos ( DEC ) cos ( H A 0 ) (12) Y = R cos ( DEC ) sin ( H A 0 ) (13) Z = R sin ( DEC )

At this point we have the 3D position of the stars in the World Coordinate System. In order to apply the rotation matrix calibration, we need to know the position of these stars in the acquired images (after the distortion has been removed from these images).

Initially, the position of the reference stars is set manually, for a reference frame for the left (Feleacu) and for the right (Marisel) observation sources. Ideally, the star tracking system will compensate the Earth's motion and the position of the calibration stars should remain constant for all the image sequence. Unfortunately we have found that small errors may appear and sometimes significant changes are visible, as shown in Figure 6.

As the calibration is very sensitive to the input data, it is crucial that the position of the reference stars is corrected for each frame. The correction process is automatic, using a stereovision-specific correlation technique for finding the correspondent position of the stars across the frames.

For each newly acquired frame, we compare image windows in a neighborhood of the reference star position to an original image window of the reference frame, centered in the reference point. A distance between the original window and each of the candidate windows is computed, and the centre of the minimum distance candidate is the new calibration star position. Several window comparison metrics are presented in [18]. The simplest and fastest one is the Sum of Absolute Differences (SAD), applied to a squared neighborhood (5 × 5, 7 × 7, …). For this function, the ideal matching distance is zero, when the corresponding regions are identical. In practice, we'll settle for the minimum in a given search range. The SAD matching function is described by the following equation: (14) SAD ( x N , y N ) = ∑ i = w 2 w 2 ∑ j = w 2 w 2 | I R ( x R + i , y R + j ) − I N ( x N + i , y N + j ) |where (x_R, y_R) is a point in the reference frame, I_R is the reference frame as a matrix of intensity values, (x_R, y_R) is a point in the newly acquired image, I_N is the intensity matrix of the new image, and w is the size of the correlation window. For our calibration methodology we have chosen w = 11, as the processing time is not critical while the correctness of the correspondence is. The new position is searched in a neighborhood of 41 × 41 pixels around the reference position (see Figure 7).

The automatic adjustment of the calibration points has a success rate of 100% for all frames that we have processed, which means that the calibration of the rotation matrices can be applied automatically, without any supervision, for the whole observation sequence.

The calibration process starts from the equations that project the known 3D reference points to known reference image points. For each point (star) i we have a system of two equations: (15) { x i − x c f [ r 13 ( X i − X c w ) + r 23 ( Y i − Y c w ) + r 33 ( Z i − Z c w ) ] + [ r 11 ( X i − X c w ) + r 21 ( Y i − Y c w ) + r 31 ( Z i − Z c w ) ] = 0 y i − y c f [ r 13 ( X i − X c w ) + r 23 ( Y i − Y c w ) + r 33 ( Z i − Z c w ) ] + [ r 12 ( X i − X c w ) + r 22 ( Y i − Y c w ) + r 32 ( Z i − Z c w ) ] = 0where X_i, Y_i, Z_i are the reference point's 3D coordinates, x_i and y_i are the reference point's image coordinates, x_c and y_c are the coordinates of the principal point of the camera, X_CW, Y_CW, Z_CW are the components of the translation vector, and f is the camera's focal distance. The unknowns are the rotation matrix R's components, R = (r_kl), for k, l = 1,2,3.

Due to the fact that the X_i, Y_i and Z_i coordinates of the stars are proportional to their distance from Earth (Equations (11–13)), which is many orders of magnitude higher than the values of the translation vector of the cameras, the coordinates of the cameras in the World Coordinate System X_CW, Y_CW and Z_CW can be removed from Equation (15). Doing this, and using Equations (11–13), we can rewrite Equation (15) such that no assumption about the distance of the stars is needed: (16) { x i − x c f [ r 13 cos DEC i cos H 0 , i + r 23 cos DEC i sin HA 0 , i + r 33 sin DEC i ] + [ r 11 cos DEC i cos HA A 0 , i + r 21 cos DEC i sin HA 0 , i + r 31 sin DEC i ] = 0 y i − y c f [ r 13 cos DEC i cos HA 0 , i + r 23 cos DEC i sin HA 0 , i + r 33 sin DEC i ] + [ r 12 cos DEC i cos HA 0 , i + r 22 cos DEC i sin HA 0 , i + r 32 sin DEC i ] = 0

For each star we have two equations, with nine unknowns. The unknowns have only three degrees of freedom, represented by the rotation angles around the coordinate axes, and therefore we can add constraints to model the dependencies between the unknowns. Such constraints can be derived from the fact that the rotation matrix is orthogonal: (17) R ⋅ R T = I

From this constraint we derive six equations: (18) { r 11 ⋅ r 11 + r 12 ⋅ r 12 + r 13 ⋅ r 13 − 1 = 0 r 21 ⋅ r 21 + r 22 ⋅ r 22 + r 23 ⋅ r 23 − 1 = 0 r 31 ⋅ r 31 + r 32 ⋅ r 32 + r 33 ⋅ r 33 − 1 = 0 r 11 ⋅ r 21 + r 12 ⋅ r 22 + r 13 ⋅ r 23 = 0 r 11 ⋅ r 31 + r 12 ⋅ r 32 + r 13 ⋅ r 33 = 0 r 21 ⋅ r 31 + r 22 ⋅ r 32 + r 23 ⋅ r 23 = 0

In order to have a number of equations greater or equal to the number of unknowns, we need at least two reference points. For better results, multiple reference points can be used, and for this reason we rely on the known positions of 18 stars. For a set of n ≥ 2 points, we have a linear system of 2n + 6 equations, the first 2n being obtained from system (16) for each point, and the last six equations being the system (18). In order to solve this system, we use the Gauss-Newton iterative method. Starting from an initial guess, which for a rotation matrix can be the identity matrix, gradual corrections are applied until the estimation converges to the right solution [19].

The rotation matrix calibration process is applied independently for both cameras. At the end of this process, the stereoscope is ready to identify and measure the parameters of the moving objects from the image pair.

Stereovision-based 3D reconstruction relies on matching features from the left and right images, followed by triangulation. If the sky objects were point-like in the image space, the correspondence search process would be straightforward. However, we have already seen that this is not the case, and the image signature of these objects is a line segment, having a length proportional to the speed of the object, and to the exposure time of the camera.

If the conditions were ideal, the moving object would become visible at the same time in both images of the stereo pair, and it will also disappear at the same time. If this were the case, the easiest way to find the corresponding points is to look at the ends of the line segments. Unfortunately, there are several causes that make this approach a bad idea. First, the two observation sites have different background illumination conditions, a condition that affects the thresholding of the object's features and may cause different length of segments in the two images. Second, the line segment's length may be influenced by the Gaussian noise of the image, or by the background stars that happen to be near the object, and these conditions may be slightly different for the two cameras. Third, due to the low cost equipment used for camera triggering (off the shelf GPS receivers connected to PC's) there may be some errors in the camera synchronization process, errors that may influence the start of the exposure period (which influences the position of the first end of the linear segment) or the duration of the exposure (which influences the segment length). All these conditions make a compelling argument against using the ends or the middle of the segment for matching. Fortunately, there is a more reliable property of the moving object related line segment, which will accurately allow us to perform accurate, sub-pixel based matching between the images: the trajectory line itself.

While the trajectory line is unbounded, and therefore the correspondence points are difficult to find, there is another restriction that drastically limits the search space: the epipolar constraints. The epipolar constraint says that for each point P_L of the left image, the possible correspondences in the right image are located on a line called the epipolar line. This constraint is shown in Figure 9: the optical centers of the two cameras form, together with the unknown 3D point X that needs to be measured, a plane that intersects the two image planes forming the epipolar lines. One can see that the plane is completely determined by one projection ray (caused by one point in one of the images) and the line joining the cameras optical centers. This means that knowing a point in the left image and the camera parameters (intrinsic and extrinsic), we can compute the epipolar line that will contain the corresponding right point P_R.

According to [23], the coefficients of the epipolar line ax + by + c = 0 from the right image can be computed using the coordinates of the left image point P_L = [x_L y_L 1] and the 3x3 fundamental matrix F: (26) [ a b c ] T = FP L

The fundamental matrix is computed from the intrinsic and extrinsic parameters of the two cameras [23]: (27) F = M R − T EM L − 1where: (28) M L = [ − f L 0 x CL 0 − f L y CL 0 0 1 ] M R = [ − f R 0 x CR 0 − f y CR 0 0 1 ]are the intrinsic matrices of the two cameras, and E is the essential matrix, which can be computed as: (29) E = RS

R is the relative rotation matrix between the cameras: (30) R = R CR T R CL

S is determined by the relative translation vector T: (31) S = [ 0 − T ( 3 ) T ( 2 ) T ( 3 ) 0 − T ( 1 ) − T ( 2 ) T ( 1 ) 0 ]

The relative translation vector is the difference between the absolute translation vectors of the cameras in the world coordinate system: (32) T = T CR − T CL

As the epipolar lines are easily computed, we can design a simple algorithm for extracting the correspondences between the objects of the left and of the right image. The algorithm will be based on the following steps:

Extraction of the center of mass from the objects in the left image. The center of mass will be P_L.

The corresponding points in the left and right images are now available, and thus the triangulation process can be applied. This triangulation will transform the image point pair (P_L, P_R) into a single 3D point P_W. This 3D point lies at the intersection of the projection rays passing through P_L and P_R (the lines passing through the image points and the optical centers).

Because the camera parameters are known, we can write the equation of the line passing through the 3D point P_W and the optical center of the left camera C_L [24]. The perspective projection leads to the following relation: (33) [ X W X W X W ] = μ R L [ x L − x CL y L − y CL − f L ] + [ X CL Y CL Z CL ]where μ is a scaling factor depending on the distance Z_W. Multiplying with R L T we obtain: (34) [ x L − x CL y L − y CL − f L ] = μ − 1 R L T [ X W − X CL Y W − Y CL Z W − Z CL ]which can be detailed as: (35) [ x L − x CL y L − y CL − f L ] = μ − 1 [ r 11 r 21 r 31 r 12 r 22 r 32 r 13 r 23 r 33 ] [ X W − X CL Y W − Y CL Z W − Z CL ]

The above equation is a system of three equations with four unknowns. The third equation can be used to write μ⁻¹ as a function of the other unknowns: (36) μ − 1 = − f L r 13 ( X W − X CL ) + r 23 ( Y W − Y CL ) + r 33 ( Z W − Z CL )

Replacing μ⁻¹ in the first two equations, one obtains the following system: (37) { x L − x CL = − f L r 11 ( X W − X CL ) + r 21 ( Y W − Y CL ) + r 31 ( Z W − Z CL ) r 13 ( X W − X CL ) + r 23 ( Y W − Y CL ) + r 33 ( Z W − Z CL ) y L − y CL = − f L r 12 ( X W − X CL ) + r 22 ( Y W − Y CL ) + r 32 ( Z W − Z CL ) r 13 ( X W − X CL ) + r 23 ( Y W − Y CL ) + r 33 ( Z W − Z CL )

This system has two equations and three unknowns. Writing the same equations for the right camera system, we have a total of four equations with three unknowns, a supra-determined system that can be solved by a least squares approach. From a geometrical point of view, the least squares solution will find the 3D point that has a minimum distance from the two projection lines, even if small calibration or matching errors will prevent these lines from intersecting [23].