Calibration of Action Cameras for Photogrammetric Purposes

The use of action cameras for photogrammetry purposes is not widespread due to the fact that until recently the images provided by the sensors, using either still or video capture mode, were not big enough to perform and provide the appropriate analysis with the necessary photogrammetric accuracy. However, several manufacturers have recently produced and released new lightweight devices which are: (a) easy to handle, (b) capable of performing under extreme conditions and more importantly (c) able to provide both still images and video sequences of high resolution. In order to be able to use the sensor of action cameras we must apply a careful and reliable self-calibration prior to the use of any photogrammetric procedure, a relatively difficult scenario because of the short focal length of the camera and its wide angle lens that is used to obtain the maximum possible resolution of images. Special software, using functions of the OpenCV library, has been created to perform both the calibration and the production of undistorted scenes for each one of the still and video image capturing mode of a novel action camera, the GoPro Hero 3 camera that can provide still images up to 12 Mp and video up 8 Mp resolution.


Introduction
During the last few years, passive sensors providing visual image information [1], as well as the development of software solutions for the extraction of point clouds from a set of un-oriented images have received increasing attention, not only from the surveying community, but also from non-experts (such as archeologists). The reason is due to some clear advantages provided from these systems: the improved resolution and sensitivity of photo sensors and the availability of low cost off-the-shelf digital cameras which can be used in several photogrammetric applications, such as aerial mapping (e.g., aimed to structural and archaeological surveying) or emergency documentation.
Normally in photogrammetric applications, amateur cameras, semi-professional and professional cameras (dSLR) are not appropriate for measurement purposes. Making it necessary to calibrate them by estimating their intrinsic parameters in order to be able to extract accurate 3D metric information from their images [2]. The estimation of these geometric characteristics, i.e., the focal length (f) of the lens, the coordinates of the centre of projection of the image (x p , y p ), the radial lens distortion coefficients (k 1 , k 2 , k 3 ) [2], is performed through the camera calibration process.
A full review of camera calibration methods and models developed in the last few years can be found in [3][4][5] describing the commonly adopted methods such as Tsai [6] or Zhang [7]. These are all based on the pinhole camera model including terms for modeling radial distortion. Nowadays there are several software applications (i.e., Photomodeler, Agisoft Lens, iWitness, MicMac, 3DF Zephir, etc.), mainly produced by the computer vision scientific community, that can automatically perform camera self-calibration. They also offer the possibility to work with several cameras and sensors to obtain dense point clouds or 3D models suitable for different fields of applicFation.
In the wide panorama of low-priced consumer grade devices [8] (including nowadays even smartphones or other similar mobile devices) [9,10], action cameras are more widespread and have been used more and more during extreme activities such as free fall, parapent flying, underwater swimming and diving. Their photogrammetric use though has not been easy since they could not supply high resolution still images and video and additionally their geometry are far away from the theoretical model of central projection due to their wide angle or fish eye lenses.
Several approaches have recently been testing the use of the GoPro Hero 3 camera (GoPro, Inc., San Mateo, CA, USA) [11][12][13], particularly for the recording and/or photogrammetric use in underwater conditions even using a stereoscopic configuration to obtain better results or to produce models from immersive videos. However, this was not possible since the baseline of the cameras was very short [14][15][16].
The camera's characteristics are ideal not only for underwater conditions, but also in UAV configurations. The camera is very light and additionally it can be controlled through a WiFi connection from a cell phone or tablet [17].

Camera Specifications
The weight (Table 1) and dimensions (Table 2) of the camera are ideal for photogrammetric applications, especially when it is necessary to place it on an UAV or on top of a diver's mask. The GoPro Hero 3 camera has a rather big sensor which is able to provide four different still image resolutions (Table 3) and nine different video resolutions (Table 4).  The modes of the camera used to capture still images or video sequences vary depending on the needs of the application and the user demands. The frame rate of the coarser video resolution (WVGA 848 × 480) is about 240 FPS, while the finer resolution (4K Cin 4096 × 2160) can provide a minimum of 12 FPS (Table 4).
Additionally, the image dimensions ratio also varies and can be 4:3, 16:9 or 17:9. Therefore, the sensor part used to acquire the still images or the video sequences is not always the same. For this reason separate self-calibration procedures have to be applied in order to calculate the camera calibration parameters. Using our software, we have calibrated the GoPro camera in several of its resolutions, however for photogrammetric purposes the most useful is the largest one, hence for the accuracy assessment of our calibration procedure (see Section 3) we have used only 12 Mpixels GoPro images.

Calibration Model and Undistorted Images
A modified mathematical of the Brown Calibration model [18] has been used to determine all the appropriate parameters that determine the lens deformation appearing on an image [19]. The specific model uses odd and even order polynomial coefficients to model the radial and the tangential distortion of the lenses (Equations (1-4)): The developed software, containing all the procedures needed to calibrate not just the GoPro camera, but any imaging sensor that is able to create digital photographic images, is compiled in Microsoft Visual Studio 2010, using the functions of the graphics library OpenCV (version 2.4.3). Within the OpenCV library there are also functions that implement the modified Brown Calibration Model. With the use of certain image analysis and adjustment functions of OpenCV it was feasible to create a software application running in Windows OS that is able to:  Identify a big number of ground control points (targets) on images printed on a special calibration sheet (chessboard or circle patterns);  Calculate through adjustment all the camera calibration parameters; except for the radial and tangential calibration parameters the camera constant (f) value and the principal point's coordinates (x o , y o ) are determined;  And (optionally) create the undistorted (idealized) images ( Figure 1).
The estimated accuracy of the calibration is given by the OpenCV routine that simultaneously calculates the extrinsic and intrinsic parameters of all the images used in the calibration process. The value is given in pixels and is referred to as Average Reprojection Error. It concerns the difference of target points location defined in the original images from their estimated location using the calculated exterior, interior and calibration parameters.

Recognized Problems and Finding Solutions
Although there exists an OpenCV sample software application (running under the Windows command line environment) able to deliver camera calibration, it cannot be used to calibrate high resolution digital cameras (more than 20 Mpixels). The specific software was originally created for the calibration of small frame cameras and especially for web cameras in Computer Vision Applications.
Certain problems appeared when we tried to identify control points of the printed calibration sheet on high resolution images. Especially, when we tried to calibrate a full frame camera (Nikon D800 (NIKON CORPORATION, Tokyo, Japan)) of a 36.3 Mpixels resolution the software was unable to identify the points of a chessboard calibration field. The effect of defocus in the part of the image that was away from the center of the image (or the point in focus) introduced the biggest problem ( Figure 4).
Therefore we decided to use a regular grid of circles as calibration field. In the case of the circular grid, the barycentric algorithm that we used to identify the exact location of the circles' centers, was not affected by the defocusing effect. Another issue appearing in some cases is the bad lighting of the calibration sheet during the image capture. Not all the people that want to calibrate their cameras have the experience and awareness to deliver appropriate calibration image shots, especially students that want to use their amateur cameras for photogrammetric recording purposes. Low luminance and a gloss effect ( Figure 5) that might appear when the light is hitting the calibration field page from a certain angle might confuse the software and block the algorithm from identifying the control points on the page.
In general all calibration software functions as a "black box" application and the user has no option to adjust the settings of the algorithms to identify the control points. When using wide angle lenses and in cases where the camera location has a big inclination with the calibration page, the size of the control points' circles varies a lot. In some cases the circle radius of the most remote circles could be three times smaller than those close to the lens, especially in the case of wide angle lenses ( Figure 6).
This last issue severely hinders the success of the calibration adjustment due to possible wrong numbering of the control points on the calibration field. The calibration adjustment function assigns internally to every control point a ground space coordinate according to the distance parallel to the x and y direction of the columns and rows of the calibration sheet. Therefore for the case of a 5 × 7 (rows × columns) calibration grid and a distance of 37 mm between each row and column, the first point on the top left corner of the calibration field gets ground space coordinates (0, 0, 0), and all the control points on the same (first) row have the same y (= 0 mm) and increasing x (from 37.0 to 222.0 mm) and the control points in the succeeding columns get x coordinates from 0 mm to 222 mm and y coordinates from 37 mm to 148 mm. For all the points the height (z coordinate) is the same and is 0.0. Therefore the calibration sheet has to be planar in order to achieve the highest accuracy of the calibration parameter calculation. To identify properly a control point at its correct location on the image the software applies a sorting method. For the case of a regular grid of circles of 5 rows and 7 columns the first seven points, identified on the calibration image, are supposed to be visibly closer to the top edge (first image row of the image). However, if the camera's primary axis (implemented by the line connecting the center of lens and the center of the sensor) is rotated very much the software erroneously sorts the control points and the software fails to calibrate the camera (Figure 7).   . High inclination of a camera's principal axis giving wrong sorting and numbering of control points (red colored points' codes). The point with No. 8 (code in white color inside the circle) is erroneously getting the code No. 4 (red colored code) just because its j coordinate on the image (distance from first image row) is smaller than the j coordinate of all the other points below the green line.

Software Options
For all the above-mentioned problems and to increase the functionality of a calibration application, we decided to implement a tool with a graphical user interface giving to the user the opportunity to have better control on the calibration algorithms, and therefore to:  Adjust the threshold algorithm and isolate the circular control points' grid from any noise that might appear on the edges of the calibration images;  Define the minimum radius of the control point circles since the wide angle lens and the camera position might affect rapidly the variance of the circles' diameter;  Examine the correct location of the control points on the calibration sheet by providing the correct numbering of all the control points on the image;  Undistort the still images and create their idealized respective images; the idealized images may be used in single photo photogrammetric procedures to provide rectified photos, i.e., of building facades [20];  Isolate frames of a video sequence capturing the circles calibration grid and save them as independent images to perform the calibration of the video sensor; since the calibration of the video sensor is performed the undistortion of any video sequence is feasible.
The created software is characterized by its simplicity and the possibility to interact with other 3D recording software. The user has just to import a set of images using the menus of the Graphical User Interface (GUI) of the software and define the sensor size of the camera to calibrate. The images should be captured in such a way that the calibration page will cover almost the total width and height of the sensor from different locations above of the planar surface of the calibration field ( Figure 8). We avoided using a single image calibration setup [21,22], since the result could be inaccurate due to potential asymmetry of the camera construction. Although the information of the sensor's dimension optionally delivers the calibration parameters in pixels (f, x o , y o ) it is important for a photogrammetrist to identify the intrinsic parameters of the cameras in millimeters. The rest of the parameters (k 1 , k 2 , k 3 , p 1 , p 2 ) are dimensionless numbers.
The sensor dimensions of any commercial camera is easy to be found using the Digital Photography Review [23]. Reviews of almost all of the new and old commercial cameras, including their basic features, are available on the site. Unfortunately though, there is no appropriate information for the GoPro camera or for any cell phone camera. For the GoPro Hero 3 camera the sensor size information can be derived from the official site of the manufacturing company [24], since it gives the dimensions of a single pixel on the camera image in microns (1.55 µm). Therefore for the highest still image resolution the sensor size is estimated to 6.20 × 4.65 mm. Although, the dimensions of the sensor could be calculated from its diagonal (1/2.3") we decided to use the value calculated by multiplying the image width and height with the pixel size given at the official site of the manufacturing company, because the calculated focal length in the case of the pixel size defined sensor dimension (6.20 × 4.65) was 2.6 mm, a value much closer to the nominal (3 mm) given by the manufacturer of the camera. In the case of the sensor size defined by its diagonal in inches, the focal length was estimated at 2.59 mm. In any case the created undistorted images are not affected at all by the sensor size in mm.
Since the user has added the images of the calibration sheet, has defined the sensor size dimensions and also the distance of the control points grid in mm, the calibration (identification of control points on every image and adjustment to calculate the calibration parameters) can follow. The software browses every image in a graphical window on which the user has to define:  The threshold value to separate the control points circles from the surrounding area  The minimum radius of circles to be identified as control points and  The area including points (Figure 9) Although the target identification process is semi-automatic, the total time spent by an average user (namely the students of our post graduate course in IUAV) to adjust the correct parameters and define the area including the targets did not exceed a couple of minutes. Since this operation is performed only once for every camera (or at least once every six months for professional photogrammetric labor) we decided to follow an absolutely correct calibration process, allowing the user to watch and verify the correct location and numbering ( Figure 10) of targets on the calibration field, rather than letting the software perform it without any human interaction and operate as a "black-box" device.  After the user has used the images and the control points have been identified, the software prints in the main window application the results (Figure 11), which can be saved for future use in a project (text) file. In our case the focal length of the GoPro camera was about 2.60 mm and the estimated accuracy (Average Reprojection Error) of the calibration was 3.91 pixels. The calibration project file might be used in the future to undistort images taken with the same camera's modes like the one used to capture the calibration images. Figure 11. Calibration parameters as they are calculated for the 12 Mpixels still images from the GoPro Hero 3 camera.
The image undistortion procedures can follow right after the calibration procedure or can be performed in another occasion since the calibration project file has been stored in the computer's disk file system. The user has just to load the images (at least two) and for every single image, the software performs the undistortioning process (Figure 12a,b). Finally, although not so much photogrammetry related, an important feature of our software is the possibility to calibrate the sensor when capturing video sequences and perform further the undistortioning of a whole video file.
Since the part of the sensor that is used to capture every frame of any video mode is smaller than the full resolution of 12 Mpixels we must calculate the exact dimensions of the sensor to perform the calibration. For this reason another simple software applications was created, which is able to calculate the part of the sensor used to create lower resolution (<12 Mpixels) images, using an affine transformation. We can load and browse in two separate graphics windows a full resolution 12 Mpixels image and a smaller one, taken from the same camera location, and if we can identify at least three common points, we can estimate the dimensions of the sensor part used to capture the low resolution (still or video) image (Figure 13), given that the sensor size of the full resolution image is 6.20 × 4.65 mm.
The video file depicting the calibration sheet can be loaded in the software. The user can isolate any image from the video and save it as a single calibration image. Following the similar procedure of still images' calibration we can perform the calibration of the video frames. After the completion of the video frame calibration, the user can load an original video file and recreate another one, idealized, with respect to the original, using the undistorted frames ( Figure 14). Finally, the software has the ability to import xml files of camera calibration parameters produced by Agisoft Lens or Agisoft Photoscan. Agisoft Lens is a free software that can calculate the same calibration parameters (f, x p , y p , k 1 , k 2 , k 3 , p 1 , p 2 ) using a calibration screen, while Agisoft Photoscan is a commercial low cost application that can calculate these parameters using just the conjugate points on overlapping scenes of the recorded objects. Unfortunately, the software does not provide the possibility to create the idealized images, but using the undistortion option of our software we can create the undistorted images produced by any calibrated camera.

Accuracy Assessment
The use of a widespread and low-cost digital camera combined with easy to use calibration software, such us the free one implemented by the authors (also considering other commercial solutions), offers even non-expert users the possibility to obtain metric data: just considering the immediate results obtainable in teaching or in other different researching areas such as archaeology.
We have tested the use of the GoPro Hero 3 camera on top of a Parrot AR.Drone 2.0 (GPS edition) and have taken a series of images to test the functionality of the camera to provide images of 3D architectural details on a facade not easily reachable from the ground. Both the camera and the drone was controlled from separate mobile devices. The images taken from drone were used to create a complete 3D model of the logia in the university campus giving most accurate results.
For the 3D photogrammetric processing we have used the software Agisoft Photoscan Professional (version 1.04) and the accuracy in alignment of the GoPro distorted images was about 0.035 m ( Figure 15). Using the undistorted ones (Figure 16), the RMS is halved (0.015 m).
Moreover, as shown by some tests we performed using different modes of the camera and capturing schemes and considering other published experiences [21], the geometric scheme that optimizes results, in terms of accuracy, is that of zenithal strip coverage of the objects to be recorded (e.g. parallel to the average plane of the surveyed surface for the architectonical facades), with high overlap (Table 5). Oblique photos demonstrated greater RMS error in the alignment as can also be seen in Table 6 and Figure 16.
This integrated system (GoPro + Parrot AR.Drone) is going to be applied and an in depth test in a archeological surveying campaign that will be carried out in the ancient city of Sepinum, in the south of Italy, to verify its reliability, comparing the results obtained by using a "typical" photogrammetric equipment (such as calibrated camera mounted on a drone).   Within the archaeological documentation or supporting restoration, undistorted images are useful when we want to rectify a planar surface (i.e., building's facade) to further digitize its architectural details. After several tests we have made, we came to the conclusion that the accuracy of the rectification is increased significantly when undistorted images are used.
In the case of the GoPro camera, the originally created images are not appropriate to be used for the rectification of planar facades. RMS error (Table 7), due to the distortion, exceeds 1cm and additionally, points that lie very close to the edges of the image are totally unusable (Figure 17a). The rectification of the undistorted image gave very good results (Table 8) and also the curvature effect (of straight lines in ground space) was totally eliminated (Figure 17b).  Figure 17. The rectification of the original (a) and undistorted GoPro images (b) lead to an RMS error of the collected GCPs of 0.0103 m and 0.002 m respectively using the VeCAD-Photogrammetry application [20]. It is also visible the curvature of straight lines on the original image (a).
(a) (b)  The same rectification procedure was used also for original and undistorted images of a dSLR NIKON D100 camera (NIKON CORPORATION, Tokyo, Japan) equipped with a very good quality lens (NIKOR 20 mm/2.8 D AF). The RMS errors (Tables 9 and 10) and rectification residuals (Figure 18a,b) were very small, however they were worse than the undistorted GoPro camera case.

Conclusions
The possibility to calibrate a low cost camera and especially the GoPro Hero 3 camera which is additionally an action camera is providing lots of benefits:  The cost of camera is very low and does not exceed 500€;  The cost of accessories necessary to use it in extreme conditions is zero since it is included in the package;  The cost of additional equipment to carry the camera for other activities is very small. For UAV applications a low cost drone might be more than enough to carry it and perform remote shootings using just a cell phone applications.
The developed application itself is easy to use by novice users, gives a very good visual corrected result (of the undistorted still images and videos) even to non-photogrammetrists, but provides professional photogrammetrists all the appropriate information to apply the resulting calibration parameters and the derived products for further exploitation in photogrammetric software.

Author Contributions
C.B., F.G., P.V. and V.T. conceived and designed the experiments; V.T., P.V. and C.B. performed the experiments; C.B. and V.T. analyzed the data; F.G. contributed equipment, materials and analysis tools; C.B. and V.T. wrote the paper.