Machine Vision System for Automatic Adjustment of Optical Components in LED Modules for Automotive Lighting

This paper presents a machine vision system that performs the automatic positioning of optical components in LED modules of automotive headlamps. The automatic adjustment of the module is a process of great interest at the industrial level, as it allows us to reduce reworks, increasing the company profits. We propose a machine vision system with a flexible hardware–software structure that allows it to adapt to a wide range of LED modules. Its hardware is composed of image-capturing devices, which enable us to obtain the LED module light pattern, and mechanisms for manipulating and holding the module to be adjusted. Its software design follows a component-based approach which allows us to increase the reusage of the code, decreasing the time required for configuring any type of LED module. To assess the efficiency and robustness of the industrial system, a series of tests, using three commercial models of LED modules, have been performed. In all cases, the automatically adjusted LED modules followed the ECE R112 regulation for automotive lighting.


Introduction
LED-based lighting systems have been extensively used in the automotive industry for more than 30 years.Initially, they were used for interior lighting and rear signalling functions, but the continuous progress of optoelectronic technology made them a viable alternative for forward lighting [1].Due to their reliability, low power consumption, compact size, and long life time, white light LEDs (WLLED) are becoming a popular source for headlamp illumination [2,3].
One of the main barriers to use WLLED sources in vehicle forward lighting was the luminous efficiency, but in 2006, it was exceeded the limit of 100 lm/W, enabling their use in this application [4].For the first time in 2007, a luxury car was equipped with an LED headlamp [5].This was the beginning of a new era in vehicle front lighting, and nowadays, this technology is outpacing other options such as Xenon or incandescent bulbs [6].
The main function of a headlamp is to illuminate all objects on the roadway, avoiding inducing glare to drivers or pedestrians.To this end, headlamps project a light pattern that must follow the Economic Commission for Europe (ECE) R112 regulation [7].Light patterns have to meet the luminous intensities defined in the ECE R112 at certain test points.In a headlamp, light patterns are generated by the LED module [8].One of its optical components is the thick lens, which is made of plastic [9].Due to the high volume of demand for modules, thick lenses are manufactured by injection molding and specifically in multi-cavity molds.Even if identical parts are injected at the same time, the tolerances in the different cavities cause dimensional deviations among them [10,11].This has a significant bearing on the photometric results of the module.
As a result of the above, the mass production of modules require high precision manufacturing and assembling of the optical components included in them.Any dimensional variation in its optical components has an influence on the final module assembly.LED modules are currently assembled to a fixed position.This implies that when there are dimensional variations in their optical components, most of them have to be rejected and manually adjusted to meet with the ECE regulation.
The above confirms the industry's need for an automatic adjustment system capable of optimally assembling LED modules.The system should also comply with the following requirements: • Reliability and robustness: The hardware devices and the software algorithms used to obtain the optimal fixing position and adjust the module should guarantee that it complies with the regulation.• Capability for real-time adjustment: Since the automatic adjustment system is one element of the production chain, the process must be performed in the pre-established amount of time.These cycle time requirements will significantly affect the positioning algorithms.• Adaptability: The automated adjustment system should be easily adapted to different models of LED modules.For this reason, it is absolutely necessary that the system's components may be included, eliminated, or even modified with ease.With the intention of achieving this goal, we introduce a novel automatic adjustment system designed for LED modules.This system possesses versatility and can be tailored to various part models.The system follows a component-based development approach [12], which allows us to combine commercially available components (programmable logic controller (PLC) control software and computer vision algorithms) with custom-made ones (very specific image processing algorithms, data presentation strategies, etc.).From an academic point of view, this MVS design approach is valuable for new devices to be integrated into an Industry 4.0 environment and its new paradigm, Zero Defect Manufacturing [13,14].
The remainder of this paper is as follows.First, the definition of the problem and related works are introduced in Section 2. Section 3 details the design of the machine vision system, analysing its hardware and software architectures.The experimental validation of the system, using a commercial LED module, is presented in Section 4. The paper ends with the conclusions given in Section 5.

Definition of the Problem and Related Work
In this section, we describe the structure of an LED module and how it is currently assembled to meet the criteria defined in the regulation.We also list previous proposals that address a similar issue: the automatic assembly of parts with dimensional variations.

LED Modules for Automotive Lighting
LED-based automotive headlamps include a variety of lighting functions such as low beam, high beam, daytime running light, and fog beam.Basically, these lighting functions follow two types of lighting patterns and what mainly differentiates them is the inclusion or non-inclusion of the cut-off line [6].The principle of the cut-off line is to protect people by decreasing glare on the roadway.The low beam and the fog beam require a high-contrast cut-off line, while the high beam and the daytime running light are without it.
Regarding illumination patterns, the high beam provides a clear vision to the driver but should only be used to illuminate far-off distances where visibility is mostly low.Its lighting pattern is an ellipse and, because of this, it induces serious glare to people on the roadway.The low beam pattern is much more complicated because of the cut-off line (Figure 1).As can be seen, the dark part above the cut-off line should provide limited illumination to avoid glare.The bright part, below the cut-off line, should cover the important viewing zone for the driver.
Currently, there are automotive headlamp models that combine the low beam and high beam in a single LED module [15,16].Some automotive lighting companies refer to this concept as BiLED [17,18] and it is mainly characterised by its compactness and reduced mechanical complexity.Figure 2 shows the BiLED concept developed in 2015 by one of these companies [17].In this design, the light source comprises high power multichip LEDs arranged on a two-layered PCB and three optical active components responsible for the light beam distribution.These optical components are (i) the reflector collector, to recover and address the LED light flow; (ii) the folder mirror, to define the focal plane; and (iii) the lens, to deploy the light distribution on the road.A plastic frame assures the relative position of these three optical components and the PCB.It must be noted that the dimensional variability of each individual component can affect the optimal location of the focal plane.This variability can be corrected with a specific assembly position for each module.This is why the folder was designed having oblong screwing holes, allowing it to find an optimal relative position between submodules (Figure 3a,b).

Manual Positioning and Quality Control
In the mass production of BiLED modules, nonconformities are detected due to the manufacturing process of the active optical components and the processes associated with their assembly.In addition to not complying with the current regulations, the color of the cut-off line may be the reason for the rejection of the part.Cut-off colors that are too bluish or reddish make the modules unacceptable to the customer.The rejected BiLED modules can be reworked in special working stations as the one shown in Figure 3c.These stations where designed and constructed for a manual positioning of the folder according to a visual inspection of the cut-off projection.This manual positioning process had the following steps:

•
Remove the parts that prevent access to the active optical components.

•
Loosen the screws securing the optical component.• Switch on the module and project the light beam onto a front screen.

•
Manually adjust the relative position of the optical components according to the aspect of the projected image.This step was subjected to the limitations of visual perception and the fatigue caused by the observation of light distributions with strong light-dark contrast.

•
Once the correct position of the components has been found (limited by the poor visual precision of the human eye), lock this position by screwing on the optical part.

•
Mount and screw the rest of the non-functional components of the module.

•
Check the photometry of the module on the photometric bench again.
If after the manual readjustment steps the result was compliant, the module was labelled and packaged for assembly inside the headlamp; however, if the result was not compliant, all the readjustment steps had to be performed again.All of this led to an increase in time and cost overruns in the production of the module.
The above mentioned justifies the need for an automatic positioning system capable of obtaining the optimum relative position of the module's optical components and their attachment to each other.The automatic adjustment system will eliminate module reworks, increasing the efficiency of the production process of BiLED modules.

Related Works on Automatic Assembling of Parts with Dimensional Variations
Assembly is one of the most important stages of a company's entire manufacturing process.It is especially critical in certain industries such as aeronautics or automotive, where products with high precision requirements are manufactured [19].The final assembly accuracy of products is dependent on many factors such as the dimension and geometric tolerances of the parts to be assembled, the assembly clearance of parts, the load deformation, the deformation caused by residual stress release, and the adjustment of the assembly process [20].
As the assembly process directly affects the product quality, it is very important to carry out an assembly precision analysis and optimisation.In this regard, there are numerous contributions that propose models for achieving assembly precision requirements at a low manufacturing cost.In [21], a semantic-based assembly precision optimisation method was proposed considering the process capacity illustrated via an aircraft inner flap.The tradeoffs between the cost and achievable variation limits of the entire manufacturing chain was studied in [22].A method based on a general part digital twin model was presented in [23] to improve the reliability of the assembly precision.The proposal considered other factors that are usually ignored, such as the assembly-positioning error and the deformation of the mating surface.
One way to increase the assembly precision of a product is to reduce the tolerance range of the parts.However, this cannot be reduced unlimitedly and it can lead to higher costs [24].A clear example of this problem are the plastic parts, which are increasingly being used as the main elements of diverse commercial products.One of the most widespread drawbacks of these types of parts is their dimensional variations or manufacturing tolerances, especially in ones with complex geometry or considerable size [10].This issue was analysed in [25], where a prototype for assembling the cover lens and the housing of vehicle headlamps was presented.After analysing different technologies, a fuzzy controller was implemented, which, based on the data provided by laser and micrometric contact sensors, gave orders to linear magnetic actuators to fix both parts.
According to the previous works reviewed, the assembly precision optimisation process contains a wealth of expert experience and knowledge which can be used as important information sources.Our proposal for an automatic adjustment system for LED modules is based on the manual position system which is utilised for reworks.As with the manual system, measurements and adjustments are required for the assembly process to ensure that the LED module follows the ECE R112 regulation and the specific requirements defined by the customer.

Description of the Machine Vision System
A Machine Vision System (MVS) for the precision assembly of LED modules (Figure 4) has been designed, constructed, and programmed.The automatic assembly is based on a measurement and adjustment procedure.Measurements are obtained from the hardware devices to acquire the low-beam lighting pattern projection at 25 m (photometry devices) and the image processing algorithms to obtain the required information for the control algorithm.The assembly optimal position is obtained via the control algorithm based on the reference, which is previously defined via a calibration procedure.In this section, we present the MVS hardware and software architectures.Some diagrams of the Unified Modelling Language (UML), a widely adopted tool for system engineering modelling [26], implement the MVS description.The camera and the photometric tunnel are hardware devices which interfaces with the Camera software component.This software component is in charge of acquiring and processing the image.For the image acquisition, the libraries provided by the camera manufacturer are required.For the processing, specific libraries such as the OpenCV and proprietary computer vision algorithms are needed.
The MVS includes a series of elements which allow us to position and assemble the LED submodules.These elements are the tooling and the screwdriver.Both interfaces with the software component Positioning and are controlled by a PLC connected via EtherCAT to the host PC using the TwinCAT environment.The PLC also controls the lighting of the LED module through the power drivers.
Two HMI screens are part of the MVS hardware architecture.The HMI 1 screen interfaces with the component software GUI.This component enables us to show results and tune the camera and processing parameters.The HMI 2 screen interfaces with the PLC component AutomationProject.Through this component, it is possible to directly control and move the devices that manage the relative movement and adjustment of submodules.In addition, the finite state machine is implemented which controls the order of execution of the software components and how they relate.We next present a further description of the image acquisition and position adjustment devices.

Image Acquisition Devices
The devices related to the image acquisition are the vision sensor and the photometric tunnel (Figure 6).The vision sensor is one Ethernet Basler colour camera with a 1.3 Mp resolution and an acquisition rate of 30 fps.It is used for the study of the regions of interest of the low-beam pattern.The photometric tunnel allows us to acquire the lighting pattern projection at 25 m.It includes a lens, a background panel, and the necessary tool for the vision sensor placement.In addition, it is a closed structure that prevents any type of light contamination during the image acquisition.

Position Adjustment Devices
The tooling is the device where the relative movement between submodules takes place (Figure 7).Since the precision for the folder movement is two-tenths of a millimetre and the oblong screwing holes allow a displacement of 2 mm, eleven discrete positions are available for the assembling.With each of these eleven positions, a different pattern of cut-off light is obtained.Once the best position is achieved, the folder screws are fixed using a three-axis screwdriver.

Software Architecture
The MVS software architecture follows a component-based development approach [27] which is illustrated in the UML component diagram presented in The Automation Project controls the order of execution.Through the offered services, Trigger and Lighting, the component switches the BiLED module and triggers the camera to start the image acquisition.The required service is Position, provided by the Positioning.Each time it receives a new position, the relative position of the BiLED submodules is modified by the tooling.When the position does not vary between two consecutive iterations, it means that the optimal position has been reached and the assembling concludes.
The Positioning requires the services, Vision Inspection Result and PositioningConf, to offer Position and Optimal Position.The Vision Processing requires Image and ProcessingConf services to offer Vision Inspection Result.The main algorithms, executed in both components, are detailed below.

Image Processing
The computer vision algorithms have to perform photometric and chromatism measurements of the low-beam pattern.Basically, these measurements are the cut-off sharpness and the cut-off color.In order to match the ECE R112 regulation, the low-beam pattern is required to contain a clear cut-off line, which has to achieve a certain standard in illumination contrast.In addition, the customer imposes limits to the cut-off colour which must be met for the LED module to be accepted.How to obtain both measurements is detailed below, as well as the previously needed calibrations.
-Photometric.This calibration establishes a correlation between the gray level of the image and a standardised illuminance value.A procedure is established using a standard module to apply corrections to the intensity values acquired by the vision sensor.-Geometric.This calibration determines the relationship between the pixels in an image and sexagesimal degrees in spherical coordinates.To achieve this, a calibration pattern (Figure 9) is placed on the background panel of the photometric tunnel.The calibration pattern is placed at a distance of 700 mm and the length of the square in the center measures 43 mm.The pattern is acquired by the vision sensor and processed to obtain the length of the square in pixels (170 pixels).The desired pixel-degree ratio can be obtained from: where n p is the number of pixels between two points in the image, and θ is the angle formed by these two points in spherical coordinates.
• Cut-off measurements.For these measurements, the V point in the horizontal part of the cut-off line has to be located (Figure 10b).Traditional image processing algorithms are used for this.As the gray level image shows a bimodal histogram with a homogeneous background, the Otsu algorithm is used for segmentation [28].Then, via the Hough transform, the cut-off line shape is extracted [29].The lower intersection point is the desired V Point.
-Sharpness.It is determined by vertically scanning three sections placed through the horizontal part of the cut-off line (at H 1.5 • , 2.5 • , and 3.5 • , Left or Right, depending on LHD/RHD from the V point).During the scanning, the gradient G is determined using the formula: where E is the illuminance, β is the vertical position in degrees, and δ is the scan resolution, also in degrees.The sharpness of the cut-off line is the point with the highest gradient with respect to the H-H axis.
-Chromatism.The low beam color has to be inside the specific limits defined in the CIE diagram.Figure 10b

Optimal Position Adjustment
Figure 11 shows a flow diagram of the designed algorithm to achieve the optimal position in which the LED module has to be assembled and adjusted.First, to determine this position, it is necessary to carry out a calibration phase.In this phase, already assembled modules, which have been validated in a photometric tunnel and meet with the regulation, are placed on the MVS tooling.Modules are lighted with the purpose of obtaining the cut-off measurements with the MVS.According to the previous description of the image processing algorithms, these measurements are the sharpness and the chro-matism.The sharpness can be mathematically modelled as CW std n,v , where n is the number of modules used in the calibration phase and v indicates each of the three vertical lines over which the maximum gradient is obtained.The colour features for the chromatism are stored in the array CL std n, f , where f denotes each of the two features considered to measure the low beam colour.Subsequently, six characteristics describe the LED modules used for calibration (D std ).The first two are related to the cut-off sharpness (Equations (3) and ( 4)) and the remaining four to the chromatism (Equations ( 5)-( 8)): where the variables CW std , CL std 1 , and CL std 2 correspond to the equations (Equations ( 9)-( 11)).
Second, the optimal position adjustment procedure for new modules is carried out in the production phase.This position has to be reached according to the following sequence:

•
Step 0: Check if the LED module model has a previously learned optimal position.To do this, a query is made on the database of learned positions.If the model already has a registered position, this is the initial one.If not, it is initialised to an intermediate position.The position vector is initialised as follows: where n is the initial position (learned or intermediate).

•
Step 1: Set the LED module to the position p i .First time i = 0.If p i is outside the position range allowed by the tooling, the module is adjusted in the intermediate tooling position.For this module, it is not possible to reach an optimal position, so go to Step 4. If it is within the range, go to Step 2.

•
Step 2: Acquire and process the cut-off shape image to obtain the cut-off measurements: ), the module has been correctly adjusted and assembled and the position is recorded in the database; if so, go to Step 4. If not p i = p i+1 , then go to Step 1.

Experimental Case of Study: BiLED Module
The experimental validation was conducted with commercial BiLED modules.A total of 112 modules of three different models were used for the calibration phase.For the production phase, 134 modules were used, also belonging to the three models used for the calibration.Both phases are detailed below.

Calibration Phase
To carry out the calibration, as a starting point, modules adjusted to the customer's requirements and validated in a photometric bench were taken as a reference.Specifically, the three models of BILED modules identified as TD-IZQ, IO1-TI-IZQ, and F56-USA-IZQ.A total number of 112 modules were used for this task (40, 48, and 24 modules for each model, respectively).The BiLED models and the number of them were selected by the customer based on his specific necessities.Table 1 shows the features cut-off sharpness, percentage of blue, and percentage of red obtained by the MVS for each of the former modules.
Based on the parameters extracted from each module, the acceptance ranges for each of the models were obtained.Table 2 shows these acceptance ranges.As can be seen, the most critical ranges of acceptance are those of the model F56-USA-IZQ.This means that, for this model, it will be more complicated for the MVS to reach an optimal position.

Production Phase
In this phase, BiLED modules were automatically adjusted and assembled by the MVS.The optimal assembly position and the cut-off measurements, for the reached positions, are shown for each of the BiLED models in Tables A1-A3.For each of the tables, the columns present the following information: module ID, the cut-off sharpness, the chromatism analysis (% red and % blue), and the optimal position.If it is not possible to find an optimal assembly position for the module within the acceptance range, it is represented as "-".In this case, and according to the positioning algorithm, the module is assembled in an intermediate position and it is separated for a subsequent manual rework or rejection.
The MVS performance was tested with 134 BiLED modules belonging to the following models: TD-IZQ model (58), IO1-TI-IZQ (37), and F56-USA-IZQ (39).A detailed description of the results of the production phase is now given and discussed.
For the TD-IZQ model, the results of the inspection of 58 modules are displayed in detail in Table A1.The MVS correctly adjusted 83% of the analysed modules.The rest did not meet the acceptance range for any of the tooling positions.The cut-off sharpness feature was correct for 97% of the modules and the colour features were within the range for the 83% percentage of red and the 95% for the percentage of blue.Furthermore, most of the modules reached their optimal sharpness and colour features in positions 4 and 5. Figure 12 shows these results.
For the IO1-TI-IZQ model, the results were quite similar.In this case, of the 37 modules, 89% were optimally adjusted.Almost for all the modules the three features were within the established acceptance ranges.The optimal position that was repeated most frequently was number 4. These results can be seen in Figure 13 and in detail in Table A2.The worst results were obtained with the F56-USA-IZQ model.Of a total of 39 modules, 38% had the three features within their acceptance ranges, while 92% showed an acceptable cut-off sharpness; for a little more than half of them, 54%, it was not possible to be within the range of the percentage of red.This was the main reason why the MVS was not able to adjust 62% of the modules.The results for the percentage of blue were better as 72% of the modules were within the range.Once the optimal position was adjusted, the most repeated were number 4 and number 5. Figure 14 and Table A3 show the former data.
It is worth mentioning that all the automatically adjusted BiLED modules followed the ECE R112 regulation when checked in a photometric bench.

Figure 1 .
Figure 1.The low-beam lighting pattern.Left: the cut-off line shape; right: low-beam projection in an inspection tunnel.

Figure 2 .
Figure 2. LED module of a headlamp producing both low-and high-beam light "BiLED module".

Figure 4 .
Figure 4. Machine vision system for the automatic precision assembly of LED modules.

Figure 6 .
Figure 6.Image acquisition devices.The photometric tunnel and the colour camera inside the tunnel.

Figure 7 .
Figure 7. Tooling and the cut-off light pattern for a sample of the eleven discrete positions of the folder.
shows the limit specified by the customer in the CIE diagram.RGB values of the pixels inside the ROI in the cut-off line are changed to the XYZ CIE coordinates.Only X and Y coordinates are taken into account to place the RBG values in the CIE diagram.The customer defines the black line in the CIE diagram and a specified percentage of points should remain on the left (% blue) and the others should remain on the right (% red).

Figure 9 .
Figure 9. Calibration pattern used to perform the geometric calibration by the MVS.

Figure 10 .
Figure 10.Measurement obtained from the cut-off pattern: (a) measured sections on the left side of the V point used to obtain the cut-off sharpness; (b) color measurements in the CIE diagram from points placed on the border of the cut-off line.

Figure 11 .
Figure 11.Algorithm for the optimal position adjustment.

Figure 12 .Figure 13 .
Figure 12. Results of the analysed modules belonging to the TD-IZQ model.

Table 1 .
Features obtained for each unit inspected during the calibration process.

Table 2 .
Acceptance ranges obtained for each model.

Table A2 .
Results of the analysed modules belonging to the IO1-TI-IZQ model.

Table A3 .
Results of the analysed modules belonging to the F56-USA-IZQ model.