Inter-Laboratory Correlation Exercise with Portable Emissions Measurement Systems (PEMS) on Chassis Dynamometers

Featured Application

Repeatability and reproducibility of the PEMS measurement method.

Abstract

The recently introduced Real Driving Emissions (RDE) light-duty vehicle emissions regulation requires testing with Portable Emissions Measurement Systems (PEMS) during type approval and in-service conformity. The studies on the accuracy of PEMS today are limited. An inter-laboratory correlation exercise with PEMS took place in Italy in 2017. Eight laboratories measured exhaust emissions from a Golden Euro 6 gasoline vehicle with a Golden PEMS installed in it, along with the individual lab’s own PEMS, following the regulated laboratory method (bags from the dilution tunnel). The data of the exercise were used to estimate the repeatability and reproducibility of the methodology with PEMS. The statistical analysis estimated reproducibility of 2.9% (bags) to 5.5% (lab PEMS) for CO₂, 20–25% for CO (all methods), 23–31% for NO_x (all methods), and 29% (tunnel, Golden PEMS) to 39% (lab PEMS) for particle number. The mean differences of the PEMS to the regulated method were ±1.5 g/km (or ±1%) for CO₂, <16 mg/km (or <5%) for CO, <4 mg/km (or <11%) for NO_x and 1 × 10¹¹ particles/km (40%) for particle number. The results of this study confirm the satisfactory performance of PEMS and the permissible tolerances introduced in RDE regulation.

1. Introduction

Vehicle emissions performance has been controlled since the 70’s in laboratories [1]. In current regulations worldwide, light-duty vehicles follow prescribed test cycles on chassis dynamometers and the whole exhaust gas is diluted in a dilution tunnel. A sample of the diluted exhaust gas is collected in bags and analyzed at the end of the test. However, for heavy-duty applications, only the engines with their after-treatment devices are type approved.

The first on-road testing started in the 90’s using Portable Emissions Measurement Systems (PEMS) [2]. PEMS were introduced in regulations for in-use compliance (in United States (US) since 2007) or in-service conformity (in the European Union (EU) since 2014) of heavy-duty vehicles without the need of removing the engine from the vehicle for testing it in the laboratory. Recently, PEMS testing was introduced for type-approval of light-duty vehicles (in EU since 2017) and will be necessary for in-service conformity with the introduction of the 4th RDE (Real-Driving Emissions) package. The measurement uncertainty of PEMS is taken into account in the limits that vehicles have to meet during the on-road testing: In the US with the measurement allowance program [3], in the EU with the measurement uncertainty framework study [4].

Testing with PEMS for market surveillance purposes will begin in 2020 in the EU [5]. For such testing, the instruments will have to fulfil the requirements of an under development standard from the European Committee for Standardization (CEN) (Comité Européen de Normalisation), which will assess the PEMS performance in the laboratory under various boundary conditions (e.g., ambient temperature and pressure, vibrations) [6]. Additionally, the testing laboratories, which are not designated technical services, will have to be accredited according to the ISO (International Organization for Standardization) 17025 [7] and 17020 [8]. ISO 17025 accredited laboratories typically participate in inter-laboratory exercises (or proficiency testing or round robins as they are sometimes called), described in the ISO 17043 [9], to determine whether they are within the repeatability and reproducibility limits of a reference method. The inter-laboratory exercises are also used to assign a certified value to a reference material. They also aim at evaluating the performance of a method through the repeatability and reproducibility criteria. For example, inter-laboratory exercises (round robins) were conducted when the particle number method (e.g., [10,11,12,13]), the particle number PEMS procedure [14], or the new World Harmonized Light-duty Test Procedure (WLTP) [15] were to be introduced in the regulations. Such exercises are also common practice in the automotive industries to assess their uncertainty, however, the publicly available data is limited.

On-road PEMS testing has become a necessary source for emission inventories because they provide the “true” vehicle emissions under a wide range of operating conditions. Laboratory testing has limitations to simulate some real-life cases, for example, large road gradients, strong accelerations, and variations in altitude. In some cases, it was found that laboratory and on-road emissions had differences by a factor of ten. Today PEMS are robust and reliable tools for on the roads; however, the studies addressing their measurement uncertainty are limited.

The main objective of this study is to present the repeatability and reproducibility of the PEMS measurement method and compare these values with the respective values of the regulated laboratory method. The results of this study are a first step in defining (internal) acceptable limits for the participating laboratories. The possibility to run inter-laboratory exercises only with PEMS, without chassis dynamometers testing, will also be discussed.

2. Materials and Methods

CUNA (Commissione Tecnica di Unificazione nell’Autoveicolo) is the Italian Standardization Body for automotive, federated to UNI (Ente Nazionale Italiano di Unificazione), Italian Standardization Body. CUNA, as proficiency testing provider, organizes annually inter-laboratory activity among Italian laboratories. The goal is to determine the repeatability and reproducibility of the regulated methodology over time. Additionally, in 2017, CUNA organized a PEMS inter-laboratory exercise, as it will be described below.

2.1. Test Protocol

Thirteen laboratories conducted three repetitions of the cold start World harmonized Light-duty Test Cycle (WLTC) with pre-defined road-loads and gear-shift strategy, as described in the recently introduced European Union regulation [16]. For all these tests no PEMS were connected to the tailpipe. The emissions were measured, as defined in the regulation, from bags collecting diluted exhaust gas from the dilution tunnel during the test cycle. For particle number emissions, real-time analyzers were measured directly from the dilution tunnel. All laboratories fulfilled the regulated requirements regarding instrumentation and maintenance plans.

After the official tests, eight of the laboratories conducted another three repetitions of the WLTC with a Golden PEMS installed inside the vehicle. During these tests, five of the laboratories used in addition their own PEMS (installed outside of the vehicle) measuring simultaneously with the Golden PEMS. The comparison of PEMS to the bag results is called “validation” test in the regulation and is used to confirm the proper installation and operation of the PEMS. In the regulation permissible tolerances are also given. For example, for CO₂ the limits are 10 g/km or 10%, whichever is larger. For Particle Number (PN) the limits are given in the amendment [17] and are 1 × 10¹¹ particle/km or 50%, whichever is larger.

Two of the participating laboratories conducted additionally a pre-defined Real-Driving Emissions (RDE) test cycle in the laboratory using both PEMS. For this test, no bags were taken due to the long duration of the cycle, but real-time measurements from the dilution tunnel that were integrated to simulate the bag measurements. Details of the test protocol can be found in

Table 1. Test protocol in chronological order. Golden Portable Emissions Measurement Systems (PEMS) is the OBS-ONE (2 inches) (Horiba, Kyoto, Japan). The number in brackets indicates the exhaust flow meter diameter. All laboratories repeated 3 times the World harmonized Light-duty Test Cycle (WLTC), except E (2 repetitions). RDE means that a Real-Driving Emissions cycle was additionally conducted in the laboratory. MOVE is the PEMS from AVL (Graz, Austria).

Laboratory	Golden PEMS	Laboratory PEMS	Comment
A	Yes	AVL MOVE (2 inches)	RDE
B	No	AVL MOVE (2 inches)
C	Yes	Horiba OBS-ONE (2 inches)	RDE
D	Yes	No
E	Yes	AVL MOVE (2.5 inches)
F	Yes	No
G	Yes	No
H	Yes	AVL MOVE (2.5 inches)
8 Labs	7 Labs	5 Labs	2 Labs

.

2.2. Golden Vehicle and Laboratories

The Golden vehicle that circulated to the laboratories was a Euro 6b compliant gasoline port-fuel injection 1.4 L passenger car (thus with no applicable PN limit). All laboratories used reference fuel from the same batch, but their own driver. The first laboratory at the end of the round robin repeated the test and confirmed that the emissions remained the same, within experimental uncertainty. The laboratories that measured with PEMS, all located in Italy, in alphabetical order, were:

• FCA Powertrain Engineering, Pomigliano d’Arco (NA)
• FCA Powertrain Engineering, Cento (FE)
• FCA Product Engineering, Sangone, Torino (TO)
• Ferrari SpA, Maranello (MO)
• Innovhub SSI-SSC, San Donato Milanese (MI)
• Joint Research Centre, Ispra (VA)
• Lamborghini SpA, S. Agata Bolognese (BO)
• Landi Renzo, Cavriago (RE).

2.3. Instrumentation

The Golden PEMS was the OBS-ONE provided by the manufacturer (Horiba, Kyoto, Japan). It was installed inside the vehicle with a 2-inch-flowmeter connected at the end of the vehicle tailpipe [18,19]. The PEMS measured CO₂ and CO with heated NDIR (non-dispersive infrared detection), NO_x with heated CLD (Chemiluminescence detection) and particle number with CPC (Condensation Particle Counter) after a hot catalytic stripper [20].

The PEMS from the laboratories were the MOVE from AVL (Graz, Austria) or the Horiba OBS-ONE with 2- or 2.5-inch flowmeters. The AVL MOVE measured CO₂ and CO with NDIR, NO_x with NDUV (non-dispersive ultraviolet) [21] and particle number with DC (Diffusion Charger) after a hot catalytic stripper [22].

All participating laboratories used the same span concentration levels for the Golden PEMS, but for their own PEMS they used their typical span concentration levels. The zero calibration of the Golden flow meter was conducted with the dilution connected and in operation, while for the laboratory flow meters, the laboratories followed their own procedures.

2.4. Calculations

For the evaluation of the results, the following steps were followed. For each lab, the arithmetic mean values of the emissions were calculated. Then the mean of the means and their standard deviation were used for graphical purposes. For a normal distribution, about 68% of the values are within one standard deviation of the mean. The assessment of repeatability and reproducibility followed ISO 5725 (Appendix A) [23]. The methodology is appropriate for cases where the sample is the same for all participating laboratories, which is not necessary for the circulating vehicle. The vehicle introduces an uncertainty that is difficult to quantify and to separate from the uncertainty of the methodology. Any potential change of emissions over time (which was not the case here) is also difficult to take into account in this methodology. For this reason, in addition, for each test the differences between PEMS and bags were calculated and then their mean and standard deviation were compared with the limits set in the regulation. The limitation of this approach is that the calculated differences, which are assumed to be the uncertainty of the PEMS, include the uncertainty of the reference method (bags).

3. Results

Initially, the absolute emission levels and their variability will be given. Then the absolute and relative differences of PEMS to the regulated method from each test will be averaged.

3.1. Absolute Emission Levels

Figure 1 presents the mean emissions of the vehicle from all laboratories as measured with the Golden PEMS, their own PEMS (Lab PEMS) and the laboratory regulated method with bags (or real-time signal from the tunnel for PN). For the gaseous pollutants, the differences are small and insignificant. For PN the tunnel emissions are lower by about 22% compared to the PEMS. The statistical analysis for repeatability and reproducibility can be found in Appendix A. In general, the statistical analysis showed reproducibility of 2.9% (bags) to 5.5% (lab PEMS) for CO₂, 20–25% for CO, 23–31% for NO_x, and 29% (tunnel, Golden PEMS) to 39% (Lab PEMS) for PN. The variance of Lab PEMS was similar to the laboratory variance for CO and NO_x, while for CO₂ and PN it was higher. The reasons will be discussed in the “Discussion” section.

3.2. PEMS Differences

As a second approach for each test, the absolute and relative differences of the Golden and Lab PEMS to the bags were calculated. The means and the standard deviations of the differences for the pollutants are presented in Figure 2. The mean differences of PEMS to the bags were ±1.5 g/km (or ±1%) for CO₂, <16 mg/km (or <5%) for CO, <4 mg/km (or <11%) for NO_x and 1 × 10¹¹ particles/km (40%) for PN. These differences indicate negligible, if any, bias for the gaseous pollutants. The scatter of the differences lies within the permissible limits for the gaseous pollutants, with CO and NO_x reaching the relative permissible limits in some cases. However, for PN, it seems that the PEMS measurements at tailpipe are significantly higher than the measurements at the dilution tunnel. The PN scatter (error bar) is acceptable (30%).

4. Discussion

The pollutant with the lowest variance is CO₂. The reproducibility values can slightly vary depending on the laboratories that are considered. As an example,

Table 2. Bag CO₂ [g/km] mean emissions, repeatability and reproducibility of the inter-laboratory exercise for different combinations of laboratories.

	13 Labs (No PEMS)	8 Labs (No PEMS)	8 Labs (with PEMS)	5 Labs (No PEMS)
Laboratories	13	8	8	5
Mean value [g/km]	149.2	149.6	150.6	149.1
Reproducibility	2.3%	2.1%	2.9%	2.1%
Repeatability	1.7%	0.8%	1.6%	1.1%

gives the bags CO₂ emissions of different combinations of laboratories and using (or not) of PEMS. The column of the “13 labs” gives the results of the official inter-laboratory exercise with all 13 participating laboratories, where no PEMS were used. The “eight labs (no PEMS)” column refers to the official results without any PEMS connected of the eight laboratories that repeated later the tests using the Golden PEMS. The “8 labs (with PEMS)” column refers to the eight laboratories results when they used the Golden PEMS. The “five labs” column refers to the official results without any PEMS connected of the 5 laboratories that repeated later the tests using their own PEMS in parallel to the Golden PEMS.

Regarding the mean value, there were up to 1.5 g/km of CO₂ difference (149.1 vs. 150.6 g/km). The mean values of the 8 laboratories when using PEMS or when not using PEMS had 1 g/km difference. However, this difference is not due to the addition of PEMS: extracting a flow from the exhaust gas with the PEMS would reduce the final bag result if no correction of the automated systems for the extracted flow would be applied (but an increase was seen). In addition, this extracted flow is very small: around 3 L/min from each PEMS, for exhaust flow rates of 100 L/min (idle) to >2000 L/min (high speed). This means that the difference of 1 g/km is due to the variability of the different set of measurements. The reproducibility values vary from 2.1% to 2.9%, and they are in all cases lower than the reproducibility of the Golden PEMS (3.9%) or the Lab PEMS (5.5%), even when considering an equal number of laboratories (eight for Golden, five for Lab PEMS).

The (laboratory) reproducibility values of this study and others publicly available from inter-laboratory correlation exercises with gasoline cars are given in

Table 3. Reproducibility levels (%) from inter-laboratory exercises with gasoline vehicles. In italics values based on PEMS.

Emission Level	CO2	CO	NOx	PN	Comments
Euro 3 1	1.8	7	24	30	4 labs, NEDC [11]
Euro 4 1,2	2.0	5	18	44	4 labs, NEDC [12]
Euro 3	5.0	40	40	-	9 labs, many cycles [24]
Euro 6	1.6	15	29	9	11 EU labs, WLTC [15]
Euro 6 eq.	1.3	12	24	-	16 Asian labs, WLTC [15]
Euro 5 3	2.1	-	37	22	7 labs, WLTC [14]
	2.4	-	36	20	7 labs, 1 Golden PEMS
Euro 6	2.9	22	23	29	8 labs, WLTC, this study
	3.9	25	25	24	7 labs, 1 Golden PEMS
	5.5	20	31	39	5 labs, 5 Lab PEMS

¹ Results of original studies divided by the coverage factor 1.96 to make them comparable with the rest studies of this table. ² Estimated from graphs. ³ Based on standard deviation and not proper statistical analysis.

. The results are in good agreement and, according to the studies of Table 3, the CO₂ reproducibility is around 2–3%, CO is typically 15–25%, NO_x and PN between 20% and 40%. The PEMS reproducibility values are on the same order.

Table 4. Mean differences of PEMS from bags from inter-laboratory exercises with gasoline vehicles. In brackets the relative standard error (Coefficient of Variation), i.e., the standard deviation of the differences divided to the mean difference.

Emission Level	CO2	CO	NOx	PN	Comments
Euro 5	+7.1% (3.6%)	-	+1.5% (15%)	+6% (20%)	1 Golden PEMS [14]
Euro 6	−1.0% (4.0%)	−0.7 (14%)	+2.0% (9%)	+36% (32%)	1 Golden PEMS
	+0.9% (3.4%)	+1.5% (13%)	+11.3% (18%)	+41% (33%)	5 Lab PEMS

examines whether the PEMS absolute levels are also in good agreement with the regulated bags methodology. Indeed in most cases, the mean differences to the bags are within the scatter of the measurements (not statistically significant differences). The results confirm that there is no bias from PEMS with the exception of PN (see below). The differences of one (Golden) PEMS to the laboratory results were smaller than from many (laboratory) PEMS.

The previously mentioned results (Appendix A) showed that the PEMS reproducibility is at the same levels (for CO and NO_x) or only slightly higher (for CO₂ and PN) compared to the laboratory results. A detailed analysis showed that for PN there is a bias, while for CO₂ the scatter is higher (Table 4). Although, not in the scope of this paper, the following points should be considered:

• The installation of the flowmeters is critical. Sudden curves, sharp bends or diameter changes can result in high errors. Additionally, uncertainties at idle and low flow rates can result in over-or underestimation of the CO₂ (e.g., Reference [14]).
• The exhaust flow measurement uncertainty at idle can influence the CO₂ results (for the other pollutants the effect is small because cars typically have low emissions at idle and low loads/speeds) [25,26]. The zero calibration should be conducted with the dilution tunnel operating in order to minimize any influence of the under-pressure created by the constant volume sampler (CVS). Investigations with “open” CVS (i.e., the flowmeter is connected to the tubing via a cone that permits ambient air also to enter) would minimize pressure and pulsations from the CVS and should be further investigated in order to evaluate the true performance of the flowmeter on the road.
• The difference of the PN results can be high: partly due to PEMS uncertainties, but mainly due to lower PN concentrations reaching the dilution tunnel (diffusion losses, thermophoretic losses, coagulation etc.) [22].

As a final check, the consistency of the PEMS was evaluated by comparing the laboratory PEMS measuring simultaneously with the Golden PEMS, using their own flow meters, for both the WLTC and the RDE cycles. The results are plotted in Figure 3. The mean differences are small and similar between the two cycles. The good agreement of the Golden and Lab PEMS for PN, even though they consisted of different counting technologies (CPCs and DCs) indicates that the higher emissions at the tailpipe are true and not a calibration issue. Small differences between PN systems at the same location was expected for CPC based systems, but higher differences for DC-based systems have been seen [27], especially at lower emission levels. Some attention needs to be given to the 5 g/km mean difference of the CO₂ results. This difference can be partly explained by the general slight overestimation of Lab PEMS and a slight underestimation of the Golden PEMS (see Figure 2). A closer look at the data revealed that a big part of this difference came from one laboratory. Excluding this laboratory from the analysis, reduced the difference to <1 g/km with scatter (one standard deviation) of 3 g/km. Comparing the mean exhaust flow rates calculated by the PEMS revealed that the reason for the higher results of that laboratory was due to the exhaust flow meter. While the rest laboratories flow meters agreed within ±5% with the Golden flowmeter, that laboratory’s flow meter had 11% mean difference. As mentioned above and in the literature [26] the assessment of the exhaust flowmeters is important in future studies.

The consistent and small differences of the lab PEMS from the Golden PEMS indicates that conducting inter-laboratory exercises with a Golden PEMS but without involving chassis dynamometer testing is feasible because only relative differences have to be compared. This approach eliminates the variability of the vehicle. In addition, such a comparison covers also the environmental conditions, vibrations etc. that cannot be easily simulated in a laboratory. The practicality of such exercise needs to be evaluated though. For example, it would require the installation of the Golden PEMS inside the vehicle and the lab PEMS outside of the vehicle on the tow bar (hook). Fitting two flowmeters for conducting on-road tests is also very challenging. It would also be highly recommended to validate the PEMS at the beginning and at the end of the exercise to confirm that there was no drift. Finally, it has to be ensured that the Golden PEMS will be certified [6] and that the environmental conditions have minimum influence on it, even though the temperature and vibrations variations will be much smaller inside the car compared to outside. Alternatively, it could be installed in a conditioned trolley outside the vehicle.

5. Conclusions

An inter-laboratory exercise with PEMS in eight laboratories in Italy showed that the reproducibility of the PEMS methodology is at the same level (CO 20–25%, NO_x 23–31%) or only slightly higher (CO₂ 5.5% vs. 2.9%, PN 39% vs. 29%) compared to the laboratory legislated method. The reproducibility values of the Golden PEMS were at the same levels as the laboratory method. The higher scatter of the CO₂ measurements was attributed to the exhaust flow measurement, while the higher absolute levels of the PN were attributed to particle losses taking place from the vehicle until the dilution tunnel. These results confirm the permissible tolerances of the regulation. Additionally, they open the possibility of conducting inter-laboratory exercises only with PEMS, without the need of chassis dynamometers.