Preventing Wine Counterfeiting by Individual Cork Stopper Recognition Using Image Processing Technologies

Abstract

Wine counterfeiting is a major problem worldwide. Within this context, an approach to the problem of discerning original wine bottles from forged ones is the use of natural features present in the product, object and/or material (using it “as is”). The proposed application uses the cork stopper as a unique fingerprint, combined with state of the art image processing techniques to achieve individual object recognition and smartphones as the authentication equipment. The anti-counterfeiting scheme is divided into two phases: an enrollment phase, where every bottle is registered in a database using a photo of its cork stopper inside the bottle; and a verification phase, where an end-user/retailer captures a photo of the cork stopper using a regular smartphone, compares the photo with the previously-stored one and retrieves it if the wine bottle was previously registered. To evaluate the performance of the proposed application, two datasets of natural/agglomerate cork stoppers were built, totaling 1000 photos. The worst case results show a 100% precision ratio, an accuracy of 99.94% and a recall of 94.00%, using different smartphones. The perfect score in precision is a promising result, proving that this system can be applied to the prevention of wine counterfeiting and consumer/retailer security when purchasing a wine bottle.

1. Introduction

One major problem that modern society faces today is counterfeiting, reaching a global scale. The search for solutions to combat and prevent this issue is nothing recent, and the search for user-friendly low-cost solutions is growing. According to a report published by The European Union Intellectual Property Office in April 2016 [1], “the value of geographical indication infringing products in the EU was approximately €4.3 billion in 2014 (...)”. From

Table 1. Percentage of geographic indication product counterfeiting; adapted from [1].

Percentage of Counterfeited Products	Counterfeited Products
12.7%	Spirits
8.6%	Wines
0.1%	Beers
11.5%	Fruits, Vegetables and Cereals
11.0%	Fresh Meat and Meat Products
10.6%	Cheeses

, it is clear that spirits and wines are the largest portion of products affected by counterfeiting. Within this context, some technologies have been proposed in the literature and applied in industrial domains to prevent wine and spirit counterfeiting, such as RFIDs, NFCs, QR codes, barcodes, digital water marking, labels, holograms, tags, among others, and some companies are specialized in the development of new and hybrid strategies to make wine counterfeiting difficult. A common idea behind these approaches is the need to add a defining trait or symbol, usually in the package, to improve the consumer safety and increase the difficulty to counterfeit, while providing serialization for each bottle of wine. However, this added feature has some associated cost, increasing the price per bottle. This paradigm led to the following research question: “Is there a way to perform individual object recognition for every wine bottle without the need to add extra characteristics?”. By performing a product breakdown of the structure of a wine bottle, the following components are usually present: glass bottle, wine, cork stopper, capsule and printed labels.

Cork is a vegetal tissue extracted from the cork tree. This natural material possesses a collection of properties not found in any other natural or man-made materials: it is elastic, lightweight, impermeable, has good thermal and acoustic insulation, is resistant to attrition, as well as its texture displays random patterns [2]. The random patterns displayed by cork have the potential to provide a solution complying with the constraints mentioned before. Some issues need to be answered: Is it possible to use the visible natural features of the cork to perform individual object recognition? Is it possible to perform individual object recognition in a way that does not require adding extra components to the product while ensuring its authenticity? Is there a way to use the cork’s natural patterns combined with state of the art image processing technologies to perform individual object recognition? To answer these questions, the following structure for the methodology was used: (i) a literature review focusing on local feature detectors/descriptors and object fingerprinting; (ii) the design of a system architecture for wine anti-counterfeiting purposes; (iii) the design, development and testing of an artificial vision algorithm capable of cork individual object recognition; and (iv) characterization of the proposed system.

A new concept in the context of anti-counterfeiting systems called RIOTA (Recognition of Individual Objects using Tagless Approaches) is introduced in this work. RIOTA uses a non-invasive tagless approach, leading to individual object recognition for anti-counterfeiting purposes with zero added information and “zero added finishings”. The details of RIOTA are further explained in Section 4. The design and implementation of a novel wine anti-counterfeiting scheme based on the natural features of cork to attain individual object recognition, using state of the art image processing techniques that suit the RIOTA’s formalization are also presented in this article. This wine anti-counterfeiting method is composed of a two-stage process: (i) the enrollment phase, where a photo of every cork stopper, already placed inside the bottle (or the necessary info to perform individual object recognition) is registered in a database during the bottling process; and (ii) the verification phase, where a common user/retailer using a smartphone with a pre-installed app takes a photo of the cork inside the bottle; this photo (or the necessary info) is then queried in the database, and the relevant information is retrieved for the user. Therefore, the authors highlight two main contributions:

• RIOTA’s concept formalization in anti-counterfeiting systems;
• A wine anti-counterfeiting scheme based on natural patterns of cork stoppers.

2. Literature Review

This section presents a literature review targeting local feature detectors/extractors, a few examples of object fingerprinting and some technologies used for wine anti-counterfeiting. Global features (e.g., template matching) are discarded from this study due to known limitations, like intolerance to partial occlusion, computational cost (searching in the space of all possible correspondences), etc. Texture descriptors like Local Binary Patterns (LBP), Gabor filters or Gray Level Co-occurrence Matrices (GLCM) were not included because these kinds of techniques do not provide enough discrimination between the texture patterns present on the surface of the corks. This will be further discussed in Section 4.3 and Section 7. Machine learning techniques (e.g., Convolutional Neural Networks (CNNs) and Support Vector Machines (SVMs)) are also ignored, since these methods usually are computationally costly and arbitrarily complex; also, from the application point of view, training a classifier each time a photo is registered in a database is infeasible, making machine learning not suitable for this specific application.

2.1. Local Feature Detectors

A fairly known approach to object recognition is the use of feature detectors: techniques for the extraction of keypoint descriptors combined with a matching process. Over the years, some feature detectors have been proposed, namely: Feature from Accelerated Segment Test (FAST) [3], Scale-Invariant Feature Transform (SIFT) [4], Speeded Up Robust Features (SURF) [5], Good Features to Track (GFTT) [6], Oriented FAST and Rotated Binary Robust Independent Elementary Features (BRIEF) (ORB) [7], Binary Robust Scalable Keypoints (BRISK) [8], STAR and Maximally-Stable Extremal Regions (MSER) [9], Histogram of Oriented Gradients (HOG) [10] and Haar-like features [11], AKAZE [12], among various others. Focusing on the keypoint extraction task, some descriptor extraction procedures are also suggested, such as SIFT, SURF, HOG, AKAZE, Fast Retina Keypoint (FREAK) [13], BRIEF [14], ORB and BRISK. Finally, in order to recognize an object in an image, a correct match between the descriptors is needed. This matching stage can be performed in a heuristic way or by using brute force. A common implementation based on a heuristic approach is Fast Library for Approximate Nearest Neighbors (FLANN) [15]. This method employs several optimizations to improve its time performance; for example: appropriate selection of the keypoints to match and the usage of k-d trees to speed up the n-dimensional search. The brute force approach is computationally expensive, with each descriptor of the queried image being compared with all of the descriptors of the stored image in order to find a correspondence. Despite all of the research done in the field of object recognition, applications using individual object recognition come across the dilemma of recognizing and distinguish objects of the same class.

2.2. Object Fingerprinting

According to [16], “Fingerprinting an object is the ability to identify an individual object based on unique features such that if the object is seen by the system in other unfamiliar environments, or views, it can still be recognized”; in other words, it is the individual object identification in different scenes based on unique features. The problem addressed in the mentioned article is an object tracking system using a view-based classifier cascade that actively learns to recognize the generic class of the object, also using a fuzzy graph matching fingerprint representation. The results presented in this article showed that the system was capable of recognizing specific individuals in the selected class of objects.

A SIFT-based image fingerprinting approach was proposed in [17], by taking as a premise the fact that if the SIFT algorithm is a reliable choice to localize objects in an image, it also can be a promising candidate to analyze the image content and, therefore, determine if two images contain identical information. The SIFT keypoints and descriptors are extracted from the queried image and then are considered as a fingerprint of this image. Next, a best-bin-first matcher [18] is used to match the keypoints with the ones previously stored in a database. If the number of matched keypoints is larger than a threshold T, there is a possibility that the images are homologous, advancing to the next stage of the algorithm; otherwise, the two images are heterogeneous, ending the algorithm execution. Outliers are removed by assuming that keypoints generate a cluster, and the cluster’s centroid is calculated. To remove the outliers, the mean and the standard deviation distances are calculated; for each keypoint, if the distance is greater than or equal to the mean plus a standard deviation, then it is removed. The last step is detecting the homology. To prove the assumption, the authors tested their approach on a dataset composed of 3000 independent images selected from CD-ROMs “Art Explosion 800,000”. The range of selected images does not include several images of the same instance, meaning that the calculated fingerprint is not unique for two identical objects.

In [19], the authors propose an artificial vision method and an acquisition system to distinguish a bolt from a stack of bolts by taking advantage of the pear-skin finishing given to the bolts. The acquisition system is composed of a white diffuser made from translucent plastics, a black ring absorber and a macro lens. This setup prevents ambient light from falling on the metal surface, directly enabling repeatable image features to be captured from a common camera. The proposed system is highly dependent on the pear-skin finish given to the bolt, and the acquisition system was designed to take advantage of the combination of bolt plus finishing. Moreover, the authors state that their approach can be applied to tiny mass-produced metal parts with the same finishing. The impact of this research may prevent metal bolt counterfeiting and/or help metal bolt traceability.

2.3. Wine Anti-Counterfeiting Methods

Some technologies have been used to avoid the counterfeiting of wine bottles. One of them is the usage of RFID tags. Tran and Hong proposed an RFID-based anti-counterfeiting system [20] that combines two protocols: tag authentication and database correction. These authors claim that the tag authentication protocol increases the usability for the costumers by allowing them to authenticate RFIDs tags without the need for authenticating the reader and the server, while the database correction allows the RFID tag to be updated by the seller or the server. Thin Film Electronics ASA company (also known as “Thinfilm”) developed an NFC-based technology called OpenSense™ Technology [21]. The NFC tag has one particularity: it possesses a loop connected to the NFC tag, which allows the storage of two codes instead of one. The code read by the application when the loop is closed is different than the one read when it is opened. Since the loop is attached to the opening zone of the product when the user opens the product (i.e., the bottle), the loop is opened. This technology is used for anti-counterfeiting and tampering detection purposes. To prevent the counterfeiting problem, Krishna and Dugar centered their attention on the usage of QR codes [22]. The wine anti-counterfeiting is achieved with a one-time QR code scan methodology. This method works as follows: the QR code is registered in the database along with a field that indicates if the QR code as been read or not. The user receives a different message when trying to read the QR code for the first time, meaning that it is a genuine product, or when he/she tries to read the QR code of the n-th time, meaning that is a non-genuine product (or the code was scanned earlier). Another common approach is the usage of serial codes or tags. Prooftag™ is a technology company [23] that develops tracking and security solutions for documents and products. Regarding product anti-counterfeiting, this company combines the serial codes, tags and bubble dispersion, offering three solutions: The first one is BUBBLE TAG™ and consists of a transparent polymer displaying a random bubble pattern and a printed identifier on a tag. The bubbles appear in positions, shapes and sizes, from which it is not possible to build a matrix from the generated product. As a consequence, each bubble group is unique and impossible to replicate, even by Prooftag™. Taking the user perspective, the authentication is performed in three steps: (i) the user inserts the identifier from the BUBBLE TAG™ product with a web browser; (ii) the image with the bubble pattern corresponding to that key is retrieved; (iii) the user compares and validates visually if the bubble pattern returned is the same as the one he/she possesses. The second, Ramdot™, is based on the random optical dispersion of particles made by a process developed by Prooftag™ on their production equipment. During the production, the particles are dispersed in a random way, creating unique aggregations. The particles are created in different colors. This technique also uses the visual comparison of images made by the user. The last one, FIBERTAG™, has the same working principle as the previous ones, having as the object of analysis a random set of fibers printed on a tag. These tags are available for any kind of paper.

3. Research Questions, Goals and Proposal

From the literature review, it can be seen that several algorithms and strategies have been presented in the field of object recognition. The search, discovery and matching of relevant features leading to object recognition in a scene comprise a classification problem: To what class does this object belong? The topic of distinguishing objects of the same class using artificial vision methods is under-explored and just recently has become a topic of research interest [19]. Moreover, artificial vision applications using cork are oriented toward the visual inspection topic of the cork stopper for quality control purposes [24].

There are no wine anti-counterfeiting applications using no extra items to authenticate the wine bottles, and there are no artificial vision applications using cork stoppers as a way to authenticate the wine bottles. Being aware of this gap, some research questions arise, namely:

• How does one prevent wine counterfeiting?
• Are the natural patterns of the cork stoppers individually recognizable?
• Can a photo of the cork stopper be used to identify a wine bottle?
• Are photos of cork stoppers usable for anti-counterfeiting?

As such, two main objectives were formalized as follows:

• Proposing an anti-counterfeiting strategy;
• Proposing a methodology for individual cork stopper recognition.

To answer the research questions and fulfil the objectives, a wine anti-counterfeiting scheme is proposed in Section 4.2 along with a formalization of RIOTA (see Section 4) and a specific approach for individual cork stopper recognition (i.e., recognize and distinguish one cork stopper among several cork stoppers) (Section 4.3).

4. RIOTA

As mentioned in the literature review, most of the discussed methods employ some kind of added information and/or special treatment to provide some form of unique object recognition. For that reason, the authors have devised a non-invasive tagless approach, leading to individual object recognition for anti-counterfeiting purposes with zero added information and “zero added finishings”, called RIOTA (Recognition of Individual Objects using Tagless Approaches). In this section, all steps ranging from the RIOTA formulation to the implementation details are presented.

4.1. RIOTA Formulation

RIOTA uses the concept of “zero added finishings”: the absence of any type of surface finishing and/or coating added to a material or product beyond those already used in the manufacturing process that most likely would allow for individual object recognition. The usage of a tagless approach combined with zero added finishing will allow a cost reduction per manufactured unit (compared to tag approaches).

The idea behind RIOTA is to be a generalized application capable of recognizing and distinguishing individual objects between others of the same class. RIOTA possesses four important characteristics:

• Non-invasive property: in order to recognize and distinguish an object, the RIOTA system will not need to be in contact with that object;
• Tagless approach: the object does not include any kind of sensor and/or tag (e.g., RFID, NFC, etc.) to help with its recognition and/or its distinctiveness;
• Zero added info: the object does not need any kind of added information, such as serial numbers, barcodes, QR codes, etc.;
• Zero added finishing: the object to be recognized does not include any kind of special surface treatment, finishing and/or coating, visible or invisible to the naked eye, that could allow for individual object recognition.

To achieve the previously-mentioned goals, a wine anti-counterfeiting scheme and a methodology that suits the RIOTA formalization are described in the following subsections.

4.2. Wine Anti-Counterfeiting Scheme

The wine anti-counterfeiting scheme proposed in this work based on the natural features of cork stoppers is divided into two stages:

• enrollment phase: After the bottling process, an image is captured at the top of cork (Figure 1) and is registered in a database (only one photo per bottle is registered). The actual information recorded may be an image or all the necessary information that will allow individual object recognition. Additional details may also be recorded as needed, such as year of production, lot, etc.;
• verification phase: A common user, with a camera (e.g., smartphone), takes a picture of the product under verification; the captured photo or all the necessary information that will allow individual object recognition is uploaded to a server; then, the matching algorithm is employed, providing the appropriate information, which is then sent back to the user (see Figure 2), either notifying if the product is registered in the database or not (meaning genuine or not).

This scheme uses a photo of the top of the cork as the Region of Interest (RoI) to authenticate a wine bottle. The RoI detection and the matching algorithms are detailed in Section 4.3 and Section 4.4.

4.3. Procedure and Algorithm Design

As mentioned in the Introduction, cork possesses a collection of properties not found in other natural or man-made materials, thus presenting itself as an interesting object to be uniquely recognized. The tests were made using real natural/agglomerate cork stoppers used in wine bottles, exemplified in Figure 3. The steps followed to accomplish the results presented in Section 5 are described as follows:

• Randomly select cork samples;
• Construct the database by capturing one photo per object;
• Capture a set number of images per cork (one, two or three) at different rotations, positions and scales using four different cameras;
• Iterate every image captured in Step 3 and compare with all images present in the database;
• Store the result of the comparison for further analysis;
• Characterization of the proposed approach.

To carry out the necessary testing of the algorithm, two datasets were generated,

Table 2. Characterization of the natural cork stopper dataset.

	Natural Cork Database	Natural Test Set 1	Natural Test Set 2
Number of photos	100	200	300
Smartphone	Xperia Z1 Compact	Xperia Z1 Compact	Asus Zenfone 2
Camera Resolution	20.7 MP	20.7 MP	13.0 MP
Smartphone Release Date	January 2014	January 2014	March 2015

and

Table 3. Characterization of the agglomerate cork stopper dataset.

	Agglomerate Cork Database	Agglomerate Test Set 1	Agglomerate Test Set 2
Number of photos	100	100	200
Smartphone	Xperia Z3 Compact	Asus Zenfone 2	Asus Zenfone 5
Camera Resolution	20.7 MP	13.0 MP	8.0 MP
Smartphone Release Date	September 2014	March 2015	April 2014

. For the creation of those datasets, two databases were built: one populated by 100 photos representing 100 different natural cork stoppers and the other with 100 photos representing 100 different agglomerate cork stoppers.

The first dataset is comprised of two test sets of photos captured by two different cameras, taken from the natural cork stopper database, resulting in a total of 500 images; similarly, the second dataset is also comprised of two test sets of photos captured by two different cameras, taken from the agglomerate cork stopper database, resulting in a total of 300 images. Note that, since the photos of both cameras were taken manually, they have different scales/resolutions and rotations. Only one photo per cork is registered in the database. The diameter of the top of the corks used ranges from 19 mm–23.5 mm. For the datasets’ creation, no special attention was given to the resolution of the captured images. All captured images were scaled down to 640×H (wherein H is the corresponding height to keep the aspect ratio), and the images were compressed as JPEG. The photos were captured using a macro lens placed on the smartphone to enhance the details in the surface of the cork. The photos were captured by placing the smartphone at a distance of approximately 2 cm or 3 cm from the top surface of the cork stopper.

In order to evaluate the proposed approach, sensitivity and specificity tests were performed, and the precision, recall and accuracy ratios were calculated.

The algorithm that suits RIOTA’s formalization, presented in Figure 4, was designed based on the premise given in [17]: “Intuitively, since the SIFT algorithm is able to localize objects in an image, it can also help us to determine whether two images contain the identical content”. This is not exclusive to SIFT and can be considered also for other local features detectors/extractors like SURF, BRIEF or ORB.

Texture descriptors like GLCM and LBP were firstly considered as a hypothesis, but were discarded. Looking at the second row of Figure 3, it is possible to see that between corks, there are no significant differences in texture. In other words, to be able to recognize one cork stopper among several cork stoppers, texture descriptors do not provide enough discrimination.

This proposed algorithm can be divided into two phases: identification of the region of interest and matching of the detected RoIs. In this case, the RoI is bounded by a circumference, as can be seen in Figure 3. The identification step takes one RGB image, converts it to grey, then applies a Gaussian blur filter to reduce some acquisition noise; next, a Canny edge detector is applied to find the edge circumference. To detect the circumference, a least square fitting algorithm is applied [25], and both the radius and center of the circle are obtained. Finally, the RoI is delimited on the grey image, and four corners are painted in black (Figure 5), reducing the area to match (and the matching errors) for the next step.

The second phase matches the two RoIs obtained in the previous stage. As such, from the list of methods presented in Section 2, the approach uses the ORB detector, descriptor and brute force matcher to match the keypoints. The underlying details that the problem presents make ORB a prime choice over the other state of the art detectors/descriptors: invariance to rotation and translation is necessary, but scale invariance can be discarded, possibly improving time performance. Since ORB possesses these crucial characteristics and there is no need to use a richer feature detector/descriptor, this satisfies the problem’s requirements.

Next, the number of keypoints is compared with a predefined threshold $T1$. This is an intermediate step, included for time performance purposes; if there is not sufficient evidence that the images represent the same object (the number of keypoints is smaller than $T1$), then there is no need to continue the program execution, and it is assumed that the images represent two different objects, ending the program execution. Otherwise, it is possible that the two images represent the same object, continuing to the next step, entitled “validation gate”. This step tries to eliminate some wrongly-matched keypoints by calculating and comparing the relative distances between them (this will be further explained in greater detail in Section 4.4). Finally, in order to decide if the two test images belong to the same object, the number of keypoints detected is compared to a pre-defined threshold $T2$. If the number exceeds the threshold, the two photos taken belong to the same object; otherwise, the photos are from two different objects.

4.4. Validation Gate

This section explains and details two important tasks for the described implementation. After applying the ORB detector, descriptor and brute force matcher to get a correspondence in two images, a validation gate $v_g$ is applied to test if each matched keypoint is correctly matched. Let $I1$ and $I2$ be two vectors of keypoints of Image 1 (input RGB image) and Image 2 (queried database image), respectively. These vectors are sorted in ascending order of their distances (these being a measure of similarity; in this case, computed using the Hamming distance), so that the matches with the highest probability of being correctly matched come first. Each keypoint is represented by its position coordinates and orientation (Equations (1) and (2)). With this, it is possible to define a reference point in each image (Equations (3) and (4)) by taking advantage of the sorting done in these two vectors, assuming that $m_1$ is correctly matched with $m_1^{\prime }$.

$$\begin{array}{c}\hfill I_1=\{m_1,m_2,\dots m_m\},m_i=\{x_i,y_i,{\theta }_i\}i=1\dots m\end{array}$$

(1)

$$\begin{array}{c}\hfill I_2=\{m_1^{\prime },m_2^{\prime },\dots m_n^{\prime }\},m_j=\{x_j^{\prime },y_j^{\prime },{\theta }_j^{\prime }\}j=1\dots n\end{array}$$

(2)

$$\begin{array}{c}\hfill P_{I_1}=(x_1,y_1)\end{array}$$

(3)

$$\begin{array}{c}\hfill P_{I_2}=(x_1^{\prime },y_1^{\prime })\end{array}$$

(4)

Then, the spatial distance d of each keypoint to the reference point in each image can be defined as the Euclidean distance between them. The scale of the two images may differ, and to guarantee a fair comparison between $d_{I1}$ and $d_{I2}$, the Euclidean distance is corrected by a factor calculated from the detected radius, $r1$ and $r2$ (Equations (5) and (6)).

$$\begin{array}{c}\hfill d_{I_1}(m_i,P_{I_1})=\frac{\sqrt{{(x_i-x_1)}^2+{(y_i-y_1)}^2}}{r_1}\end{array}$$

(5)

$$\begin{array}{c}\hfill d_{I_2}(m_j,P_{I_2})=\frac{\sqrt{{(x_j-x_1^{\prime })}^2+{(y_j-y_1^{\prime })}^2}}{r_2}\end{array}$$

(6)

$$\begin{array}{c}\hfill v_g(d_{I_1},d_{I_2})=|d_{I_1}---d_{I_2}|\le r0\end{array}$$

(7)

$$\begin{array}{c}\hfill r0=k\sqrt{2}\end{array}$$

(8)

Finally, to reject or accept a matched pair, the absolute value of the difference of $d_{I1}$ and $d_{I2}$ is calculated. Ideally, this value should be equal to zero. However, due to unavoidable errors during the acquisition process, like distortions caused by the lens of the devices, a tolerance box $r0$ is defined. This tolerance eliminates wrongly-matched point pairs. Using the previously-mentioned assumption, $m_1$ corresponds to $m_1^{\prime }$, the absolute value of the difference of the relative distances of a correctly-matched point should be approximately equal to zero. In the case of an incorrectly-matched keypoint, this value will be greater than zero. Obviously, a tolerance is needed, represented in this work as $r0$, defined as the distance in pixels from which a matched pair point is considered valid. This distance is defined in Equation (8), wherein the term $\sqrt{2}$ represents the largest Euclidean distance of two adjacent pixels and k is a real constant, greater than one. The value of k was tuned manually, resulting in $k=1.5$.

5. Results

This section presents examples of the application of the proposed approach. Representative results and critical analysis of the validation gate are also presented.

Table 4. Ratio results on the natural cork dataset.

Test Performed	Precision	Recall	Accuracy
Natural Cork Database with Test Set 1	100.0%	100.0%	100.0%
Natural Cork Database with Test Set 2	100.0%	94.33%	99.94%

and

Table 5. Ratio results on the agglomerate cork dataset.

Test Performed	Precision	Recall	Accuracy
Agglomerate Cork Database with Test Set 1	100.0%	100.0%	100.0%
Agglomerate Cork Database with Test Set 2	100.0%	94.00%	99.94%

present the comparison results between the database and the test sets, both in natural and agglomerate cork, respectively. The ratio results presented in all tables (Table 4, Table 5 and

Table 6. True negative ratio results of the inter-dataset tests.

Test Performed	True Negative Rate
Natural Cork Database with Agglomerate Cork Test Set 1	100.0%
Natural Cork Database with Agglomerate Cork Test Set 2	100.0%
Agglomerate Cork Database with Natural Cork Test Set 1	100.0%
Agglomerate Cork Database with Natural Cork Test Set 2	100.0%

) were calculated using Equation (9).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Precision(\%)=\frac{TP}{TP+FP}\times 100\\ \hfill Recall(\%)=\frac{TP}{TP+FN}\times 100\\ \hfill TrueNegativeRate(\%)=\frac{TN}{TN+FP}\times 100\\ \hfill Accuracy(\%)=\frac{TP+TN}{TP+TN+FP+FN}\times 100\end{array}\end{array}$$

(9)

Another test performed includes the comparison between the database of natural corks with the two test sets collected for the agglomerate corks and the comparison between the database of agglomerate corks with the two test sets collected for the natural corks. Table 6 presents the results obtained for this test. This was performed to retrieve feedback about how reliable the proposed approach is at uniquely identifying a cork among the others. Critical analysis of the validation gate, circle detection and the overall approach is discussed in Section 6.

Figure 5 shows the results of the usage of the validation gate. The cork on the right has been rotated along with the keypoints to help with reading. It can be verified that some mismatched keypoints were removed, which improved the confidence level of the proposed algorithm.

6. Discussion

The recall rates shown in Table 4 and Table 5 are not perfect. By analyzing each resulting false negative, it can be concluded that the causes of these ratios originate from the RoI detection, perspective changes, blurred photos, variations in illumination, etc. Inspecting the natural cork tests individually, the percentage of the false negatives due to the wrong RoI detection is 58.82%. Regarding the agglomerate cork false negatives, 50% of them are due to the wrong RoI detection.

The RoI detection is based on two major steps (Figure 4): (i) segmentation (composed by the conversion from the RGB color space to a grayscale intensity image (RGB2Gray), the application of a Gaussian filter to remove noise (GaussianBlur Filter) and a Canny edge detector); and (ii) circle detection (using least square fitting to estimate the circle). Performing a more thorough analysis, the RoI detection problems are always localized in the segmentation stage; the proposed segmentation fails at correctly detecting the cork area, and the problem lies in the chosen Canny edge detector fixed minimum and maximum thresholds. Due to the normalization in the validation gate (Equations (5) and (6)), an incorrect segmentation will increase the odds of accepting a false negative result.

With an improvement/change of the algorithm’s segmentation stage, the recall rate would most likely rise.

Regarding the validation gate, its application proves to be beneficial, improving the matching results by up to 20%. It also made possible the 100% precision rate in all tests, including the comparison between the natural cork test sets and the agglomerate cork test sets, which in the context of an anti-counterfeiting system looks promising.

7. Research Directions and Applicability Issues

This work presented an image-based approach to handle the wine counterfeiting problem. Naturally, from the application point of view and due to its novelty, some issues may arise. This section serve to discuss some of these issues.

The first issue that can be identified comes with the proposed anti-counterfeiting scheme, requiring that the query image be compared with all images in the database. With an eventually ever-growing database, this is not a viable solution. An alternative is to combine the information already used in the label of the bottle like the lot number or the serial-key (some expensive wine bottles use a unique serial number). This solution, despite being pragmatic, makes it necessary for the user to accomplish two tasks: (i) insert a number and (ii) take a photo of the cork. This could prove to be inconvenient, from the user point of view. To overcome this usability issue, an automatic solution to query the image in the database using the information of the photo taken will reduce the search space, thus improving the time performance, and this is a more pleasant solution for the end-user. Some technologies naturally emerge from this context, namely: image hashing and content-based image retrieval systems. In the near future, this topic will be addressed.

Another relevant discussion is the viability of the proposed method for wine anti-counterfeiting. A relevant question to answer is: How easy or difficult is it to deceive this system? Since the method relies on the existing visual aspects present on the surface of the cork, there are two main ways to cheat the system: (i) find an instance that possesses the same visual characteristics as a single cork (e.g., a printed photo of a genuine cork), which responds in the same way as a previously-registered cork stopper; or (ii) reuse a genuine cork to circumvent the system (e.g., remove the cork stopper and reinsert it, into the same bottle or another one). Generally, the attack presented in (i) is described in the literature as spoofing attacks, and these are commonly found in face biometrics systems. Since the approach in this work relies on the matching of feature descriptors, there is a strong hypothesis that the system would recognize a print of a high-definition photo of a cork as a genuine photo of a cork stopper. From the user perspective, this does not present itself as a limitation; it is trivial to distinguish a printed paper from a real cork stopper without any specialized equipment. However, taking into account an autonomous system, completely devoid of human input, this presents itself as an issue that needs to be addressed. Therefore, a closer look as the face anti-spoofing biometric systems that do not use face features (i.e., eye blinks, mouth movements) will most likely benefit the proposed anti-counterfeiting scheme, making it more robust to attacks. Regarding the counterfeit attacks described in (ii), some details need to be highlighted. For a counterfeiter to be able to reuse a cork stopper, the following steps must be executed: 1, remove the capsule; 2, remove the cork stopper; 3, insert the cork stopper in a bottle (the same or another bottle); 4, insert an equal capsule; 5, encapsulate the bottle (compress the capsule mechanically in the case of an aluminum capsule or by heat in the case of a PVC capsule). A particular, important detail to note is that it is mandatory for all wine bottles to be encapsulated, due to legal reasons. Thus, for a counterfeiter to succeed, he/she must have access to the capsules (the same as those intended to be used in the bottle to be counterfeited) and find a way to fasten the capsules on those bottles. This, perhaps, is the simpler step in this counterfeit chain. For Step 2, the removal of the cork stopper needs to be done by meeting one simple requirement: not damaging or altering the visual characteristics of the top surface of the cork stopper. This condition must be also followed in Step 3, when inserting the cork stopper into another bottle. Moreover, the initial relative distance between the cork and the bottleneck must be kept when reinserting the cork stopper into the bottle, in order to minimize the visual changes of the pair cork stopper/capsule. Albeit that this is unquestionably difficult, it is plausible. Notice that, between Steps 2 and 3, there are some relevant elements that require a detailed analysis. As stated before, cork is a vegetal tissue that possesses, among others, an elastic property. In other words, it is capable of being compressed to fit inside the bottle and, depending on the elastic domain of the cork and conditions, recovering to the initial state. Other important notion to understand is the time frame between the removal of the cork stopper and the insertion of the same cork stopper. The longer this process takes, the more likely it is for the cork stopper to achieve its equilibrium state. At the same time, the longer the counterfeiter takes, the more unlikely it is for the visual properties to remain the same (remember that the method requires a macro lens). Combining all the variables that a counterfeiter needs to manage, one conclusion can be drawn: it is a hard process to handle, with many assumptions. Note also that, from the counterfeiter point of view, the product’s counterfeiting is only feasible when the resources spent are lower than the profits achieved, and considering all the above-mentioned variables, it is hard to predict and discuss if the obtained profits justify the wine counterfeiting.

Taking this discussion even further, let us assume the worst case scenario: a counterfeiter can reuse the same cork stopper and trick this system. There is no evidence that the same process will work in another bottle with the same success rate. The reason to support this relies on the methodology employed in this anti-counterfeiting method, which is individual object recognition. The usage of minutiae/details to achieve individual object recognition is advantageous in the context of wine anti-counterfeiting. This fact overcomes the expected limitation of using general features of the cork for wine anti-counterfeiting purposes. Once the forger discovers a way to circumvent the system and counterfeit one bottle, he/she most likely has found a generalized way to delude this system.

Finally, another issue that must be taken into account is the requirement of capturing a photo by the user. As mentioned in the Discussion section, Section 6, this photo must not be blurred. Otherwise, it may compromise the result. From this problem, these questions arise: How easy is it to take a good picture? How does one measure the easiness? Arguably, some assistive indicator can be provided to the user by the smartphone application (e.g., draw a circle in the smartphone camera app to help the user point at the RoI, in the same way as the square used to read the QR code; automatically measure the sharpness/blurriness and take the photo; warn the user when a photo is of poor quality; etc.). Even so, more conclusions about this issue will only appear after testing this method in an industrial setup and testing using “personas” or real end-users.

8. Conclusions

A wine anti-counterfeiting scheme based on natural features/patterns of cork stoppers combined with image processing techniques was proposed in this article.

The anti-counterfeiting is achieved using a two-phase scheme: (i) during the bottling process, every wine bottle is registered in a database using a photo (or the necessary info to allow object recognition) of the top of the cork already inside of the bottle; (ii) after that, an end-user/retailer verifies the authenticity of a wine bottle by capturing a photo at the top of the cork using a smartphone with a pre-installed app, and the according information is retrieved for the user.

The first step of RIOTA’s formulation was presented and discussed, and given its characteristics, it looks to be a promising approach, possessing non-invasive properties, zero added info, a tagless approach and zero added finishing. RIOTA has advantages over other implementations [19], not requiring any kind of finishing, thus not increasing the cost per manufactured product while maintaining its purpose: the prevention of the product’s counterfeiting.

The achieved results prove that natural patterns of the cork stoppers, being irreplaceable and intricate, make cork stoppers individually recognizable. Thus, they can be used to uniquely identify a wine bottle. The 100% precision ratio both in natural and agglomerate cork using four different cameras and the accuracy results obtained, ranging from 99.94%–100.0%, prove that the proposed approach for individual cork stopper recognition using ORB combined with a brute force matcher and validation gate to eliminate outliers can be robust enough in security systems for anti-counterfeiting and/or traceability purposes. However, the recall rate points to improvements in the segmentation phase, because it is not desirable for a common user to take more than one photo of an authentic product in order to recognize it. The usage of a local feature detector provides more tolerance to noise in existing features (e.g., some small risk), enabling more resistance to noise and tolerance to partial occlusion.

The two main goals of this work have been achieved, and the research questions have been answered; yet, within this application, some new research questions may arise: Is it possible to measure the object’s richness taking into account previous knowledge? Is it possible to “hack” the proposed approach by taking a photo of a high quality printed photo of a cork? Future work includes answering these questions.

9. Patents

The system presented is expected to be used in the consumer market in the near future; as such, a pending patent process related to this work is in progress.