Next Article in Journal
Decoding Strategies for Improving Low-Resource Machine Translation
Next Article in Special Issue
Reduced Tilting Effect of Smartphone CMOS Image Sensor in Visible Light Indoor Positioning
Previous Article in Journal
Multi-Chip IGBT Module Failure Monitoring Based on Module Transconductance with Temperature Calibration
Previous Article in Special Issue
Optical Camera Communications: Principles, Modulations, Potential and Challenges
 
 
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Design of Streaming Data Transmission Using Rolling Shutter Camera-Based Optical Camera Communications

1
Department of Information and Communication Engineering, Changwon National University, Changwon 51140, Korea
2
Sensor Research Team, Gyeongbuk Institute of IT Convergence Industry Technology, Gyeongsan 38541, Korea
3
Department of Electronic Engineering, Yeungnam University, Gyeongsan 38541, Korea
*
Author to whom correspondence should be addressed.
Electronics 2020, 9(10), 1561; https://doi.org/10.3390/electronics9101561
Received: 4 August 2020 / Revised: 19 September 2020 / Accepted: 22 September 2020 / Published: 24 September 2020
(This article belongs to the Special Issue New Challenges in Wireless and Free Space Optical Communications)

Abstract

:
This paper addresses a method to transmit streaming data via rolling shutter camera-based optical camera communications (OCC). Since the amount of data that can be contained within one frame is limited, and the continuity of received data cannot be guaranteed due to OCC environmental variations, we introduce the concept of dividing the streaming data into several fragmented sets that are transmitted sequentially. We propose a superframe to contain sequential packets of fragmented data and corresponding indices, so that sequential packets for streaming data can be continuously collected. When redundant frame transmission is considered, any packet lost due to OCC environmental conditions can be recovered. Experimental results show that the proposed method can be successfully used to transmit streaming data, with the number of redundant frames required to acquire all data packets based on image resolution. In addition, we describe how to identify missing packets from a network point of view to reduce the number of redundant frames needed to acquire all the data. This paper presents baseline results of communication performance when sending streaming data via a rolling shutter-based OCC.

1. Introduction

Recently, a lot of research related to optical camera communications (OCC), which is a subset of visible light communications (VLC), has been published by industry and academia [1,2,3,4,5]. A typical digital camera consisting of a lens and a two-dimensional image sensor is capable of receiving optical data and is more beneficial than a photo detector (PD) by providing spatial separation of different light sources from the optical lens. Thanks to widespread LED light sources and digital cameras, OCC technology can to be applied in various fields, such as vehicle-to-everything (V2X) [3], indoor communication and positioning [4,5], artificial reality (AR), and display field communications [6]. In addition, OCC can be used in intelligent services, such as unmanned factory automation and security systems in combination with artificial intelligence (AI) technology, and can be considered a key technology for implementing massive machine-type communications (mMTC), which will provide connections to large numbers of devices for 5G networks and beyond in the near future.
Among the non-flicker modulation series of OCC [7], the rolling shutter-based method [8,9,10,11] focuses on scenarios where the frame rate of a complementary metal oxide semiconductor (CMOS) image sensor is slower than the flickering speed of the light source that transmits the data. By turning an LED on and off frequently enough, data modulation is not perceived by the human eye. At the camera receiver, the information sent via the visible light spectrum appears in the form of a barcode on the captured image, and the patterns of the barcode contain the transmitted data. Recently, Teli et al. [12] carried out an experimental investigation of an indoor OCC system by employing a multi-channel transmitter and a Raspberry Pi camera as the Rx for IoT applications. Nguyen et al. [13] proposed a rolling shutter-based OFDM OCC system with multiple LEDs to improve the data rate performance. Eso et al. [14] increases the link span of a rolling shutter-based OCC system up to 400 m by reducing the spatial bandwidth of the camera. Liu et al. [15] introduces existing and new solutions to deal with the constraints of the OCC link with regard to data modulation, newly developed camera structures, and post-processing of sensed signals.
However, the rolling shutter-based OCC method inherently has two important constraints. First, for the receiving camera to fully utilize the rolling shutter effect, the distance between the transmitter and the receiver end must be short, and the pixel area of the light source must be sufficiently large in the received image. Second, when a continuous stream of data is sent, it is challenging for the receiver to acquire the streamed data as a single received image. In other words, the amount of continuous data sent by the transmitter and expressed in one image frame as a rolling shutter pattern is strictly limited [5]. Although streaming data delivery requires continuous reception, some missing data exist between temporally adjacent images, which results in distortions of the received information. Therefore, the application of transmitting stream data via a rolling shutter OCC link should consider how to utilize a continuous data reception mechanism. To the best of our knowledge, a solution for recovering all continuous data using rolling shutter-based OCC has not been studied in the literature.
This paper addresses the design of a streaming data communication mechanism using rolling shutter-based OCC. To support the continuity of the received data, we propose a superframe structure consisting of sequential packets of consecutive fragmented data and corresponding indices. To enhance the capability to recover lost packets, duplicate frames are transmitted, similar to the repetition coding scheme used in conventional communication systems. Therefore, even though part of the segmented packets may be lost or damaged while receiving a rolling shutter image, all the continuous data can be recovered through reception packet recombination using an index and duplicate superframes. The experimental results show that a data-streaming service can be provided through an OCC link with the help of the proposed superframe structure and its duplicate transmission mechanism. We presented the number of required duplicates based on the image resolution and the optical rate of the image. We believe this method provides a solution, allowing users to access smart content immediately using an OCC link, and the experimental guidelines can be used to design transmission and reception processes in implementing rolling shutter-based streaming data communication.

2. Superframe Structure for Streaming Data Communication

The system model for implementing the proposed streaming data transmission is an OCC downlink. When continuous data are transmitted by an LED light source in the form of visible light, they are broadcast to each OCC receiver equipped with a camera. Considering the CMOS image sensor at the receiving end, the rolling shutter effect appears as an on/off pattern and is analyzed to demodulate the signal. Due to the nature of rolling shutter-based OCC, the amount of data obtained in one received image contains a portion of the streaming data. Therefore, if the packet size of streaming data is too large to be received in one image, data must be aggregated using multiple images. A simple combination of consecutive images does not construct all the stream data owing to the OCC environmental characteristics, such as readout time delay, delay time variation between frames, etc. Therefore, in order to receive and collect all streaming data using a rolling shutter camera, transmitted data need to be fragmented and transmitted in small units.
Figure 1 presents the proposed superframe structure consisting of small consecutive packets. Note that the term superframe is used in different purposes in the 802.15.7 standard [16] and in this paper. While superframes in the standard are defined based on media access control (MAC) protocols such as contention access period (CAP), contention free period (CFP), etc., the superframe used in this paper is designed entirely for streaming data transmission by introducing indexing concept of fragmented packets. Each packet, Xi, has an associated preamble and the Xi-th fragmented data. The preamble contains synchronization bits and a packet index for Xi. Transmitting fragmented packet index is very important for recognizing the missing packets during the transmission of streaming data in OCC and make it possible to recover the missing packets from redundant (retransmitted) frames.
Figure 2 illustrates the process of data fragmentation and indexing. The original streaming data with k bits are fragmented into N small data units, bgn, where n is the allocated bits per group, and N is the number of groups (or packets), such that 1nN. Then, the indexing process combines the parts for timing synchronization, and the index indicates the sequence number of the corresponding packets. By combining fragmented data, bgn, and an indexing packet, all N data packets are constructed. Then, consecutive data packets constitute a superframe, and data packets are sequentially transmitted via the LED. By doing this, it is possible to find a packet that gets lost or damaged in the middle of streaming data reception via an OCC link.
Figure 3 shows how to distinguish lost data packets when sending indexed data packets in superframe units. It is assumed that six packets can be captured in one image and received periodically according to the frame rate of the camera. If we assume that the streaming data to be transmitted comprises 20 packets, the index of lost packets forming the first superframe includes 7, 8, 9, 16, 17, and 18. After receiving the second duplicate superframe, the remaining lost packets are 7 and 16. When several duplicate superframes can be received consecutively, it is possible to recover all packets to construct the original superframe.
Due to interference or a noisy optical wireless channel and the capturing mechanism of the receiving camera, packets are lost with a certain probability. The proposed superframe structure is partitioned into multiple packets, and enables appropriate action to recover a loss. Using the packet index included in a data packet, the receiver can successfully recover lost packets by reconnecting to a streaming data set. In addition, the proposed scheme allows the OCC receiver to start capturing images from the transmitting LED starting at any time and smartly decoding them to recover all the streaming data, even in the presence of jitter in the frame rate [2].

3. Data Detection

Because the optical rate of an LED is much faster than the shutter speed of an image sensor, the LED changes its state multiple times during the image capture time. Therefore, the on/off states of the LED source are captured and shown as rolling shutter patterns in an image. Then, the transmitted data can be retrieved via the image processing and data decoding technique. Depending on the modulation scheme, one data bit can be presented by one or multiple light states. To mitigate the flicker effect of the LED transmitter, we use Manchester encoding for data modulation.
Although the commercial camera is assumed to operate at a fixed frame rate, such as 30 fps, unpredicted variations in the frame rate exist in practical operation. In between two neighboring image frames, the LED still blinks, but the camera cannot record it. Consequently, a data packet might be lost from the streaming data.
From a received image containing a pulse profile, the rolling shutter patterns can be converted to a binary array from which useful information can be extracted. As shown in Figure 4, the on/off patterns in the rolling shutter image present transmitted pulse profiles. The duration of the on/off patterns is presented as the pixel width of pulses, and is related to the optical rate of the transmitting LED and the camera capture rate. By adjusting these values, an array of images containing on/off patterns of different durations and intensities can be obtained. Using a predetermined threshold, pulse profiles are binarized into binary sequences.
As shown in Figure 5, each on/off pattern of binary sequences is composed of several pixel values. Because the camera cannot capture the transition strictly, a single pulse often occupies different pixel numbers. However, by measuring the average pixel width of on/off patterns, binary sequences are converted into pulse sequences. Considering Manchester encoding on the transmitter side, we can demodulate the bit data from the pulse sequence by comparing two consecutive pulse states. Additionally, it is also possible to demodulate data from binary sequences by using on/off transition patterns of bit sequences [5]. After the demodulation procedure is completed for each rolling shutter image, all demodulated data are merged to form the data as a whole. Then, if there are packets with lost data, we recover the transmitted streaming data by following the procedure explained in Section 2.

4. Simulation Results and Discussion

4.1. Simulation Results

A feasibility test of the streaming data transmission scheme was conducted in an indoor environment. In order to investigate the performance according to the length of the fragmented packet size, we use the streaming data of length 26 (for short data case) consisting of ASCII alphabet characters from A to Z. In addition, we also use the length 52 streaming data (for long data cases), including both lower- and upper-case letters. Based on the results of actual experiments in these two cases, the relationship between the number of packets and the number of additional transmissions can be verified. By transmitting streaming data composed of 52 and 26 fragmented data packets and by capturing a rolling shutter image, we measured the number of frames required to demodulate all the fragmented data. For the OCC-based experiment, a commercial LED luminary with an Atmega8A microcontroller was used to transmit visible light data at a 10 kHz optical rate. A 30-fps camera in a smartphone was used as a data receiver, and the sizes for captured images were set to 3264 × 2448 and 1600 × 1200. The images were taken at line-of-sight (LOS) distances, centered on the LED lights. In OCC, the Tx-Rx distance affects the resolution of the received image frame. Moreover, if we use the ‘zoom’ function of the camera, this affects the resolution of the image as well. This means that it is reasonable to consider the image resolution instead of the Tx-Rx link distance in evaluating the performance of OCC. To minimize the light blur effect in the rolling shutter image, the camera ISO, which amplifies or decreases the image sensors’ ability to read and gather light during an exposure, was set to 800. The fragmented data packets consisted of 20 bits with synchronization bits (6 bits), index bits (6 bits), and data bits (8 bits).
Table 1 shows the experimental results of receiving 52 fragmented data packets obtained from a rolling shutter image. Generally, the larger the number of pixels in the vertical direction, the greater the number of data packets included in one image frame. When we had a resolution of 3264 × 2448, we got 8.78 packets and 193.99 bits per image, whereas we got 5.58 packets and 130.61 bits at 1600 × 1200. The number of image frames required to receive all 52 pieces of streaming data when sequentially transmitting 52 pieces of fragmented data was 12 frames for 3264 × 2488 and 11 frames for 1600 × 1200. Table 2 shows the results when sending 26 fragmented data packets, considering short streaming data. Since the total number of packets in the streaming data was halved, all the data were received within seven consecutive frames. From the experiments, we can observe that the number of redundant frame transmissions required to receive the entire streaming data is not significantly related to the camera resolution. Therefore, the transmitting end can design a duplicate frame transmission method without seriously considering the performance and specifications of the camera at the receiving end.
Figure 6 presents the average number of missing packets based on image resolution and the length of fragmented packets in the streaming data. When the number of packets was relatively large and a high-resolution image was used, we can see that the number of missing packets was drastically reduced in the consecutive received frames. However, if the number of packets was not large, the number of missing packets was consistently reduced, regardless of the image size. In addition, we can find that the average number of missing packets decreases as the number of redundant (or retransmitting) frames increases, and is saturated after enough redundant frames are transmitted. Note that this tendency is consistent regardless of image resolution and packet size. At the beginning, when redundant transmission is attempted, the probability of including the missing packet in the redundant frame gradually decreases. If more than a certain number of packets are received, the probability of including the missing packets in the redundant frame is low, so the reduction slope becomes gentle.
Note that the main purpose of this paper is to propose how to recover the missing packets of streaming data caused by frame-to-frame discontinuity property of OCC and investigate the feasibility of the proposed scheme. Therefore, data rate analysis according to the specifications of the transmitter and receiver of OCC are reserved for a future task.

4.2. Discussions

As shown in Figure 3 and Table 1 and Table 2, it is necessary to transmit duplicate frames to recover missing packets so all the data can be demodulated. Because duplicate frames should be transmitted, it is inefficient in terms of transmission rate. This problem can be solved efficiently by adopting the automatic repeat request (ARQ) concept. As shown in Figure 7: (1) an acknowledge signal containing a request to turn the LED on is transmitted from the receiver to the transmitter; (2) the transmitter begins sending information on ready status to communicate using the LED; (3) an acknowledge signal containing a request to send packets is transmitted from the receiver to the transmitter; (4) the transmitter begins sending streaming data and, afterwards, the index of lost packets is determined at the receiver; (5) the index information is sent to the transmitter; (6) the transmitter collects only the lost packets and transmits them. Note that this procedure significantly reduces the number of additional frames required to obtain complete information. For example, considering the rolling shutter image at 3264 × 2448 in Table 1 and Table 2 above, this ARQ approach requires, at most, two additional frames to demodulate all data. Therefore, this scheme ensures reliable data transmission while reducing the additional transmission overhead required to recover the lost packets.

5. Conclusions

This paper proposed a method to solve the packet loss problem when transmitting streaming data in rolling shutter-based OCC systems. Owing to OCC-based parameters such as the optical rate of the light source, the frame rate of the receiver, and camera performance, a single received frame does not contain all the streaming data because some of the data between the neighboring frames can be missed. By proposing a superframe structure consisting of index information and fragmented data packets, all data can be obtained from consecutive frames. Through experiments, we demonstrated the transmission of duplicate superframes that are required to obtain all the streaming data, based on image resolution and the optical rate. In addition, we suggested an ARQ type approach that can solve the packet loss problem efficiently with fewer duplicate frames.
The main purpose of this paper is to propose how to recover the missing packets of streaming data caused by the frame-to-frame discontinuity property of OCC and investigate the feasibility of the proposed scheme. Therefore, an analysis of the maximum data rate regarding the high-bandwidth LED transmitter and high-resolution camera receiver deviates from the scope of this paper. Data rate analysis, according to the specifications of the transmitter and receiver of OCC, can be considered as a future task. In addition, we will perform an evaluation of the overall network performance of the proposed scheme, considering the ARQ approach in transmitting the missing packets, in a future work.

Author Contributions

Conceptualization, S.-Y.J.; methodology, S.-Y.J., J.-H.Y.; validation, J.-H.Y., B.W.K.; writing—original draft preparation, B.W.K.; writing—review and editing, S.-Y.J., J.-H.Y. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the 2019 Yeungnam University Research Grant.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

References

  1. Teli, S.; Cahyadi, W.A.; Chung, Y.H. Optical Camera Communication: Motion over Camera. IEEE Commun. Mag. 2017, 55, 156–162. [Google Scholar] [CrossRef]
  2. Kim, B.W.; Jung, S.Y. Flicker-Free Optical Camera Communications Based on Compressed Sensing. IEEE Commun. Lett. 2016, 20, 1104–1107. [Google Scholar] [CrossRef]
  3. Yamazato, T.; Kawagita, N.; Okada, H.; Fujii, T.; Yendo, T.; Arai, S.; Kamakura, K. The Uplink Visible Light Communication Beacon System for Universal Traffic Management. IEEE Access 2017, 5, 22282–22290. [Google Scholar] [CrossRef]
  4. Takai, I.; Harada, T.; Andoh, M.; Yasutomi, K.; Kagawa, K.; Kawahito, S. Optical vehicle-to-vehicle communication system using LED transmitter and camera receiver. IEEE Photonics J. 2014, 6, 1–14. [Google Scholar] [CrossRef]
  5. Kim, B.W.; Jung, S.Y. Transition-based Data Decoding for Optical Camera Communications using a Rolling Shutter Camera. Curr. Opt. Photonics 2018, 2, 422–430. [Google Scholar]
  6. Kim, B.W.; Jung, S.Y. Display Field Communication: Fundamental Design and Performance Analysis. J. Lightwave Technol. 2015, 33, 5269–5277. [Google Scholar] [CrossRef]
  7. Nguyen, T.; Islam, A.; Hossan, T.; Jang, Y.M. Current Status and Performance Analysis of Optical Camera Communication Technologies for 5G Networks. IEEE Access 2017, 5, 4574–4594. [Google Scholar] [CrossRef]
  8. Danakis, C.; Afgani, M.; Povey, G.; Underwood, I.; Haas, H. Using a CMOS camera sensor for visible light communication. In Proceedings of the IEEE Globecom Workshops, Anaheim, CA, USA, 3–7 December 2012; pp. 1244–1248. [Google Scholar]
  9. Chow, C.W.; Li, Z.Q.; Chuang, Y.C.; Liao, X.L.; Lin, K.H.; Chen, Y.Y. Decoding CMOS Rolling-Shutter Pattern in Translational or Rotational Motions for VLC. IEEE Photon. J. 2019, 11, 1–5. [Google Scholar] [CrossRef]
  10. Chen, C.W.; Chow, C.W.; Liu, Y.; Yeh, C.H. Efficient demodulation scheme for rolling-shutter-patterning of CMOS image sensor based visible light communications. Opt. Exp. 2017, 25, 24362–24367. [Google Scholar] [CrossRef] [PubMed]
  11. Ji, P.; Tsai, H.; Wang, C.; Liu, F. Vehicular Visible Light Communications with LED Taillight and Rolling Shutter Camera. In Proceedings of the 2014 IEEE 79th Vehicular Technology Conference (VTC Spring), Seoul, Korea, 18–21 May 2014; pp. 1–6. [Google Scholar]
  12. Teli, S.R.; Matus, V.; Zvanovec, S.; Perez-Jimenez, R.; Vitek, S.; Ghassemlooy, Z. Optical Camera Communications for IoT–Rolling-Shutter Based MIMO Scheme with Grouped LED Array Transmitter. Sensors 2020, 20, 3361. [Google Scholar] [CrossRef] [PubMed]
  13. Nguyen, H.; Pham, T.L.; Nguyen, H.; Nguyen, V.H.; Jang, Y.M. Rolling MIMO-OFDM for Optical Camera Communication System. In Proceedings of the 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), Fukuoka, Japan, 19–21 February 2020; pp. 103–106. [Google Scholar]
  14. Eso, E.; Teli, S.; Hassan, N.B.; Vitek, S.; Ghassemlooy, Z.; Zvanovec, S. 400 m rolling-shutter-based optical camera communications link. Opt. Lett. 2020, 45, 1059–1062. [Google Scholar] [CrossRef] [PubMed]
  15. Liu, W.; Xu, Z. Some practical constraints and solutions for optical camera communication. Philos. Trans. R. Soc. A 2020, 378. [Google Scholar] [CrossRef] [PubMed][Green Version]
  16. IEEE Standard for Local and Metropolitan Area Networks—Part 15.7: Short-Range Optical Wireless Communications. Available online: https://ieeexplore.ieee.org/document/8697198 (accessed on 23 April 2019).
Figure 1. Superframe structure for transmitting streaming data.
Figure 1. Superframe structure for transmitting streaming data.
Electronics 09 01561 g001
Figure 2. Data fragmentation and indexing structure.
Figure 2. Data fragmentation and indexing structure.
Electronics 09 01561 g002
Figure 3. An example of receiving fragmented data using a rolling shutter-based optical camera communications (OCC) receiver.
Figure 3. An example of receiving fragmented data using a rolling shutter-based optical camera communications (OCC) receiver.
Electronics 09 01561 g003
Figure 4. Pulse profile acquisition and binarization.
Figure 4. Pulse profile acquisition and binarization.
Electronics 09 01561 g004
Figure 5. An example of pulse data detection from binary sequences and data modulation.
Figure 5. An example of pulse data detection from binary sequences and data modulation.
Electronics 09 01561 g005
Figure 6. Average missing packets from consecutive frames.
Figure 6. Average missing packets from consecutive frames.
Electronics 09 01561 g006
Figure 7. Streaming data transmission using the ARQ approach.
Figure 7. Streaming data transmission using the ARQ approach.
Electronics 09 01561 g007
Table 1. Received streaming data consisting of 52 fragmented packets (sequential transmission).
Table 1. Received streaming data consisting of 52 fragmented packets (sequential transmission).
ResolutionListResults
3264 × 2448Avg. Packets/Frame8.78 (Max: 9, Min: 8)
Avg. Pulses/Frame389.59
Avg. Bits/Frame193.99
Req. No. of Frame12 Frame
1600 × 1200Avg. Packets/Frame5.58 (Max: 6, Min: 5)
Avg. Pulses/Frame262.98
Avg. Bits/Frame130.61
Req. No. of Frame11 Frame
Table 2. Received streaming data consisting of 26 fragmented packets (sequential transmission).
Table 2. Received streaming data consisting of 26 fragmented packets (sequential transmission).
ResolutionListResults
3264 × 2448Avg. Packets/Frame8.76 (Max: 9, Min: 8)
Avg. Pulses/Frame389.52
Avg. Bits/Frame193.92
Req. No. of Frame5 Frame
1600 × 1200Avg. Packets/Frame5.61 (Max: 6, Min: 5)
Avg. Pulses/Frame263.03
Avg. Bits/Frame130.67
Req. No. of Frame7 Frame

Share and Cite

MDPI and ACS Style

Kim, B.W.; Yoo, J.-H.; Jung, S.-Y. Design of Streaming Data Transmission Using Rolling Shutter Camera-Based Optical Camera Communications. Electronics 2020, 9, 1561. https://doi.org/10.3390/electronics9101561

AMA Style

Kim BW, Yoo J-H, Jung S-Y. Design of Streaming Data Transmission Using Rolling Shutter Camera-Based Optical Camera Communications. Electronics. 2020; 9(10):1561. https://doi.org/10.3390/electronics9101561

Chicago/Turabian Style

Kim, Byung Wook, Jong-Ho Yoo, and Sung-Yoon Jung. 2020. "Design of Streaming Data Transmission Using Rolling Shutter Camera-Based Optical Camera Communications" Electronics 9, no. 10: 1561. https://doi.org/10.3390/electronics9101561

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop