Crowdsourcing Framework for QoE-Aware SD-WAN

The exponential increase in bandwidth-sensitive multimedia trac on the Internet has created new challenges and opportunities. With the shifting focus from service availability to service quality, there is a need to have quality management measures to serve the high needs of ecient transmission and delivery in time-constrained environments over IP networks. Quality of Experience (QoE) is now considered the most important measure to achieve the twin goals of application eciency and user satisfaction from a user perspective. In this paper, we propose a framework that can be used to collect real-time QoE feedback through crowdsourcing and forward it to SD-WAN controllers to enhance streaming routes based on real-time user quality perceptions. We analyze how QoE can be affected by different network conditions, and how different streaming protocols compare against each other when the network parameters change dynamically. We compare the real-time user feedback to predened network changes to measure if participants will be able to identify all degradation events, and to examine which combination of degradation events are noticeable to the participants. These QoE timestamped feedback is sent back to the SD-WAN controller continuously in order to locate problems and bottlenecks in the current service paths and to enable network controllers to take corrective action by rerouting the streamed trac. Our aim is to demonstrate that real-time QoE feedback can enhance cloud-based services and can adjust services quality based on real-time, active participants’ interaction.


Introduction
Real-time multimedia content streaming over the Internet has gained prime applications in several industries such as communication, education, interactive gaming and entertainment. The main bulk of Internet tra c nowadays is multimedia content, in particular on-demand video and live video streaming.
Real-time multimedia tra c requires high bandwidth, which should be allocated dynamically according to tra c priority. With the emergence of high functioning mobile devices, network service providers are in continuous efforts to support a wider range of applications and quality of service (QoS) requirements with highly utilizing network capacities [1]. Most of real-time streaming content is originated from applications that use Real-time Transport Protocol (RTP). Once mobility is introduced, packet delays and losses in multimedia streams become more common and maintaining QoS becomes more challenging.
The multimedia service can be characterized by: 1) the content it supplies, 2) the transmission means it uses to supply this content, and 3) the services it employs to enable content exchange between different parties. Multimedia service design also takes into consideration the user requirements, which include: cost of the service, ease of content accessibility, content quality and multimedia desirability [2]. Furthermore, the Internet usage has shifted towards content-centric rather than host-centric. At the same time, User expectations are also continuously elevating and the multimedia content providers are becoming more aware of the importance of service quality.
Traditionally, service providers evaluated the quality of multimedia streaming by focusing on network conditions and its corresponding QoS parameters such as delay, jitter, bandwidth and packet loss.
However, more recently the attention has shifted toward Quality of Experience (QoE), which is a more subjective and user-centric assessment technique that is concerned with the user perception of the service. As such, QoE-based assessments are quickly becoming the guidelines for managing user quality expectations.
In terms of transport services, Real-time Transport Protocol (RTP), along with Real-Time Control Protocol (RTCP) or Real-Time Streaming Protocol (RTSP), provide a reliable foundation for real-time services [3].
The emergence of Software-De ned Networking (SDN) has promised better control and management of end-to-end service quality in the networks [4]. Leveraging SDN's advantages such as dynamic programmability, central control, cost e ciency and adaptability to changes in networking environment, makes the Software-De ned Wide-Area Network (SD-WAN) a desirable architecture to control QoE for multimedia streaming applications and services. With real-time user feedback during live video streaming, it will provide a better QoE by trying to enhance streaming routes using SD-WAN controllers.
Most past research contributions in this area have focused on the quality of video streaming, video analytics, network QoS, and metrics for conducting QoE. In those works, QoE is usually measured and assessed after completion of streaming session/or paired comparisons, and results are sent back to management systems to make changes for enhancing the user QoE in future streaming. But current proposals do not provide insights on how QoE feedback can be gathered in real-time and how it should be communicated to the network controllers to allow dynamic network changes to enhance QoE and streaming quality while participants are observing the session. This is the research gap that we are attempting to address in this work.
We propose a framework based on a combination of real-time QoE measurement application and QoS quality parameters which can accommodate a variety of streaming protocols. This framework emphasizes the dependability between QoE and QoS and how the overall user QoE perspective can be affected. We study how dynamic changes in the network could affect the performance of different streaming protocols, and how the streaming protocol adjusts to network changes, and consequently, the perceived QoE of the streaming content. The main protocols used in our model are RTP, RTP over TCP, SCTP, and UDP.
The proposed model is based on real-time alerts of quality degradation from users' side during live video streaming over a cloud-based SD-WAN environment. A QoE-rating application is deployed on the user end devices to collect user feedback during a real-time streaming session. Through this application (which can be deployed as a plugin on web browsers or multimedia players) users send individual or cumulative feedback to the SD-WAN controller (directly or indirectly through a data summarization gateway) to inform the controller about potential problems in the video streaming. This QoE feedback would enable the SD-WAN controller to investigate problems in the service path and to take corrective action by making changes to the topology, resources or routes of the content delivery network.
The main feature of our proposed framework is the ability to identify changes in the QoE of video streams with dynamically changing network conditions in real-time. The questions we try to answer are the following: Will the participants be able to identify quality degradations at the same time as the network changes? Does the content of the video affect quality-rating decisions by the participants despite a noticeable quality degradation? When comparing objective analysis against Human perceived quality, would the results be consistent? Which protocols are most suited for unstable or rapidly changing networks? Is there consistency in protocol performances when video content changes, or when different events and scenarios occur?
The rest of this paper is organized as follows. In Sect. 2, a review of the relevant literature is presented. In Sect. 3 we present an overview of the SD-WAN network and QoE measurement models. In Sect. 4 we present our crowdsourcing QoE-Aware design. In Sect. 5, we will describe the implementation of our test bed. Section 6 discusses performance results and an analysis of the proposed model. Section 7 provides conclusions and directions for future work.

Related Works
A comprehensive discussion of network and service performance indicators for multimedia applications have been presented in [2]. The most important performance indicators include the following: One-way end-to-end delay (including network, propagation and equipment) for video or audio should be within 100 to 150 ms.
Mean-opinion-score (MOS) levels for audio should be within 4.0 and 5.0. MOS level for video should be between 3.5 and 5.0.
End-to-end delay jitter must be short, normally less than 250 µs.
Synchronization of intermedia and intramedia should be maintained using suitable algorithms. To maintain intermedia synchronization, differential delay between audio and video transmission should be within − 20 ms to + 40 ms.
The following parameters should also be taken into consideration while designing a QoE framework for multimedia services are described in [5]: • Video quality at the source. • How the content is delivered over the network and QoS quality parameters.
• User perception, expectations and ambiance.
More recently, crowdsourcing techniques have been considered for collecting user QoE feedback. In [6], authors designed a crowdsourcing framework that overcomes some of the disadvantages of MOS technique, namely: 1) di culty and inconsistency for participants to map their ratings to 5-point MOS scaling, 2) rating scale heterogeneity, and 3) the lack of a cheat detection mechanism. By introducing the ability to have QoE measured in a real-life environment using crowdsourcing rather than a controlled environment in a laboratory, this new approach provides comparable consistency as of the MOS methodology. Another approach, OneClick framework [7], captures user perception in a simple one-click procedure where experiments are held to gather user feedback and then collected data are processed to calculate the accumulative QoE of all users. Programmable QoE-SDN APP was discussed in [8], which aims to improve QoE for video service customers by minimizing the occurrence of stalling events in HTTP Adaptive Streaming (HAS) applications, and by utilizing the forecast and rating estimates provided by mobile network operators.
In order to tackle the requirements of multimedia over IP, multimedia services should have the ability to classify tra c, prioritize different applications and make the necessary reservations accordingly. The Internet Engineering Task Force (IETF) developed an Integrated Service framework that consists of realtime and best effort services. RTP, along with RTCP, and RTSP, provide a reliable foundation for real-time services. However, this framework has had limited deployment due to complexity and backward compatibility.
Some past research efforts have focused on the speci c use of SDN controllers and the importance of the selection of SDN controllers in designing network models. Recently, research done in [9] focuses on using intent-based programming in Open Network Operating System (ONOS) [10] to allow more dynamic monitoring and rerouting services by using intents. Intent Framework [11] enables applications to provide network requests in form of a policy, not as a mechanism. Intents provide a high-level abstraction where programmers only focus on the task that should be accomplished, rather than how these tasks will be translated into low-level rules and how these rules can be installed into network devices, by only expressing required intentions via high-level policies. Those researches aim to enhance Intent Framework to compile more than one intent at the same time and to re-optimize paths based on ow statistics.
Leveraging SDN in routing would allow service providers to customize routing services for applications [12]. This approach is based on a new open framework called Routing as a Service (RaaS) by reusing virtualized network functions. Upon selecting appropriate functions, the authors build customized routing services on the routing paths for different applications.
Several prior works have examined the possibility of managing QoS and QoE using the advantages of SDN architecture. In [13,14] the authors focused on how QoE could be managed e ciently over cloud services, and investigated the challenges facing QoE management in cloud applications, especially the quality of multimedia streaming. The goal of QoE management in that environment was to provide high quality services to user on the cloud while taking into consideration relaying costs behind such quality.
Using QoS over SDN in [15], the authors designed an approach to introduce QoS into IP multicasting using SDN in order to have proper and exible control management of the network environment. OpenFlow protocol was adopted to allow a controller to monitor IP multicasting statistics for each ow to provide end-to-end QoS. They implemented a learning algorithm to allocate required network resources without dropping those low priority packets which could impact the performance for those ows. It thus demonstrated that SDN could be used for network quality management. The next section will discuss past contributions in the area of QoE Models.
A. In-Network QoE Models In [16], the authors proposed an In-network QoE Measurement Framework (IQMF). The user feedback is not considered as an input parameter in this scheme, however the streams are being monitored within the network. Two QoE metrics are adopted by IQMF for measuring experience: 1) quality of video and 2) switching impact over HTTP Adaptive Streams. IQMF offers those measurements for QoE as a service through an API (Fig. 1). This service could be provided to a content distributor or a network provider. By leveraging SDN, it allows control plane to interact with IQMF framework allowing it more exibility to analyze and measure participant's QoE. It also enables IQMF to utilize tra c managements dynamically and provide scalability for deploying more measurement agents. IQMF interacts with OpenFlow controller that keeps the forwarding behavior of the network to ensure that all necessary information about ow duplications is provided to allow better monitoring of QoE.
The QoE measurement framework operates by ltering HTTP packets in the traversing tra c. It then identi es HTTP GET requests, and examines those requests for identi cation of the manifest les such as Media Presentation Description le (MPD). MPD parser extracts information from the MPD ledifferent representations that includes references to different resolutions, quality multitude and playback codecs. The measurement engine then merges the parsed information with supplementary details from the HTTP packet lter in order to monitor the behavior of the user whilst playback continues.
Another model, described in [17], aims to enhance the capabilities of Dynamic Adaptive Streaming over HTTP (DASH) -a standard for multimedia streaming that changes the quality of content presentation automatically in accordance with network conditions. In this research, authors take into consideration the QoE as perceived by the user, and then integrate user perception with dynamic changes in the content.
Such enhancements will provide more e cient QoE measurements and increases positive feedback by the users. This model allows automated estimation of MOS measurements from QoS parameters such as video bitrate, video frame rate and video quantization parameter.
The following three metrics were used in this model: 1. Buffer under ow/over ow: to prevent freezing images and losing packets, buffer thresholds were speci ed. TCP has been used as well for reliable transmission.
2. Frequencies and amplitude of switching quality: the frequency of quality switches of the represented content was identi ed as one of factors affecting QoE.
3. QoS media parameters: the parameters associated with the content of media.
It was found through experiments that it takes seconds to measure presentation intervals that are affected by media parameters. However, it takes more time intervals in terms of switching quality and rebuffering impacting QoE. Where, "the representation quality switch rate, required a recursive approach where the MOS is calculated based on previous MOS variations in order to take into account the entity of the quality switch in addition to the rate" [17]. This model has shown potential in enhancing DASH logic of adaptation capabilities for selecting the best video quality levels, through integrating the QoE monitoring.
OpenE2EQoS model, discussed in [15], aims to introduce QoS into IP multicasting using SDN to have a exible control management of the network environment. In this approach, OpenFlow protocol was adopted to allow controller to monitor IP multicasting statistics for each ow to provide end-to-end QoS. The system makes use of the Additive-Increase / Multiplicative Decrease (AIMD) algorithm to enhance adaptive learning of e cient bandwidth utilization over time to improve QoS. An N-dimension statistical algorithm is used in that approach to redirect low priority tra c packets from overly crowded links while maintaining priority for multimedia packets.
In [18] a method has been proposed for predicting QoE by using machine learning algorithms leveraging SDN. An architecture is designed that uses previously-measured MOS values from users, collected during different network conditions. This data along with objective measures are supplied to machine learning algorithms to predict MOS values for the current network conditions. The SDN QoE Monitoring Framework (SQMF) [19] is a monitoring application that aims to preserve QoE for both video and VoIP applications in real-time regardless of unexpected network issues, by continuously monitoring network parameters and using QoE estimation models. In [20] a new QoE-Aware management architecture over SDN was proposed, which was able to predict MOS by mapping different parameters of QoS into QoE. The proposed framework was designed to autonomously control and allocate underlying network resources infrastructure with the ability to avoid QoE degradation, optimize resource use and improve QoS performance.

B. Crowdsourcing QoE Models
A general crowdsourcing framework for QoE capture was discussed in [6]. The objective in that work is to overcome some disadvantages of MOS technique by utilizing paired comparison technique, as well as the ability to have QoE measured in a real-life environment using crowdsourcing rather than a controlled environment in a laboratory. Four case studies were conducted using audio and video content to evaluate the effectiveness of the proposed framework.
The key features of using this framework for QoE evaluation is: 1. It could be generalized for different types of multimedia content with no need for adjustments. This framework is a promising evaluation technique to measure QoE, however as the authors of the study pointed out, this is not a QoE evaluation; rather, it is Quality of Perception (QoP). As they point out, "QoP re ects a user detectability of a change in quality or the acceptability of a quality level" [6].
The OneClick Framework [7] captures user perception in a simple one-click procedure. Whenever a user is not satis ed with the quality of the viewed content, they can click on a button that informs the system of their dissatisfaction. OneClick includes two main steps: 1) experiments are held to gather users' perception feedback during different network conditions. 2) Collected data are then processed to identify QoE measurements. Figure 2 shows the full OneClick process assessment technique with the following steps: 1) Preparing test materials (optional); 2) asking subjects to do experiments; 3) inferring average response delays; 4) modeling the relationship between network factors and click rates; 5) predicting the click rate given each network factor; 6) summarizing an application's QoE over various network conditions by comfort region.
Given the simplicity of OneClick's approach to user feedback, we have used a similar idea for the user side of our framework; namely to allow the users to express their displeasure with a click when the quality of the streaming deteriorates.

Network Model And Assumptions
A. Network Model A real-time cloud-based content-delivery network model is shown in Figure 3. The network model is based on four tiers: The content provider tier produces the multimedia content for distribution over the network.
The content can be live stream or broadcast, music, video on demand etc., though live video content is our main focus in this paper. Content is made to be accessible and processed by the various servers and forwarding modules in SD-WAN in the next tier.
SD-WAN consists of a collection of interconnected network switches, controlled by one or more centralized controller(s) through a southbound protocol such as OpenFlow. The OpenFlow protocol is an open-source, standard protocol that governs communication between the SDN controllers and switches. OpenFlow allows full control over packet ows where the controller speci es the routing paths as well as how packets should be processed.
The end-user tier contains the wide range of end devices that are used for accessing the content and provide continuous, real-time feedback of QoE in form of perceived video quality through SD-WAN. The end-user tier connects to SD-WAN through a number of gateway servers which form a data summarization tier and are responsible for delivering the content, as well as collecting and summarizing the crowd feedback to the controllers. We will describe the role and functionality of each tier in more detail in the following sections.
The software-de ned and centralized nature of SD-WAN allows e cient tra c engineering to meet dynamic service requirements. An SD-WAN controller can have a global view of the network, making it an ideal physical substrate for cloud-based content-delivery network (CCDN) environments. Implementing QoE applications over SD-WAN allows us to further enhance the quality of streaming by adjusting video quality based on user feedback, not just by relying on QoS SLA.

B. QoE Model
There are two main approaches for QoE assessments: subjective and objective methods. Subjective techniques are based on user interaction and feedback. As discussed earlier, the most common subjective approach is Mean Opinion Score (MOS) based on a quality rating system on a scale from 1 to 5, where 1 stands for 'Bad' and 5 stands for 'Excellent'. 3.5 is the minimum acceptable threshold for a video MOS [5]. MOS scaling still may give room for inaccurate representation of user perception [6], due to the non-similarity of the scale interpretation by different participants.
While subjective approaches are considered to re ect the user perception more accurately, they are expensive to roll out because such QoE assessments require a large scale of participants in order to obtain reliable results. They are also time-consuming, as traditional QoE experiments are conducted in controlled lab-environment, making it di cult to collect su cient results from experiments in a limited time-frame [6]. QoE crowdsourcing techniques have been proposed to overcome these constraints; where taking advantage of employing a diverse group of online participants, getting subjective results becomes relatively cheaper and more e cient than traditional ways. Crowdsourcing allows subjective measures based on both video-pair comparisons as well as MOS-based rating comparisons, with the exibility to choose participants' demographics if certain demographics required for speci c results.
Despite their scalability, crowdsourcing experiments lack supervision, which makes some results not fully trustable. Researchers should be able to identify trusted and untrusted participants. This can be achieved by designing the crowdsourcing campaigns based on certain best practices. These best practices are majorly concerned with technical implementation aspects of the experiment, campaign and test design and thorough statistical analysis of results [21]. Campaigns should be simple enough for participants to understand how the experiment is designed and what is required to complete it.
As opposed to subjective assessment techniques, objective QoE assessment techniques are mostly based on network analysis and technical comparisons that aim to produce a quantitative assessment. Quantitative QoE assessment is tightly related to the QoS of an application or service. Peak-Signal-to-Noise-Ratio (PSNR) is considered an objective approach for measuring quality, as it assesses how much similarity exists between two different video images. It is widely used in video streaming assessment, where the higher PSNR value the higher similarity between the original and received video images. One drawback of PSNR is that it does not take into consideration how human perception works.
Structural Similarity Index (SSIM) [22] is another measurement approach for estimation of perceived visual distortion based on the structural distortion of the video. SSIM addresses the shortcoming of PSNR by combining other factors such as contrast, luminance and structure similarity and compares the correlation between the perceived video images and the original video images, hence considered a Full-Reference model. The higher the ratio, the higher the structural similarity. Another objective approach is Video Quality Metric (VQM) [23], which is a metric to measure the perception of video quality as closely as to human perception. The VQM metric is designed to be a general-purpose quality model for a range of video systems with different resolutions, frame rates, coding techniques and bit rates. VQM measurement takes noise, blurring, and block and color distortions into consideration. VQM gives an output value of zero if no impairment perceived, and increases with a rising level of impairment.

A. QoE Crowdsourcing
The proposed model is based on real-time QoE crowdsourcing feedback of quality degradation during live video streaming, over a cloud-based SD-WAN environment. Figure 4 highlights details of our model, in which a streaming server transmits multimedia content over the SD-WAN environment. During the video streaming, dynamic network changes and events may affect the perceived quality at the user end. Several streaming protocols can be considered such as RTP, SCTP, TCP and UDP for data transmission. SD-WAN routes the video stream and delivers it to the participating users. A QoE-rating application is deployed on the user end devices to return user feedback during a real-time streaming, where participants click on a dislike button when they feel the quality has been degraded.
The QoE-rating feedback application is designed to send REST API requests (summarized through intermediate servers for scalability) to the SDN controller to inform the controller about potential problems in the current video stream and possible request of corrective actions such as tra c rerouting. The QoE rating application can be deployed as a plugin on web browsers, multimedia players, or as a desktop application. In contrast to the MOS technique, participants do not have to decide between different grading scales; instead, they only alert the SD-WAN controller to quality degradation, therefore providing more decisive feedback. The timing between feedbacks could be an indication of quality; i.e. more frequent feedback ("dislike" clicks) indicate a lower quality than less frequent feedback. As such, all clicks are timestamped before transmission. The intermediate servers will collect this feedback and provide summaries (e.g. number of dislikes within a given time interval) to the QoE-control algorithm on the SD-WAN controller for potential actions.
Providing QoE feedback would enable the SD-WAN controller to detect problems in the service path and to take corrective action by making changes to the virtual topology of the content delivery network, reassigning users, or rerouting tra c. Ideally, a resource optimization algorithm such as [24] can be executed in real time to respond to QoE degradation, however the complexity and processing time of such algorithms must be considered in order to provide an effective remedy in real time. The use of crowdsourcing would provide a more scalable and e cient method for collecting feedback, whether to correct real-time problem or for creating performance benchmarks, as we shall explain in the next sections.

B. Data Summarization
The issue of scalability is an important challenge in the use of crowdsourcing for quality of service control. Certain streaming content services may have millions of users at any time, and receiving and analyzing feedback from them in real time could become a bottleneck. In order to address this issue, we add two elements in our design: 1. An intermediary data summarization layer is placed between the end-user tier and SD-WAN to implement a hybrid fog computing operation. The nodes in this layer are responsible for receiving feedback from users in their region, summarizing the feedback and sending it back to the network controller(s). For instance, this feedback could include the number of dislikes received over a reporting period. The number and location of these intermediary nodes can be optimized to accommodate any processing limits at the controllers.
2. Furthermore, a minimalist approach is used in the design of the user feedback data. As discussed in Section III, our framework relies on negative feedback (dislike button) that is only given when there is a problem. Therefore, no feedback is expected in all regions and all times as long as the streaming quality is acceptable. When a congestion or failure scenario causes a spike of negative feedback in a region, the intermediary-layer server in that region will collect the feedback and send a summary report to the central controller. This approach will allow for service scalability to a wide area network.
It must be noted that QoE response time is inherently different from QoS responses. In case of userinitiated QoE, the feedback is controlled by user actions that typically span a few seconds. QoE-correcting actions from the controller can also be executed within a similar time scale, and as such, the limiting impact of propagation delays is less signi cant in QoE-aware services, as opposed to QoS-aware systems C. Use of the proposed framework for performance baseline Aside from real-time response to QoE feedback, this framework can also be used for planning and performance analysis of future streaming. In such scenario, the video stream is saved under different protocols and network conditions to measure QoE via user participation. These saved videos are used to construct a paired comparison stimulus on any chosen crowdsourcing platform in a campaign to be advertised to users asking for participation. Such campaigns include instruction of requirements for participation, consent forms, and terms of any designed stimuli. It includes processed videos and paired comparison stimuli are constructed accordingly. Figure 5 illustrates the crowdsourcing pair comparison model design.
Paired comparisons can be seen as an alternative for the MOS-based method, due to its simplicity where the need to decide between ve different ratings is eliminated, making it more intuitive judgment. The paired comparison technique makes it easier for users to express their opinions, decision making and have an easier experience interaction when multiple factors are applied.
In order to build the performance benchmark, participants are hired to rate videos in a paired comparison stimulus with different streaming protocols to provide feedback on which video has the higher QoE. In any crowdsourcing model design, there must be a method to identify reliable trustworthy participants since experiments are not controlled and lack monitoring that traditional controlled lab environments provide. To verify the reliability of the participant, trap questions must be introduced. Participants are presented with a golden question, where a stimulus constructed between the original video with no applied processing as a reference and an event degradation processed video. If participant didn't rate original video with higher QoE, then the participant is considered unreliable and their results are excluded.
A campaign is run over a period of time depending on the number of participants required to conduct the study. Rating scores can be computed at any point during the campaign to show trends and results.
Different methods can be used to compute these scores such as Crowd Bradley-Terry Model [25].

Test Bed Implementation
A. Use-case Scenarios We created an SD-WAN environment using Mininet and ONOS remote controller, which provides an emulation of a software-de ned virtual network similar to a real networking environment, running kernel, switching and application code in one single virtual machine using a simple in-line code. VLC server was used as the video streaming application. User feedback was collected through a custom-designed plug-in for VLC client. For the purpose of these tests, we integrated the role of the intermediary layer into the hosts; i.e. assumed that the feedback response from the hosts represent a summarized feedback from the region they represent.
Two experiments were conducted using human participants, one based on paired video comparison using QoE crowdsourcing technique and the second based on real-time feedback during video streaming in a controlled lab environment. The rst experiment, Human Paired Comparison (HPC), was tested using a QoE crowdsourcing campaign of a paired video comparison on a set of processed videos using subjectfy.us web service [26]. We created comparable video sets based on four selected HD videos. Each video is 40 sec in time. Details of these videos are presented in Table 1.
We also created 5 random scenario les with changing delay and loss events every 10 or 5 sec shown in Table 2, and applied RTP, Legacy UDP and RTP over TCP as streaming protocols in different scenarios.
These processed videos were used as the golden question for verifying the reliability of participants in comparing original HD video against processed (deteriorated) video. If participants chose the processed copy, then their feedback results were considered as untrustworthy.
For the second experiment, we created random scenario les of 40 sec in length with timed changing events every 0.5 sec during the video streaming session. We applied these scenarios to 3 HD-Quality videos, and applied RTP, Legacy UDP, SCTP, RTP over UDP and RTP over TCP as streaming protocols with a result of 150 processed videos. Using MSU-VQM objective analysis tool [27], we computed comparative results against the original videos for PSNR, VQM and SSIM.
For these experiments, we created a network topology consisting of one controller, one switch, and two hosts. By using minievent module script [28] we applied changes to adjust delay and loss over a period of time during video real-time streaming, in order to measure the impact of changes in network conditions on QoE. The parameters of our event scenarios were selected to check how We considered three network change scenarios: 1.
Fixed link delay and changing packet loss over time

2.
Switching from high delay to a substantial lower delay with a xed packet loss 3.
Gradually decreasing delay over time and then gradually increasing it while applying packet loss.
In our experiments, link delay was set in the range of [0-80ms] and packet loss was set in the range of [0 or 1%]. The ow bandwidth was xed at 50 Mbps. We created scenario les of 40 sec in length with timestamped events where delay and loss were changed during the video streaming session. For each scenario, we streamed the videos with different streaming protocols, recorded all output videos and ran a comparative analysis.
In order to examine the performance of the real-time QoE feedback on SD-WAN operation, we also set up an SDN-based re-routing experiment to demonstrate that QoE feedback can be captured and processed in real-time during live video streaming session. Feedback can be sent on-spot to the SDN controller to alert the controller of issues in the steaming service. Our intention was to demonstrate how tra c reroute could be done instantly based on QoE feedback and how the results would compare from the perception of a participant.
For this scenario we created a network of three hosts: one acts as a streaming server, the second acts as a client and the third an un-namespaced server which can communicate with ONOS SDN controller. The hosts were connected through a network of 10 OVS switch devices with 22 links and 50 ows, where three edge-disjoint paths exist between the client and the server. RTP used as default streaming protocol.
The network topology is shown in Figure 6.
During video streaming experiences, we applied network degradation parameters which affect the quality of the network to indicate if and by how much the participating user will be able to detect such changes during video streaming, and how these changes affect their QoE. Furthermore, we intended to see the timing of their QoE feedback within time window when QoS parameters change events occurs. And how streaming protocols will be able to adapt to these changing events, and if all or some of these changing events will be noticeable to participants or not.
The SD-WAN rerouting experiment was designed as following: 1. The streaming from the server to the client starts normally and is veri ed visually by the users.
2. The minievent module inserts delay or loss on speci c network links according to the speci cations of the scenario, in order to create quality degradation in the video.
3. Once the users notice the video quality degradation, they click 'Dislike' button when quality degrades. The feedback is transmitted to the Feedback server through HTTP.
4. Server communicates the feedback to the SDN controller 5. Controller resolves client and server IPs into MACs using REST.
6. Controller retrieve current intent information (The intent was created using IFWD application reactively).
7. Controller queries current tra c path.
8. Controller computes alternative paths between client and server 9. From the list of devices on the current path, remove unavoidable devices (shared by all known paths) and choose a new ObstacleConstraint. 10. Alter intent by adding obstacle to one of devices on current path (so ONOS is forced to rebuild a new path) 11. POST modi ed intent with new obstacle.
12. Wait until this modi ed intent installed and query new path.
13. New streaming path established and QoE enhancement is veri ed visually by the users.

A. HPC Experiment Analysis
For the HPC experiment, there were 2430 paired comparison questions with a total of 259 participants.
243 participants were successful and 16 failed the reliability test. Ranks were computed based on Crowd Bradley-Terry model [19]. Figure 7 shows participants ratings for Video 1 for each protocol and scenario of events. We collected similar results for other videos in Table 1.
Using MSU-VQM, we computed VQM for processed videos from experiment 1 against the original video. Table 3 shows VQM values for each resulted video. By comparing subjective and objective results, we found that HPC and VQM results were consistent, where mostly the highly ranked protocols in HPC had the best (lowest) VQM value across the three protocols per video. We note that the sequence of events in the scenarios in Table 2 was designed to allow us to monitor how protocols differ in recovering from packet loss and delay; how the QoE will be affected; and if participating users will detect changes with one parameter xed and the other changes. It was found in our experiments that for the packet loss events the quality degradation is most noticeable, however changing delay parameters in some cases was not noticeable by participants, likely due to the buffering ability of the video player on the client side.

B. SD-WAN Re-routing experiment Analysis
The SD-WAN rerouting experiment was conducted in a controlled lab environment in which each participant watched a 3-minute live streaming video and was asked to provide QoE feedback in form of a 'dislike' click when quality degrades based on timely changed scenarios of event. Participants could provide feedback clicks any time during the viewing session. A 10-second waiting time was enforced between acceptance of two consecutive clicks to ensure controller's rerouting is complete. In other words, repeated clicks within a 10-second interval were ignored. Figure 8 shows the timing of the vents for QoE-aware rerouting experiment for a sample participant, where red dots represent ping time (right axis), blue line is iperf loss percentage (left axis), green bars are VLC player errors and the gray background indicate active minievents scenarios. We observe that the streaming quality improved after every participant feedback click, where it was noted that the blue line indicated no further loss on the streaming link. It was noticed that when applying a typical delay without loss, users did not detect quality degradation and they typically do not click the feedback button, primarily due to the playback buffering capabilities of the VLC player. With gradually increasing packet loss% with minimum delay, the QoE was affected immediately and accordingly the feedback for reroute was received. By applying degrading changes on two paths out of three, it was found that ONOS kept searching for the alternative routes with every click until the quality was acceptable for the user.
With all participants, it was found that ONOS managed to reroute tra c based on users' feedback, and the quality of the stream was enhanced after controller's enhancement action. Among all participants, ONOS reaction and response for rerouting decision was consistent and timely. It took ONOS between 10-15 ms to construct a new route (intent) as a new streaming path and between 15-20 ms to reroute the tra c. These results demonstrate that the users' interactive feedback can be taken into consideration during streaming sessions and this feedback can be communicated in a timely manner to the SDN controller to alert of an existing issue and that a corrective action is required. It was shown that rerouting decision can be made on spot and quality can be enhanced based on external user feedback.

Conclusion And Future Works
In this paper, we proposed a real-time QoE crowdsourcing model. The proposed model is based on a combination of QoE measurement application and QoS quality parameters that accommodate a variety of different streaming protocols. It emphasizes the dependability between QoE and QoS and how the overall user QoE perspective can be affected. We have analyzed how dynamic changes of events can affect the performance of different streaming protocols, and accordingly perceived quality. We have compared objective and subjective results and found that both results are mostly consistent. We also demonstrated that an SD-WAN controller can receive feedback and detect problems in the service path and take corrective action by making changes to the routing paths of the content delivery network in a timely manner. Our goal was to prove that real-time QoE feedback could enhance cloud-based services and could adjust services quality based on real-time active participants' interaction.
For future work, the use of Arti cial Intelligence (AI) can be considered in this operation. With AI, it is possible to learn feedback patterns received by external user participation and an AI algorithm can decide if rerouting is required or not, where we could have a threshold for number of clicks. This AI algorithm can determine this threshold dynamically and adjust it by learning from user feedback. We can also look into algorithms for optimizing SDN environments, cloud resources and paths, taking into consideration the  The ow of a complete OneClick assessment procedure from [7]. Reprinted with permission from IEEE.