VID2META: Complementing Android Programming Screencasts with Code Elements and GUIs

Abstract

The complexity of software projects and the rapid technological evolution make it such that developers often need additional help and knowledge to tackle their daily tasks. For this purpose, they often refer to online resources, which are easy to access and contain a wealth of information in various formats. Programming screencasts hosted on platforms such as YouTube are one such online resource that has seen a growth in popularity and adoption over the past decade. These screencasts usually have some metadata such as a title, a short description, and a set of tags that should describe what the main concepts captured in the video are. Unfortunately, metadata are often generic and do not contain detailed information about the code showcased in the tutorial, such as the API calls or graphical user interface (GUI) elements employed, which could lead to developers missing useful tutorials. Having a quick overview of the main code elements and GUIs used in a video tutorial can be very helpful for developers looking for code examples involving specific API calls, or looking to design applications with a specific GUI in mind. The aim is to make this information easily available to developers, and propose VID2META, a technique that automatically extracts Java import statements, class names, method information, GUI elements, and GUI screens from videos and makes them available to developers as metadata. VID2META is currently designed to work with Android screencasts. It analyzes video frames using a combination of computer vision, deep learning, optical character recognition, and heuristic-based approaches to identify the needed information in a frame, extract it, and present it to the developer. VID2META has been evaluated in an empirical study on 70 Android programming videos collected from YouTube. The results revealed that VID2META can accurately detect and extract Java and GUI elements from Android programming videos with an average accuracy of 90%.

1. Introduction

Developing modern software systems is challenging due to the increasing complexity of software and the vast technological knowledge that developers need to obtain in order to effectively implement it. To obtain this knowledge, developers often need to refer to external resources and frequently consult informal online documentation in the form of questions and answer (Q&A) websites, tutorials, and API documentation due to the wide availability and variety of these resources. Recently, programming screencasts have become a popular resource among developers [1,2] due to their engaging and interactive nature [3,4,5]. In addition, the number of programming video tutorials is rapidly growing, since it is generally much easier for programmers to prepare them than to create a text-based tutorial [2,6].

Although the amount of Android programming screencasts at the disposal of developers is vast, the code information presented in these videos cannot be easily accessed, searched, and navigated. This is due to the nature of the data, as videos contain a sequence of frames or images, which make it hard to fully explore the content of the videos without manually skimming through them. To find a video of interest, a developer typically writes a query, and a list of videos is retrieved based on the video metadata which includes the title, description, and tags. Then, developers decide which video is the closest match to their information needs based on (i) reading the video metadata and/or (ii) manually skimming over the content of the video. Unfortunately, currently available video metadata are not always descriptive to developers, typically containing general information about a programming task rather than code-related details [5,7]. At the same time, manually skimming over videos is not efficient and can lead to missing useful parts of videos [8,9], given that people spend less than a second analyzing an online resource to determine its relevance [10]. Therefore, there is a need to complement Android programming screencasts with additional metadata that specifically describe the code and GUI screens appearing in the videos, making them easier to search, inspect, discover, and navigate.

To augment videos with code and GUI-related metadata, this paper introduces VID2META, which is an approach that operates on video frames and analyzes their visual and textual content to automatically extract and list Java code and GUI elements (represented in XML code), as well as GUI screens. To begin with, VID2META employs an object detector to locate the regions that contain a Java code and GUI elements (I refer to these regions as code-editing windows) and GUI screens (GUIs). Then, it uses optical character recognition (OCR) to extract the text present in the code-editing windows and proceeds to classify this text as either a Java or XML code using deep learning (DL) methods. Last, VID2META uses heuristic-based techniques to fix the syntax of the OCRed Java and XML code, locate and extract Java and GUI elements, and detect the incorrect ones using cross-frame information. Note that Java elements include import statements, class names, method names, and method calls, whereas GUI elements include their names such as ListView and ProgressBar.

I conducted four empirical evaluations to assess (i) the accuracy of localizing the code-editing window and GUIs using the proposed approach as opposed to a previous work [5]; (ii) the ability of the proposed approach to correctly classify the content of the code-editing window as either a Java or XML code; (iii) the effectiveness of the proposed techniques to firstly find and extract Java and GUI elements from the code-editing window and secondly detect the incorrectly extracted elements; and (iv) the accuracy of identifying and eliminating duplicate GUIs in order to remove noise and cut down on processing time. To carry out these empirical evaluations, I collected a total of 70 videos from YouTube, with all their individual video frames. I used the frames in 20 of the videos to train and evaluate the approaches for localizing the code-editing window and classifying its contents, and the remaining 50 videos to evaluate the proposed techniques that extract Java and GUI elements. The results of these evaluations illustrate that VID2META is able to successfully extract and correct Java and GUI elements from Android videos, achieving an accuracy above 94% in localizing code-editing windows, project windows, and GUIs. In addition, VID2META is able to successfully detect the content of code-editing windows and reduce the errors in the OCRed Java and GUI elements by an average of 39.5%. Moreover, the proposed approach for detecting and eliminating duplicate GUIs achieved an F-Score of 84%.

Tha major contribtions of this paper are as follows:

• VID2META is proposed which, to the best of my knowledge, is the first approach that complements the metadata of Android programming screencasts with Java and GUI elements extracted from the videos;
• Through extensive experiments, I show that VID2META can accurately (i) locate the code-editing window and GUIs in the frames of Android screencasts, (ii) classify the content of a code-editing window, and (iii) extract Java and GUI elements;
• I provide a replication package (https://zenodo.org/record/5014890, accessed on 20 July 2022) containing the complete dataset, source code, VID2META’s output, and the detailed results of the experiments.

2. Approach

In this section, VID2META is introduced, which aims to complement the metadata of Android screencasts by analyzing their video frames to extract meaningful Java and GUI elements, as well as entire GUIs.

2.1. VID2META Overview

As shown in Figure 1, VID2META takes a video as input, analyzes the visual and textual content presented in the video frames, and produces Java and GUI elements (I collectively refer to these elements as code elements). To extract code elements, VID2META needs to locate the code-editing window that contains the Java or XML code and then extract the appropriate code elements. Since the two programming languages require different approaches to extract code elements, we need to first determine what programming language we are dealing with and then apply the proper approach accordingly. I initially envisioned two approaches to classify the content of the editing window, as follows: (i) train an object detector with two classes (Java and XML) to directly determine the programming language of the code in the editing window, or (ii) train an object detector to localize the code-editing window and then use another approach to classify the text inside the editing window as Java or XML. Through a preliminary experiment, I found that the first approach failed to distinguish between Java or XML due to the similarity between the visual appearance of code written in the two languages (e.g., similar font type and color, background color, and integrated development environment window structure). Therefore, I proceeded with the second approach as follows. First, VID2META locates the bounding box of the selected file in the project window using computer vision techniques and uses OCR to extract the file extension of the selected file, and thereby detect the content type of the editing window. Second, VID2META uses a deep learning binary text classifier to predict the programming language of the code in the editing window (e.g., Java or XML). In case the content of the editing window is Java, VID2META extracts import statements, class names, and method names and calls, whereas if it is XML, VID2META extracts GUI elements from the file. Besides extracting code elements, VID2META extracts GUIs and removes duplicates using computer vision techniques. Next, more details about the pipeline of VID2META are presented.

2.2. Localizing Code-Editing Windows, Project Windows, and GUIs

A mobile programming screencast consists of n frames where each frame is typically divided into windows/regions with different sizes (e.g., a frame shows an IDE could have several regions such as an editing window, a project window, a GUI layout, etc.). To extract Java and GUI elements from those frames, we need to locate the bounding box of the code-editing windows precisely. Thereby, we are assured that when we apply OCR on only those windows, we are only extracting the code information (i.e., to avoid extracting other irrelevant information from other windows in an IDE such as a project window, an output window, etc.). The windows that are located are defined as follows.

• Code-Editing Window (CEW): This window contains Java or XML texts.
• Project window (PW): A project window is typically displayed on the left side of an IDE and includes resource files of a project in the form of a tree view.
• Graphical user interface (GUI): Visual GUI elements are placed on GUIs where users can interact with the app.

Formally, given a video V = $\{f_1,f_{1+1},\dots ,f_n\}$ where $f_i$ is a frame captured at the ith second and n is the total number of seconds in video $v_i$. I aim to detect the location of each window $w_i$ in $f_i$, if any. Note that this problem is considered as a multi-window detection and localization problem in video frames, in which each frame $f_i$ could have all possible combinations of the pre-identified windows. For example, in a mobile programming screencast, the screen of a narrator could show an IDE where the project window, an XML, and one or more GUI previews are displayed at the same time. Regardless of the number of windows displayed at the same time, all windows must be located accurately.

To recognize the windows in a frame, we need first to obtain each window’s features and feed them into a learning model. For this task, a convolutional neural network (CNN), called Inception-Resnet V2 [11], with a region-based object detector, called Faster RCNN [12], are utilized. Note that a pre-trained CNN is successfully adapted in analyzing video programming frames [13,14,15], and Faster R-CNN has shown promising results in detecting mobile GUI components [16,17]. The choice of the Inception-Resnet V2 network with Faster R-CNN is based on the empirical evaluation performed by [18] who found that this combination outperformed several other object detectors that used various CNN architectures. Several parameters impact the network’s performance during the training process. The choice of the optimizer plays an essential role in updating the weights after each epoch by back-propagation (one forward and backward pass of every training sample through the neural network). Based on the recommended optimizer by Faster R-CNN, momentum is used, which is a version of the stochastic gradient descent (SGD) [19]. The default values for the network layers’ hyperparameters are used, such as a stride value of 16.

To train the object detector, we need to manually collect videos and annotate code-editing windows with their coordinates. Next, the data collection and annotation processes are explained.

2.2.1. Datasets: Mobile Programming Screencasts

To prepare the data for the object detector’s training, I used 20 videos manually chosen from YouTube. To select the videos, I searched for various topics using the keyword Android, combining it with words such as Google Map, game development, or ListView. The final selection was limited to a maximum of three videos from each topic. A total of 11 different categories were collected. Most importantly, I ensured diversity by selecting a maximum of one video from each playlist. A total of 18 unique authors produced the videos. Concerning the visual content of the video datasets, I ensured that there was a variety of (i) regions presented in the video (two GUI previews shown at the same time, XML with UI, and others) and (ii) IDE background colors (15 white and five dark greys). After the videos were chosen, they were downloaded to the server with the highest available quality using the youtube-dl (https://github.com/rg3/youtube-dl, accessed on 10 June 2021) tool. The mean and median length of the collected videos were 608 s and 592 s, respectively.

2.2.2. Annotating Code-Editing Windows, Project Windows, and GUIs

To train a model to locate a specific window/region, we have to collect several images that contain those regions and feed them into a neural network with their coordinates. Because the regions are spatially located within an image, the neural network learns to extract the relevant spatial features (through convolutional layers) for each region, which maps to its coordinates.

The mobile programming screencast dataset was used to annotate three windows. Note that a video frame containing an IDE could have none to several windows (e.g., a project window, an editing window, and a GUI preview displayed simultaneously).

Two annotators annotated each window belonging to one of the categories with bounding box information (e.g., coordinates) and a label (e.g., CEW, PW, or GUI). Previous works that analyzed programming screencasts extracted a frame per second and classified each frame individually [13,15,20]. They also found that the manual labeling task requires time-intensive human efforts (i.e., required 100 students for the annotation process in [13]). In our case, we classified a frame, defined the bounding box of each window, and labelled its class, which adds more time efforts to the manual annotation process. Based on the study that found that programming screencasts are more static than other types of videos [21], we found that annotating the entire video while playing instead of extracting and annotating each of its individual frames is much more effective. On average, annotating a single region on a frame takes about seven seconds.

To facilitate the process of video annotation, we used a cloud-based web tool called DataTurks (https://dataturks.com/, accessed on 15 June 2021) and uploaded the 20 videos. Next, we initiated the class names with the labels. Note that since we need to evaluate the proposed approach (explained in Section 2.3) in distinguishing between Java and XML, we labeled the CEW with either Java or XML. Yet, when the network was trained, Java and XML were considered as one class that belongs to CEW. We played each video and paused it once a region of interest (RoI) appeared. We labeled each region with a class name and drew a corresponding bounding box around each region (i.e., there could be more than one region in each snapshot). The video was played while our regions and class names were still drawn. Thus, we did not need to annotate each snapshot/frame individually. We paused the video when our ROIs were no longer shown to remove them accordingly.

After annotating all videos, we downloaded the annotated datasets in the JSON format. This file included all information we needed to create a PASCAL VOC [22] format file for each image, including the ground-truth annotation information. This file is required when training an object detector. We parsed the JSON file and extracted the annotation information for each window: StartTime, EndTime, className, and BoundingBox. Then, for each second between the start and end time, we (i) extracted a frame from the corresponding video such that the frameNumber = fps * currentSecond, (ii) saved the frameNumber, className, the bounding box, and the currentSecond information. Once this process was finished, we grouped different regions based on the currentSecond and saved them in the PASCAL VOC format file. Using this step, a single PASCAL VOC format for each image could have multiple regions/windows. To confirm that our annotation was accurate, we drew a bounding box with a class label on each region based on the ground-truth information. A second annotator checked each frame, and we revised the ones that did not have an accurate annotation. The Cohen’s Kappa coefficient was 0.95, which indicated a high agreement between the annotators.

Table 1. Statistics of the dataset with respect to the annotated regions in terms of a single-region system (exactly one region in an image) a multi-region system (at least two regions in an image).

	Region	# of Images	# of Regions
	PW & Java	2077	4154
	PW & GUI	1683	3366
Multi-Region	PW & XML & GUI	1138	3414
	PW & XML	928	1856
	PW & GUI & GUI	611	1833
	GUI & GUI	101	202
	Java	513	513
Single-Region	GUI	453	453
	PW	294	294
Total		7798	16,085
	GUI	3986	7390
	XML	2066	2066
Overall	Java	2590	2590
	PW	6731	6731

shows the number of collected images and the total number of windows/regions in each image. Because not all images contained only one region (RoI), the table was divided into multi-region (ROIs) and single-region systems. The majority of the annotated images contain a project window (PW) and Java regions. There are few cases in which more than one GUI was displayed simultaneously, which makes the total number of annotated GUI reached to 7390 regions. The GUI regions could be displayed with other regions, such as (PW and XML), together with 1138 images. There are few cases where a single region was displayed on the screen, such as Java. In total, the number of annotated images and regions was 7798 and 16,085, respectively.

2.3. Detecting Programming Language

The approach proposed above locates the bounding box of code-editing window regardless of its content that could be a Java or XML file. Two different approaches were followed to the extract code or UI elements; therefore, we needed first to predict the editing window’s content. Two different approaches were proposed to predict the content of the editing window as follows. First, the selected file was located from the project window with its file extension (e.g., .java or .xml). This approach cannot be applied in several cases, such as when the project window is hidden, or if there is no selected file in the project window. Thus, a second approach was proposed based on the textual content of the editing window. More specifically, a deep learning model was employed to predict the text’s programming language written in the editing window. Let us first explain the approach that locates the selected file in the project window using a pipeline of three main steps.

First, the background color of the selected file was typically different from other files. Based on this, we needed to accurately find the background color (BG) of each file listed on the project window (PW). To detect the BG, we needed to first detect the bounding box of each text in PW. The problem of text detection is different from object detection such that for the first one, the region of interest (ROI) is a text composed of different sizes of characters, numbers, or symbols, whereas for the latter, the (ROI) has a well-defined closed boundary [23], such as the entire PW region in our case. Thus, in this task, a fine-grained text-based detection approach [24,25] was utilized. In particular, I leveraged the architecture of a connectionist text proposal network (CTPN) [26] to detect the position of each text entry (file or directory) presented in the PW as follows. First, the input images (PWs) were fed into a CNN-based feature extractor, called VGG-16 [27], to extract the feature maps. Second, to produce a fine-scale text proposal, the CTPN detector densely slid a $3\times 3$ window in the map to detect the text line and then produced text proposals. Last, to further increase the accuracy of the text proposal and find meaningful context information, the CTPN connected sequential text proposals in the feature map using a recurrence mechanism. It has been shown that using sequential context information on words improves the recognition performance [28]. Ultimately, a $P{W}_i$ predicted by $M_i$ (explained in Section 2.2) was fed into a (CTPN) that outputs a set of closed bounding boxes for each textual entry (e.g., the names of files, images, or directory) presented in the $P{W}_i$.

Second, this step’s input is the output of the previous step, which is a set of predicted bounding boxes for each textual entry presented in $P{W}_i$. Each predicted bounding box is considered as a candidate that might or might not have the region of interest (selected file). An abjectness score was defined to either zero (entry was not selected) or one (entry was selected). To detect the objectness score, I based the approach on assuming that the selected entry’s BG color is different from others. Note that the detected bounding box covers only the presented text, whereas the background color spans over the PW’s width.

For this reason, each predicted bounding box was expanded to the full width of the PW (i.e., refined bounding box (RBB)). Note that the bounding box of each $P{W}_i$ was automatically detected using the previous approach (explained in the previous Section 2.2). For each $RB{B}_i$, the dominant color was obtained by computing the (RGB) color of each pixel and count the frequency of each color.

Last, the dominant color of each $RB{B}_i$ was compared to that of $RB{B}_j$ where j ≥ i + 1 and j ≤ n (total number = $RB{B}_n$). One trivial solution is to compute the similarity between two RGB colors based on a distance metric (e.g., euclidean distance). However, two RGB colors might look very similar to the human, but they might have a large euclidean distance, and this is due to the fact that they are represented in RGB color space which led to the flaw. Thus, CIE Lab color space was designed to approximate human vision based on the color’s lightness [29]. For this reason, the color differences between two $RBBs$ were quantified by computing the ΔE and following the guidelines of the International Commission on Illumination (CIE), as shown in Equation (1). Formally, the dominant background color for each $RBB$ was converted into (L*a*b*) where (i) L represents the lightness value and ranges from zero (dark black) to 100 (bright white), (ii) a represents the green (negative direction) and red (positive direction) components, and (iii) b represents the blue (negative direction) and yellow (positive direction) components. To detect whether the background color of $RB{B}_i$ was similar to that of $RB{B}_j$, ΔE has to be ≤ a threshold. By the end of this comparison, there should only be one unique $RBB$ which embodies the selected file or directory.

$$\Delta {\mathit{E}}_{\mathit{ab}}=\sqrt{(L_2-L_1{)}^2+(a_2-a_1{)}^2+(b_2-b_1{)}^2}$$

(1)

The second approach, to detect the code-editing window’s content, is based on DL text classifier. I leveraged GuessLang (https://github.com/yoeo/guesslang, accessed on 20 June 2021), which is a deep neural network with three classification layers and a customized linear classifier. I opted to employ GuessLang as it has been successfully used in the field of SE [30,31]. GuessLang has already been trained on 30 programming languages. The list of programming languages that the model was trained on did not include Android development (e.g., Java and XML). For this reason, I fine-tuned the architecture of GuessLang and trained it on a new dataset. The new dataset contains a total of 15,000 Java and 15,000 XML files extracted from 675 open-source Android projects found in the play store and collected by [32]. During the training, the source code files were transformed into features vectors in which bigrams and trigrams were computed before they were fit into the network.

2.4. Extracting Java and UI Elements

To extract Java and UI elements from videos, I applied OCR on the predicted editing window and predicted its content using the approaches proposed above. Next, I present the approaches that extract Java and UI elements.

2.4.1. Java Elements

SrcML [33] is utilized to map the extracted Java code to an XML document. This document contains syntactic information about the source code as it wraps the text with meaningful tag names such as imports, classes, and methods. Although srcML has been successfully used in the field of SE where the entire code is available in the text format [34,35], it has never been used to the source code embedded in video frames. This poses two main limitations: (i) the OCRed code typically contains several errors since most OCR engines are sensitive to image quality and does not work well with images that contain source code [5,36,37] and (ii) each video frame contains only part of the Java element (i.e., incomplete element) since the code in programming screencasts is typically written on the fly. To detect OCR errors, I devised several steps to fix issues as follows.

First, Java elements are not recognized by srcML due to errors in the syntax of the OCRed code. There are several cases where this could occur such as if (i) the class name is not detected if the semicolon is missing before class definition, (ii) there is illegal space in the definition of a Java element (e.g., import statement), (iii) open and/or close curly brackets are missing. To solve this, I wrote a script to pre-process all OCRed codes and fix syntax errors as follows. First, I removed illegal extra spaces in import statements, class names, method names, and method calls. Second, I added a semicolon at the end of each statement if they do not exist or changed a colon to a semicolon where appropriate. Third, I fixed missing curly brackets or parentheses (i.e., some curly brackets are extracted as squares).

Second, most modern IDEs feature code suggestion where a popup window appears when developers write code to help them save time, and as such srcML extracts method names and calls from a popup window which are not part of the transcribed code. To detect if an extracted method name or call was part of the transcribed code, I checked if the full definition matches the correct format (i.e., syntactically correct). For example, a method name must have a modifier, return value, name, and parenthesis. Otherwise, it is considered as a method extracted from a popup window. It is worth mentioning that this step is essential as the extracted Java elements must be part of the transcribed code; otherwise, irrelevant Java elements are displayed and indexed.

Last, Java elements are incorrectly extracted due to OCR errors, which is very common when OCR is applied to an IDE to extract source code [5,36,37,38]. This is due to the fact that OCR is very sensitive to (i) the colorful background of the IDE, (ii) the low resolution of video frames, and (iii) the low quality of video frames. Let us denote the initial set of the extracted Java elements from a video as V = $\{e_{1,1},e_{1,2},..,e_{2,1},e_{2,2},..,e_{i,j},..,e_{m,n}\}$, in which $e_{i,j}$ is the jth extracted element from the ith frame (to be precise, the cropped editing window). As import statements, class names, method names, and method calls are extracted, I create one set for each element type (i.e., four different sets for each video). Each set contains unique elements and their counts. For example, if the same method name (e.g., onOptionsItemSelected) was extracted from a total of k frames, this means it has a count of k. The k value determines the confidence score of whether an element was correctly extracted or not. The similarity between each pair of elements was computed using a character-level similarity metric. Specifically, The normalized Levenshtein distance NLD (as described in Equation (2)) was used, and if the similarity between two elements is greater than a threshold, I discarded the element that was extracted fewer times (based on the percentage change). The rationale is that an incorrect (or incomplete) element is typically extracted fewer times than a similar correct element because OCR is less likely to confuse some characters. For example, if an element onOptionsLtemSelected was incorrectly extracted, appeared only k times, and was very similar to onOptionsItemSelected which appeared more times than k, the first term was assumed to be incorrectly extracted.

$$NLD(e_1,e_2)=1-\frac{LD(e_1,e_2)}{max(len\left(e_1\right),len\left(e_2\right))}$$

(2)

2.4.2. UI Elements

An android app contains a layout that has an invisible container such as RelativeLayout that defines visible UI elements to the users in which they can view or interact with the app. The UI elements can be declared based on (i) a list of pre-built UI elements provided by Android such as TextView, DatePicker, ProgressBar, and several others; (ii) third-party API libraries in which a user must import the required package and/or create a dependency entry in the build.gradle file; and (iii) a customized class in which a developer must create a custom Java code and define the behavior of the element. Fortunately, we can create a list of valid UI elements and check if an OCRed UI element is in that list to detect incorrectly extracted UI elements. I created a list of valid UI elements as follows. First, I utilized the official online Android documentation (https://developer.android.com/, accessed on 30 June 2021) to crawl all UI elements. Second, I used all import statements defined in Java files using the approach that extracts import statements, as well as a regular expression to find all third-party dependencies defined in the build.gradle file. Note that this file’s content was written on a code-editing window, and thereby we could easily extract its content and find UI elements defined as a dependency. Last, I applied OCR on the predicted project window to extract a list of file names, which contains names of potential user-defined UI elements. Ultimately, if an OCRed UI element was incomplete (e.g., Imagevie) or incorrect (e.g., viewelipper), it would not exist in a list of valid UI elements, and thereby it would be marked as an incorrect element.

2.5. Eliminating Duplicate GUI Screens

Extracting one frame per second from each video would result in duplicate frames, especially when it comes to video programming tutorials as the displayed contents in these videos are more static than other types [21]. The trained model locates and crops GUI screens from each frame, which results in redundant GUI screens. To detect redundant GUI screens, I adapted the mature image descriptor, the scale-invariant feature transform (SIFT) [39], to extract the robust features that are invariant to rotation, transformation, and scaling features. The key points are detected using a Hessian matrix approximation, in which SIFT assigns a distinctive descriptor (feature vector) to each keypoint.

To demonstrate the proposed approach for selecting unique UI frames, consider an input video as V = $\{f_1,f_2,\dots ,f_n\}$, in which each $f_i$ is processed through the first phase (see Section 2.2) to output the cropped $U{I}_i$, if any. Note that for each $f_i$ there could be many GUI screens or none at all. For each video, there is a new set that contains a list of the extracted GUIs denoted as UI = $\{U{I}_1,U{I}_2,\dots ,U{I}_m\}$, where m indicates the total number of extracted UIs from a video. Duplicate frames were detected and removed as follows. First, the feature vector was extracted from each $U{I}_i$ and saved in memory (to extract the features only once). Second, the euclidean distance between each extracted feature from $U{I}_i$ to that of $U{I}_j$ was computed, in which j ≥ i + 1 and j ≤ m. The two UIs were considered similar if the percentage of the matched descriptor was above a pre-defined threshold.

To assist the claim that SIFT performs the best in terms of removing duplicate UI frames from video programming tutorials, an empirical evaluation was performed on four different methods. In particular, I compared SIFT to another image descriptor, speeded-up robust features (SURFs) [39], and two other pixel-wise algorithms. I utilized two of the most popular pixel-wise comparisons (i) to detect mobile GUI changes using perceptual image differences (PIDs) [40,41], and (ii) to detect changes between programming video frames using the structural similarity index (SSIM) [14]. PID compares two images based on models of the human visual system [42], whereas SSIM compares the pixel intensities of two images [43]. The four methods removed duplicates based on a similarity threshold, and to ensure a fair comparison, I experimented with each method with different thresholds that range from 0.5 to 0.95 with a step size of 0.05.

3. Empirical Evaluation

This section introduces the proposed techniques in collecting the dataset, classifying and annotating the frames, and performing the empirical evaluation.

In this section, the empirical evaluation for VID2META is introduced in terms of (i) its accuracy in locating the ROIs within video frames and predicting the programming language, (ii) its effectiveness in extracting and fixing Java and UI elements, and (iii) its accuracy in removing only GUI screens. As such, the research questions (RQs) were formulated as follows:

RQ${}_1$ 

How accurately can VID2META classify and localize code-editing windows, project windows, and GUIs in Android programming screencasts?

RQ${}_2$ 

Does VID2META outperform previous work that localized code-editing window?

RQ${}_3$ 

How accurately can VID2META detect the programming language of the text inside the code-editing window?

RQ${}_4$ 

How accurately can VID2META extract and fix Java and UI elements?

RQ${}_5$ 

Which computer vision method performs the best in eliminating duplicate GUIs?

All RQs, except the fourth one, use the dataset collected to train the object detector (Section 2.2.1). The methodology to answer the first two RQs is explained in Section 3.1. The data collection processes to train the deep learning classifier to answer the RQ₃ are introduced in Section 3.2. Section 3.3 presents details about the process of collecting brand new videos (with source code) and creating the ground-truth Java and UI elements to answer RQ₄. Last, Section 3.4 shows the methodology used to evaluate different CV techniques to remove GUIs and answer RQ₅.

3.1. Methodology: Localizing Code-Editing Windows, Project Windows, and GUIs

To answer RQ₁, I used the annotated frames from the 20 videos and performed 10-fold cross-validation to cross video frames. To ensure that the averaged results of the 10-fold cross-validation converge to a stable value and determine the precision level accordingly, previous works [44,45] have identified the precision problem and suggested the repetition of 10-fold cross-validation until the results converge. As such, I determined a precision level of 0.01 in the experiments.. During each experiment, I split the videos into 80% training, 10% validation, and 10% testing. Thus, we can be certain that we are testing a new set of frames from untrained videos. I trained Faster R-CNN with the re-sized input image (600 × 1024) for a total of 10,000 iterations (i.e., validation loss was stable). Note that a new network weight was initialized for each fold to ensure that any information (features) learned from other folds was not used. Through back-propagation, the network weight was adjusted after each iteration using a momentum optimizer. A batch size of 16 with a stride of (8 × 8) was used as hyperparameters of the network layers. As part of data augmentation to improve the network’s learning performance, I randomly scaled some of the training regions. The implementation was based on Tensorflow API (https://github.com/tensorflow/tensorflow, accessed on 30 June 2021) on a machine with an Intel Xeon 3.40 GHz processor, 128 GB RAM, and a GeForce GTX 1080 GPU with 8 GB of memory for ∼150 h for all folds (10 different experiments).

To answer RQ₂, I compared VID2META to the open-source implementation of codemotion [46]. Unlike previous works [9,36], which only applied the Canny edge detector [47] to detect the code-editing window, codemotion proposed seven heuristics to filter out noisy edges and enhanced the detection process. The edges in the input images were detected using Canny and I followed the heuristic steps of codemotion to detect the code-editing window, which includes (i) detecting the endpoint of horizontal and vertical line segments using Hough transform [48], (ii) discarding the endpoints that are not close to the center of the image, (iii) connecting each point to another in such a way that closely forms a horizontal and vertical line, and (iv) detecting the surrounding rectangle in addition to the inner rectangle that was detected using the previous step. The output of these steps is a set of candidate rectangles, and we are interested in the ones that contain code. As the previous phase outputs a set of candidate rectangles, OCR was applied to each rectangle to predict the one that is most likely to contain the code.

The classification performance to answer RQ₁ was computed as follows. I used the standard metrics of precision, recall, and $F_1$ score. Precision is defined as $P=\frac{T_p}{T_p+F_p}$, Recall is computed as $R=\frac{T_p}{T_p+F_n}$, and the $F_1$ score is the harmonic mean of precision and recall, defined as $F=2·\frac{P·R}{(P+R)}$. I additionally evaluated the localization accuracy based on a standard intersection over union (IoU) metric [22,49,50]. Formally $Io{U}_{gt}^{pred}$ is defined as $Io{U}_{gt}^{pred}=\frac{Areaof(pred\cap gt)}{Areaof(pred\cup gt)}$, where $pred$ is the predicted location and $gt$ is the location of the ground-truth annotation. An IoU threshold determines the overall prediction performance (i.e., if an IoU is above a predefined threshold, the prediction is considered correct). Since a lower threshold would typically result in higher accuracy and a higher threshold would generally decrease the overall accuracy, I computed the results at a high IoU threshold (0.90). I computed average precision (AP) to show the overall accuracy of the model. AP is used as a standard metric in several object detection competitions [51,52]. The proposed model predicts the location of each key window with a confidence score that is sorted in descending order. Then, a prediction is considered correct if the IoU is greater than the threshold. I also computed the overall accuracy of the prediction by dividing the total correct predictions by the total number of predictions based on [email protected].

3.2. Methodology: Detecting Programming Language

VID2META is designed to detect the programming language written in the code-editing window of the IDE using two different approaches as follows.

First, the selected file from the project window was located and OCR was applied to extract the file’s name with its extension. Using the file extension, a prediction of the opened code-editing window’s content was made (e.g., Java or XML). As a prior step, we needed to evaluate the approach’s accuracy in finding the selected file in the first place. Therefore, I used all annotated PWs shown in Table 1 and fed them into the CTPN network. The CTPN outputs a prediction file that includes the coordinates of each text entry presented in a PW. I parsed each file and obtained each coordinate (xmin, xmax, ymin, and ymax). The vertical axes were only required as we needed to re-draw the bounding box to the full width of the PW (i.e., xmin = 0 and xmax = width). To this end, we ensured that each bounding box covers the entire background. Then, each pixel’s RGB values presented in the bounding box were obtained to find the most dominant one. Finally, the Δ*$E_{ab}$ distance was computed (see Equation (1)) between all pairs of any two dominant colors. Two dominant colors were considered similar if the distance between them was less than a threshold. The threshold was defined empirically by normalizing Delta (i.e., (Δ*$E_{ab}$/100) < 9).

Second, I employed a GuessLang network that uses a deep learning model with natural language processing techniques to guess a given text’s programming language. Although GuessLang has been trained to detect 30 programming languages, none of these languages were related to Android programming (e.g., Java and XML). Thereby, I fine-tuned GuessLang and trained it with two classes: Java and XML. I initially used the Android dataset collected by Businge et al. [32]. I opted to choose this dataset for several reasons: (i) the apps have already been published in the play store and (ii) it contains a diversity of apps that belong to more than 15 categories. I filtered out the files that contain code other than Java or XML, and ensured that Java and XML files are not empty. I selected a total of 15,000 Java and 15,000 XML files from 675 Android repositories. I split the dataset into 20,000 training and 10,000 testing sets. Then, GuessLang was trained on this dataset for a total of 50 epochs.

I evaluated the performance of detecting the programming language on a total of 4656 code-editing windows using the above two approaches. Note that I already annotated the content of each code-editing window with Java or XML (Section 2.2.2). For the first approach, I simply predicted the code-editing window’s programming language based on the file extension. To ensure the second approach’s practicality, I did not use the ground-truth coordinates of each code-editing window; instead, I used the cross-video code-editing predictions. I OCRed the predicted code-editing window and the entire frame. Then, I used the model to predict the OCRed editing window’s programming language and the OCRed entire frame. Note that I applied OCR on the editing window and the entire frame to compare their results. For the two approaches, the precision, recall, and F-Score were computed.

3.3. Methodology: Extracting Java and UI Elements

I used the 20 videos collected in Section 2.2.1 to determine the best thresholds for detecting incorrect Java and UI elements. The choice of these thresholds impacts the accuracy of detecting which element is correct or incorrect. For example, the NLD threshold determines the similarity between two elements, and one of them is deleted based on the NLD threshold and the count threshold (Section 2.4.1). The ground-truth Java and UI elements for each video were manually created. The ground-truth Java elements contain all unique import statements, class names, method names, and calls, whereas the ground-truth UI elements contain the names of all unique UI elements. I then applied OCR (https://www.google.com/drive/, accessed on 30 June 2021) on the predicted code-editing window and experimented with different NLD and count thresholds. In each experiment, the Jaccard similarity index (see Equation (3)) was computed, which quantified the similarity between a list of ground-truth elements and a list of predicted elements by dividing their intersection over their union. Eventually, the best NLD and count thresholds that maximized the Jaccard index for the 20 videos was chosen.

$$Jaccard(e_1,e_2)=\frac{|e_1\cap e_2|}{|e_1\cup e_2|}=\frac{|e_1\cap e_2|}{|e_1|+|e_2|-|e_1\cap e_2|}$$

(3)

To ensure robust evaluation and avoid data snooping biases on the thresholds, I manually collected a brand new 50 Android programming screencasts from YouTube. To facilitate the process of creating the ground-truth Java and UI elements, I chose Android videos that already have an attached source code. I manually validated the correctness of the attached source code for each video and made the changes accordingly. There were two cases: (i) videos do not show one or more files that were attached (I simply deleted those files) and (ii) some files were opened in the videos and were not attached (those files were manually transcribed). After I corrected the ground-truth source code files, I created the ground-truth information for Java and UI elements as follows. First, I employed srcML to map the Java file to an XML document. This XML document contains information about each element presented in the Java file, such as the import statement, class name, method name, and method call. As such, I used XPath Query to extract a ground-truth list for each element type (e.g., imports). Second, I created the ground-truth UI elements by applying a simple regex to the ground-truth XML files. To validate if an UI element is correct, the UI element has to be found in the Android API (https://developer.android.com, accessed on 30 June 2021) (i.e., I crawled all valid UI elements from the official Android documentation). I manually added third-party and custom-defined UI elements to each video’s ground-truth UI elements (I found only three third-party UI elements and one custom-defined in the 50 videos).

After preparing the ground-truth information, I processed each video as follows: (i) I extracted one frame per second, (ii) I used the trained model to predict and crop the code-editing window, and (iii) I applied OCR on those windows and categorize the content of the code-editing window using the approach discussed in Section 3.2.

To answer RQ₄, several experiments were performed as follows. First, I did not apply the approaches that fixed the extracted elements from the (i) OCRed predicted the code-editing window and (ii) OCRed the entire frame. Second, I applied the approaches (explained in Section 2.4) to pre-process the OCRed text from the (i) OCRed predicted code-editing window and (ii) the OCRed entire frame and fixed/detected errors in the extracted Java and XML elements. It should be noted that I performed some experiments on the entire frame to quantify the importance of detecting the code-editing window prior to the extracting process. Eventually, I computed the number of correct, incorrect, and missing elements for each experiment. In addition, I determined whether the results of the approaches in extracting Java elements from the predicted code-editing window were statistically significant compared to the results of blindly extracting the Java elements from the entire frame and from the predicted code-editing window. To perform the statistical analysis, I conducted the non-parametric Wilcoxon rank-sum test with a confidence interval of 95%. I used the Wilcoxon test as it only makes the assumptions of independence and equal variance, but it does not assume the data have a known distribution. I also measured the magnitude of the effect using Cliff’s delta ($\delta$) effect size [53]. Cliff’s $\delta$ ranges in the interval [$-1$, 1] and defined in Equation (4).

$$\mathit{effect}\mathit{size}=\left\{\begin{array}{cc}\mathit{negligible}\hfill & \mathrm{if}\mathit{Cliff}’\mathit{s}\left|\delta \right|\le 0.147\hfill \\ \mathit{small}\hfill & \mathrm{if}0.147<\mathit{Cliff}’\mathit{s}\left|\delta \right|\le 0.33\hfill \\ \mathit{medium}\hfill & \mathrm{if}0.33<\mathit{Cliff}’\mathit{s}\left|\delta \right|\le 0.474\hfill \\ \mathit{large}\hfill & \mathrm{if}\mathit{Cliff}’\mathit{s}\left|\delta \right|>0.474\hfill \end{array}\right.$$

(4)

3.4. Methodology: Eliminating Duplicate GUI Screens

To answer RQ${}_5$, I used ten randomly selected videos from the dataset that included GUI screens. I extracted one frame per second for each video and used the model to predict and crop the UI region. This results in a total of 3594 GUI frames, most of which are duplicates. To evaluate CV methods’ performance in detecting duplicate GUIs, we must have a set of ground-truth frames (i.e., a set of unique and sufficient GUI screens for each video). To create the ground-truth set of each video, another annotator (who was not aware of the CV methods) clustered similar UI frames together. An effective CV method must keep exactly one frame from each cluster. Keeping more than one frame from each cluster means that the algorithm failed in identifying duplicate frames, whereas not selecting a frame from a cluster means that the algorithm removed an important UI frame. I performed the empirical evaluation using SURF, SIFT, PID, and SSIM with different similarity thresholds between 0.50 and 0.95 with a step size of 0.05. Two images were considered similar if the similarity between them was greater than a pre-defined threshold.

The accuracy was measured in terms of precision, recall, and F-Score. The true positive (TP) indicates the total number of selected frames, in which each frame belongs to only one cluster. The false positive (FP) indicates the total number of duplicate frames (e.g., more than one frame was selected from a cluster). The false negative (FN) indicates the total number of unselected frames which should have been selected (e.g., none of the frames that belong to a cluster was selected). Formally speaking, precision is defined as $P=\frac{T_p}{T_p+F_p}$, in which ($T_p$ + $f_p$ = total number of selected frames). Recall is defined as $R=\frac{T_p}{T_p+F_n}$, in which ($T_p$ + $F_n$ = total number of clusters). The results are reported in terms of F-Score because it is a synthetic measure that considers both precision and recall. The average F-Score is computed over the ten videos.

4. VID2META’s Results

This section presents the results of the empirical evaluations to answer the five research questions.

4.1. RQ_1,2: Localizing Code-Editing Windows, Project Windows, and GUIs

To answer $R{Q}_1$, I computed the classification and localization accuracy and reported the results in

Table 2. The classification and localization results using 10-fold cross-validation. The average precision (AP) and the accuracy is computed with IoU@90.

Category	Classification			Localization
	Precision	Recall	F-Score	[email protected]	[email protected]
Code-editing window	0.99	0.99	0.99	0.96	0.97
Project window	0.99	0.99	0.99	0.95	0.96
GUI	0.94	0.99	0.96	0.94	0.94

. As it is very important that the predicted bounding box overlaps very well with the actual bounding box, I evaluated the localization accuracy at an IoU threshold of 0.90. Table 2 demonstrates the results and clearly shows that the model can accurately classify and locate the three windows. Even when I evaluated the model on unseen videos (i.e., cross-videos), it still performed very well, which indicates that the features of the code-editing windows and their shapes and outlines are predictable.

To answer $R{Q}_2$, I computed the IoU between the predicted bounding box of codemotion to that of the proposed approach for each input image that contained a code-editing window (i.e., 2590 Java and 2066 XML code-editing windows). The results of the evaluation are presented in boxplots (Figure 2). The proposed approach significantly outperformed code motion with median and mean IoU values of 97% and 96%, as compared to 62% and 64%, respectively. I inspected the results to see why codemotion failed to find the code-editing window of some frames accurately and found that even with some heuristics, Canny edge detection was still very sensitive to the frame’s noise and failed to detect some edges of the code-editing windows.

4.2. RQ₃: Detecting Programming Language

In total, the CTPN network predicted 104,852 bounding boxes for the text entry presented in 6731 PWs. The results in

Table 3. The classification results of finding the selected entry in the project window.

Category	Precision	Recall	F-Score
Selected	0.99	0.98	0.99
Unselected	0.99	1.00	0.99

show the precision, recall, and F-Score values for each category as follows. First, the precision of the selected category shows that the approach was successful in locating a high number of selected entries in the PW (i.e., a total of 6700 out of 6731). Second, the proposed approach that found unselected files with a 99% F-Score. Note that the background color of the unselected files ranges from white to light colors such as yellow, but since I defined a similarity threshold, they were very similar to the background of the other unselected file and different than the selected one (typically the selected one has a darker color). Thus, the result yields a high accuracy for both categories, which answered $R{Q}_3$. I also computed the results in terms of precision, recall, and F-Score for the two methods I used to predict the code-editing window’s content as follows. First, using only the selected file’s extension from the project window to determine the content (e.g., file.xml refers to the XML content), I obtained 89% for all precision, recall, and F-score for Java, whereas they were 90% for XML. I noticed that the results were highly impacted by the fact that (i) a PW sometimes is minimized, (ii) there was no selected file in the PW, and (iii) a programmer switch to another file through the tap that is typically located above the code-editing window. With these limitations, I introduced the results of the second method.

Table 4. The classification results of predicting the written language of the entire frame as compared to that of the code-editing window (CEW).

Categories	Entire Frame			Predicted CEW
	Precision	Recall	F-Score	Precision	Recall	F-Score
Java	1.00	0.58	0.73	0.99	1.00	0.99
XML	0.53	1.00	0.69	0.96	1.00	0.98

answers $R{Q}_4$ and shows that the trained model is very accurate in distinguishing between Java and XML. I evaluated the model using the extracted text from the predicted bounding box as well as from the entire frame. The results indicate that when the model guesses the entire frame’s programming language, a lot of noise is produced and yielded to a low F-Score. On the other hand, guessing the predicted bounding box’s programming language improved the accuracy by at least 26%.

4.3. RQ₄: Extracting Java and UI Elements

Table 5. The results of extracting Java elements from programming screencasts before and after applying the approaches to detect incorrect elements from the predicted code-editing window.

Java Element (GT Total)	Without Error Detection				With Error Detection
	#Correct	#Missing	#Incorrect	Jacc.	#Correct	#Missing	#Incorrect	Jacc.
Import Statements (185)	176	9	109	0.68	173	12	17	0.88
Class Names (60)	50	10	29	0.72	58	2	4	0.95
Method Names (176)	168	8	110	0.70	166	10	18	0.88
Method Calls (524)	529	13	2338	0.28	528	14	163	0.80

reveals the results of extracting Java elements from the predicted code-editing window before and after using the approaches to pre-process and detect incorrect elements. I computed the Jaccard similarity index (see Equation (3)) between the ground-truth Java elements and the extracted Java elements before and after error detection. Although the number of correctly extracted Java elements before error detection was relatively high, the number of the incorrect ones was also high, which yielded a low Jaccard similarity. This is due to the issues I previously discussed in Section 2.4.1, which were mainly caused by OCR errors. I applied the approaches to pre-process the OCRed code and detect any potential incorrect elements. As shown in Table 5, the number of incorrect Java elements was reduced significantly which indicated that the approaches could reliably detect and discard incorrect elements. The Jaccard similarity improved by 20%, 23%, 18%, and 52% for the import statements, class names, method names, and method calls, respectively. Trivially, I could extract the Java elements and detect OCR errors without even locating the code-editing window. Unfortunately, applying OCR on the entire frame and fixing OCR errors is a non-trivial challenge [5,36].

Table 6. The Jaccard index similarly between the set of the extracted Java elements from the entire frame and predicted code-editing window to that of the ground-truth annotation computed with and without error detection.

Java Element (GT Total)	Without Error Detection		With Error Detection
	Entire Frame	Code-Editing Window	Entire Frame	Code-Editing Window
Import Statements (185)	0.65 †	0.68 †	0.72 †	0.88
Class Names (60)	0.45 †	0.72 *	0.76 †	0.95
Method Names (176)	0.58 †	0.70 †	0.78 *	0.88
Method Calls (524)	0.25 †	0.28 †	0.64 †	0.80

The results were statistically significantly worse than the underlined results (0.05 significance level), ^† = with a medium or large effect size; ‘*’ = with a small effect size.

confirms this as I extracted Java elements from the entire OCRed code frame and from the OCRed predicted code-editing window and applied the proposed approaches to detect OCR errors. Notably, the lowest Jaccard index was used for those Java elements extracted from the entire frame without error detection, followed by those extracted from the code-editing window without error detection, followed by those extracted from the entire window with error detection. The highest Jaccard index was achieved when Java elements were extracted from the code-editing window and the proposed approach was applied to detect OCR errors (see underlined results in Table 6) which improved the overall accuracy by an average of 40%. The results with statistically significantly lower Jaccard scores (at the 0.05 significance level) compared to the results of the proposed approach are marked with a superscript symbol, where a cross (‘†’) indicates a medium or large effect size and an asterisk (‘*’) indicates a small effect size. The results illustrate that the proposed approach can produce statistically significant results on extracting all Java elements.

The results of extracting and detecting OCR errors in UI elements are as follows. First, a total of 72 XML files were presented in 41 videos (out of 50 videos). There were a total of 257 UI elements found in these XML files. Out of these elements, there were 253 names used from the Android API (e.g., button, fragment, and others), three widget names were used from third-party API (e.g., youtubeplayerview), and one was a custom-defined UI element. I computed the Jaccard similarity index to compare the extracted list of UI elements from the OCRed XML editing window with that of the ground-truth annotation. As the pre-defined list contained all possible valid UI elements from the three resources, an accuracy of 100% was obtained. If the approach was not applied to detect incorrect UI elements, the Jaccard index would be only 52%.

4.4. RQ₅: Eliminating Duplicate GUI Screens

Figure 3 answers the RQ${}_5$ and shows that SIFT outperformed every other method and achieved the best average F-Score that reached 84% with a similarity threshold of 60%. The second best-performing method was also based on image’s distinctive descriptor (SURF) and achieved a 78% score with a 55% similarity threshold. The highest achieved scores of PID and SSIM were 71% and 75%, respectively. The precision values of SIFT and SSIM were 91% and 78% at their best-performing similarity threshold. This means that SSIM kept more duplicate frames than SIFT as it compared the value of the same pixel position of two images rather than their features (i.e., inaccurate where there is a slight translation and very sensitive to noise). On the other hand, the SIFT descriptor is invariant to translation, rotation, and scaling. I also computed each algorithm’s running time applied on 100 GUI frames (i.e., the average time complexity is $\theta \left(n^2\right)$). SSIM performed the fastest with only 68 s, whereas PID was the slowest with 553 s. The running time SURF and SIFT were similar to 126 s and 129 s, respectively.

5. Related Work

In this section, the related work is presented (which is also summarized in

Table 7. This table surveys the related work on code extraction, mining programming screencasts, and the works that applied transfer learning in classifying/localizing image artifacts.

Code Extraction from Programming Screencasts
Paper	# of Videos	Programming Language	Frame Comparison	Detect Frame with Code	OCR Engine
Ponzanelli et al. [9,54,55]	150	Java	Pixel-Wise	Heuristics	Tesseract
Yadid et al. [36]	40	Java (Android)	N/A	Canny-Edge	Tesseract
Moslehi et al. [56]	5	PHP (WordPress)	OCR	N/A	Google Vision
Khandwala et al. [5]	20	Java, Javascript, Python, PHP, Ruby, C, C#	N/A	Canny-Edge	Tesseract
Bao et al. [38,57]	50	Java	Pixel-Wise	CNN Classifier	Tesseract
Khormi et al. [37]	300	Java, Python, C#	N/A	Manual	GDrive, ABBYY, GOCR, Tesseract, OCRAD
Mining Programming Screencasts
Paper	# of Videos	Type of Information		Aim (Method)
Zhao et al. [14]	50	Video Frames		Action Detection (CNN)
Bao et al. [58,59]	N/A (29-h Videos)	Video Frames		Extract HCI data (CV)
Bao et al. [60,61]	3	Operating System		Tutorial Authoring System (OS)
Ott et al. [13]	40	Video Frames		Frame Classification (CNN)
Ott et al. [20]	100	Video Frames		Prog. Lang. Detection (CNN)
Parra et al. [7]	75	Video Metadata		Automatic Tagging (IR and NLP)
Poche et al. [62]	12	Video Comments		Comments Classification (ML)
McGowan et al. [63]	N/A (Two Java Courses)	Student’s Engagement		Analysis of Viewing Behaviours (User Study)
Chen et al. [64]	30	IDE		Combine Slides With IDE (Web Technologies)
Ellis et al. [65]	48	Video Frames, Audio, and Metadata		Link Video to Other Artifacts (Knowledge Modelling)
Mining Image-Based Software Artifacts Using CNNs
Paper	Image Artifact	Classification or Localization?		Network
Zhao et al. [14]	Programming Video Frames	Classification		Inception Resnet V2 [11]
Ott et al. [13,20] and Bao et al. [38,57]	Programming Video Frames	Classification		VGG-16 [27]
Best et al. [66] and Ott et al. [67]	UML Diagrams	Classification		VGG-16
Moran et al. [68]	Mobile UI Components	Classification		AlexNet [69]
Chen et al. [70]	Mobile UI Components	Classification		Resnet [71]
Zhao et al. [72]	Mobile Image-Based Buttons	Classification		Resnet-101 [71]
Bernal-Cárdenas et al. [16]	Touch Indicator in Mobile UI	Both		Faster R-CNN [12] and AlexNet
Chen et al. [17]	Mobile UI Components	Both		Faster R-CNN
White et al. [73]	Mobile UI Components	Both		YOLOv2 [74]
Chen et al. [75]	Mobile UI Components	Both		Faster R-CNN, YOLOv3 [76], CenterNet [77]

) on (i) code extraction from programming screencast, (ii) mining programming screencasts, and (iii) mining image-based software artifacts using CNNs.

5.1. Code Extraction from Programming Screencasts

Several works were devoted to extracting the source code from the programming screencast by applying OCR on video frames [5,36,37,38,54]. Yadid et al. [36] and Bao et al. [38] trained a statistical language model [78] on source code and used the consolidating code across frames to detect and correct OCR errors. Ponzanelli et al. [9,55] used an island parser to cope with the noise of the OCRed code. None of these works extracted any information that is related to Android layout such as UI elements. Furthermore, I extracted fine-grained information such as API calls which would be more applicable and efficient to index and/or complement videos with.

5.2. Mining Programming Screencasts

A sizable body of work has been dedicated to facilitating the process of extracting the source code from a video programming screencast [6,9,13,36]. Ott et al. [13] proposed the use of a CNN-based classifier to predict the existence of source codes in a video frame. This work was dedicated to only solving this classification problem of the entire frame without focusing on locating the source code’s bounding box or extracting the source code. Several works locate the source code’s bounding box using the Canny edge detection algorithm [79], where it was applied to the entire IDE frame, and several heuristics were proposed to guess which bounding box contains the source code [5,36,54]. By replicating codemotion [5] that uses the Canny edge algorithm, the results show that the proposed approach is more accurate. Furthermore, none of these works extracted UI elements from Android videos.

5.3. Mining Image-Based Software Artifacts Using CNNs

It has been increasingly common within the software engineering community to fine-tune CNN models to solve image classification problems. Zhao et al. [14] proposed ActionNet that analyzes the changes that occur when a developer writes a code on-the-fly throughout a programming screencast using Inception Resnet V2 [11]. Ott et al. [13] proposed another approach that classifies video frames obtained from programming screencasts into four different categories using VGG-16 [27]. On the same line, Bao et al. [38] trained the top layers of the VGG network to classify frames into code and non-code categories. Yang et al. [80] proposed an approach to help developers find live-streamed programming videos by automatically classifying video frames to live-streamed video or pre-recoded video. The work most closely related to VID2META is proposed by Alahmadi et al. [15,81], who leveraged an object detector to locate the code bounding box. While the approach of locating code bounding box is similar to VID2META, VID2META made another contribution towards finding, extracting, and fixing Java and UI elements that would better complement Android programming screencasts.

6. Conclusions and Future Work

In this paper, VID2META is proposed, an automated approach that takes an Android video tutorial as input in which it analyzes the visual and textual content of the video to extract meaningful code elements and GUIs. Extracting those elements enables developers to (i) search code elements, (ii) decide if a video is relevant, and (iii) allow navigation throughout a video or to outside API resources. I performed an extensive evaluation of VID2META on a total of 70 videos (20 videos to evaluate the localization accuracy and 50 videos to assess the accuracy of extracting correct code elements). The results illustrate that VID2META can accurately locate the code-editing window and extract code elements with an average accuracy above 90%.

Some potential extensions can be derived from this work to advance the state-of-the-art in mining programming screencasts further. As such, I plan to create a concise and meaningful tags of video programming based on the code information presented in the frames, audio, and metadata of videos. In addition, I will study whether the current search engine (e.g., YouTube) can be improved by indexing the extracted metadata for mobile programming screencasts.