Automated Source Code Generation and Auto-completion Using Deep Learning: Comparing and Discussing Current Language-Model-Related Approaches

In recent years, the use of deep learning in language models, text auto-completion, and text generation has made tremendous progress and gained much attention from the research community. Some products and research projects claim that they can generate text that can be interpreted as human-writing, enabling new possibilities in many application areas. Among the different areas related to language processing, one of the most notable in applying this type of modeling is the processing of programming languages. For years, the Machine Learning community has been researching in this Big Code area, pursuing goals like applying different approaches to auto-complete generate, fix, or evaluate code programmed by humans. One of the approaches followed in recent years to pursue these goals is the use of Deep-Learning-enabled language models. Considering the increasing popularity of that approach, we detected a lack of empirical papers that compare different methods and deep learning architectures to create and use language models based on programming code. In this paper, we compare different neural network (NN) architectures like AWD-LSTMs, AWD-QRNNs, and Transformer, while using transfer learning, and different tokenizations to see how they behave in building language models using a Python dataset for code generation and filling mask tasks. Considering the results, we discuss the different strengths and weaknesses of each approach and technique and what lacks do we find to evaluate the language models or apply them in a real programming context while including humans-in-the-loop.


Introduction
We are digitally surrounded by computational Language Models (LMs) that guide us while writing to reduce the user effort, suggest different options for words/sentences to enhance our style, or fix our grammatical/correctness errors accurately (Kannan et al. 2016;Bryant and Briscoe 2018;Ghosh and Kristensson 2017). Many of the keys we press while writing on a keyboard act as part of the inputs to compose new datasets for those models that try to shape how we communicate with others. Nevertheless, does it happen in the same way when we write code? Succinctly, yes. According to some recent surveys found in the literature (Allamanis et al. 2018;Chen, Le, and Babar 2020), the Natural * Intern at IBM Quantum at the time of writing this paper Language Processing (NLP) subfield related to source code language includes examples of LMs used to suggest code to users (Nguyen and Nguyen 2015;Bielik, Raychev, and Vechev 2016;Cruz-Benito et al. 2018), generate automated source code (Oda et al. 2015;Tiwang, Oladunni, and Xu 2019), translate code between different languages (Nguyen, Nguyen, and Nguyen 2013), fill missing parts (Proksch, Lerch, and Mezini 2015;Li et al. 2017), and related tasks similar to those found in the broader NLP research field (Donahue, Lee, and Liang 2020;Fedus, Goodfellow, and Dai 2018).
In the NLP broad field, there is a more general understanding of the different characteristics related to the natural languages -since there exist many fields of research related to human languages-, from the minimal representation units of the language to what is a typical word of the language or if a word is a neologism or not. Source code languages share some syntax similarities with spoken languages, but it does not have the same restrictions in the sense of common words or neologisms (Allamanis et al. 2015;Karampatsis and Sutton 2019), indentation or other syntax restrictions. Every programming language has indeed reserved words and symbols to denote different actions, resources, or syntax. However, there is an essential part of the source code that is only limited by the programmer's imagination, the conventions existing, or the guides for good practices. As (Karampatsis and Sutton 2019) claim (Karampatsis and Sutton 2019), [...] traditional language models limit the vocabulary to a fixed set of common words. For code, this strong assumption has been shown to have a significant negative effect on predictive performance [...] In that paper, Karampatsis and Sutton 2019 present how segmenting words into subword units can be used to improve the modeling of source code. In the same way, other researchers (Ganin et al. 2016;Kim et al. 2016;Karpathy 2016) dug in the issues of representing source code vocabulary with a similar emphasis on modeling word by using sub-word units and trying to envision their importance when using neural networks. Nevertheless, how that word segmentation affect the accuracy or the appropriateness of the code generated or auto-completed in some modern LM using deep learning approaches? That kind of question raises the main goal for this paper: discover what kinds of associa-tions between different modern neural network architectures and tokenization models produce the best results when creating LMs to generate and auto-complete source code.
Trying to answer that central question and pursue that goal, this research aims to conduct a set of experiments combining different deep neural network architectures with different tokenization and pre-trained models to get better results in terms of accuracy and simple human evaluation for code generation and auto-completion (for example, filling the blanks).
The paper includes the following sections: Section 1 introduces the problem and questions discussed in the paper. Section 2 presents the materials and methods used for the research. Section 3 describes results achieved during the research, while section 4 discusses these findings and the implications of the results as appropriate. Finally, Section 5 presents some conclusions.

Materials and methods
We have trained a set of deep neural networks using different architectures, tokenization techniques, software libraries, and hardware setup to develop this research work. Following, we introduce the different materials and methods employed for that purpose.
Concerning the materials, we showcase two different kinds of materials used: hardware and software. In the case of the hardware materials, to run most of the different software and neural networks training, we used a computer running using Ubuntu Linux 18.04 LTS Bionic Beaver (64 bits) equipped with two Nvidia Tesla V100 GPUs (Nvidia CUDA version 10.1), a CPU with 16 cores Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz, 120 gigabytes of RAM, and 120 Gigabytes for the primary disk (HDD). In the case of the software materials, the dataset used for the experiments is the Python dataset included in the "GitHub CodeSearchNet Challenge dataset" (Husain et al. 2019). The full dataset includes 2 million (comment, code) pairs from open source libraries. In the Python portion, there are about 11 million of Python code sentences (more details about it in the code repository related to the paper). The software libraries and packages used primarily during the research were the following: FastAI (Howard and Gugger 2020), Google Senten-cePiece (Kudo and Richardson 2018), and HuggingFace's Transformers . All the supporting materials related to this paper are publicly available in a GitHub repository (Cruz-Benito and Vishwakarma 2020).
Regarding the methods, the pre-processing applied to the dataset included removing most of the code comments and autoformatting the code according to the PEP-8 Python style guide using the autopep8 1 package. On the side of the neural networks employed, we should highlight three main architectures: ASGD Weight-Dropped LSTM (AWD-LSTM) (Merity, Keskar, and Socher 2017), and a variation of the previous one using Quasi Recurrent Neural Networks (QRNNs)  instead of LSTMs (Long Short-Term Memory) (Hochreiter and Schmidhuber 1997). The third one is some of the most popular architectures in 1 https://pypi.org/project/autopep8/ NLP right now: Transformers (Vaswani et al. 2017) due to its performance and recent results. In this case, we used three different transformer models: GPT-2 (Radford et al. 2019), RoBERTa (Liu et al. 2019), and BERT (Devlin et al. 2018).
In the case of the AWD-LSTM networks, we have been using the FastAI-provided base models pre-trained using Wikitext-103 dataset ). There are no default pre-trained models in the QRNN version of those networks, so we trained from scratch. As we introduced, regarding the Transformer architectures, we have been using three standard pre-trained models as a basis: GPT-2, BERT, and RoBERTa. In each case, the exact pre-trained model used were gpt2, bert-base-cased, and roberta-base. All of them were retrieved from HuggingFace's model hub. We choose GPT-2 since it is a causal transformer (unidirectional), so it can predict the next token in a sequence and, thus, comparable with the AWD-LSTM and AWD-QRNN used in the other experiments. In BERT and RoBERTA, we used them to see how a masked modeling approach can perform autocomplete source codes. In that case, we did not use them for text generation as in the other experiments, since BERT and ROBERTA are not intended usually for text generation, although they can generate text (more diverse but slightly worse in quality) (Wang and Cho 2019). As the reader can infer from the previous explanations, we followed a transfer learning approach similar to as other researchers usually present in existing literature (Howard and Ruder 2018;Ruder et al. 2019;Chronopoulou, Baziotis, and Potamianos 2019;Eisenschlos et al. 2019) We used pre-trained models on different corpora, all of them mostly based on English texts, as the knowledge basis to later fine-tune the models on the tasks we want (source code generation and autocompletion) using the GitHub CodeSearchNet dataset.
Considering the tokenization techniques, for every AWD-LSTM and AWD-QRNN, we tested the following types of tokens: word, unigram, char, and byte-pair encoding (BPE) (Sennrich, Haddow, and Birch 2015) -albeit some studies show that BPE is suboptimal for pretraining (Bostrom and Durrett 2020)-. In the case of the Transformer models, we used the default ones from the pre-defined models: Wordpiece method (Schuster and Nakajima 2012) for BERT and BPE over raw bytes instead of Unicode characters for GPT-2 and RoBERTa.
All the deep-neural-networks-related source code is developed mainly using the FastAI library (versions 1.0.61 and 2 dev 0.0.21). In the case of the different tokenization techniques employed to research with the AWD-LSTMs and AWD-QRNNs, we have been using Google Sentencepiece (Kudo and Richardson 2018) to integrate its tokenizer -instead of the default Spacy tokenizer (Honnibal and Montani 2017)-following a similar approach than the one used in (Czapla, Howard, and Kardas 2018). In the case of the development of Transformer architectures to see how they perform filling the blanks and generating texts, we used Hug-gingFace's Transformers library combined with FastAI v2 (following the example given by FastAI library 2 ) as included on the code repository that supports this paper. To train the neural networks, we have used some techniques worth to mention (all the details are in the code repository). To find the most appropriate learning rate to use automatically, we used the function lr find provided by FastAI following the proposal by (Smith 2017). To pursue a faster convergence, we schedule the learning rate described in (Smith and Topin 2019) by using the one cycle policy (fit one cycle) in FastAI. Considering the transfer learning technique used, we trained the first "one cycle" on the top of the existing pretrained model to later unfreeze all the model layers and do a more extended training (10-30 epochs) to improve the final results. Regarding other training details, in general, we used the default parameters from FastAI, except for a fixed multiplier to control all the dropouts (drop mult) in AWD-LSTMs and AWD-QRNNs set to 0.3 because of some heuristics discovered during testing this research. For more information about the training setup and software details, please refer to the repository that supports this paper and the FastAI documentation.

Results
This section presents the results achieved after the full training of the selected NN architectures with the different tokenization models.
As outlined during the "Materials and Methods" section, we have trained three main neural network architectures using the same dataset. AWD-LSTM and AWD-QRNN were trained using different tokenization models -word, unigram, BPE, and char-, while the other architecture trained was the Transformer, using three different models (GPT-2, BERT, and RoBERTa). All the AWD-LSTM and AWD-QRNN versions were trained using one epoch to fit the model's head and fine-tuned for 30 epochs. Meanwhile, the Transformer networks were trained equally for one epoch to fit the head and fine-tune the models for 10 epochs. Table 1 displays the final metrics for the different NNs at the end of the training. In the same way, Figure 1 shows the evolution of the accuracy of each model during the training. Also, the figures 2 and 3 show the evolution of the training loss and validation loss along the training epochs.  Table 1, and Figure 1, in the case of neural networks intended for automated source code generation -AWD-LSTM, AWD-QRNN, and Transformer GPT-2-, the overall NNtokenization model combination that performed better in the case of accuracy metrics was the AWD-LSTM with char tokenization (accuracy 0.779633). The second one was the GPT-2 transformer model -BPE over raw bytes tokenization- About how the pre-training and transfer learning affects the results, the two top results regarding the accuracy come up from pre-trained models in the English language (0.779633, and 0.743738), yet the third-best result was from a non-pre-trained network (0.736358). Comparing the similar networks, the average (mean) accuracy of the AWD-LSTM pre-trained versions is 0.603031 (standard deviation -std-of 0.123144), while the average accuracy of AWD-QRNN non-pre-trained versions was 0.582587 (std of 0.103107). The only combination NN-tokenization model that worked worse when was pre-trained was the one with the word tokenization.
Regarding the losses, it is worth to comment that the AWD-LSTM char, AWD-QRNN char, as well as the three transformer models (GPT-2, BERT, RoBERTa) should be trained for more epochs (or with a higher learning rate) since the model is not fitted enough; thus, the training loss is higher than the validation loss (Table 1, figures 2 and 3).
To conclude this results section, we introduce some examples of how the different neural networks generated source code or auto-completed it. In this case, we show only on those with the best accuracy results: AWD-LSTM char, AWD-QRNN char, and GPT-2 for generation, and the two transformer models related to auto-completion: BERT and RoBERTa. In the case of the source code generation, every model faced the same challenges (Cruz-Benito and Vishwakarma 2020). Those are the results when any of them try to generate five different sentences with the input from pan- AWD-LSTM char "from pandas import defaults as _rp pandas =" "from pandas import DataFrameEncoder if self" "from pandas import OrderedDict else: passed" "from pandas import sequence if type(seq1) i" "from pandas import conditional_insertion as" AWD-QRNN char "from pandas import webbrowser self.login fr" "from pandas import __http_args or [ 'Pipeli" "from pandas import iterations is not None p" "from pandas import service params to servic" "from pandas import difference if not isinst"

GPT-2
"from pandas import time, np\n " "from pandas import pandas from time.time.datetime import Date\n with n" "from pandas import gtk, os\n from pandas_utils import pandas_utils\n import pylint" "from pandas import wcpy\n import cpy_context as cpy_context\n " "from pandas import gkpy\n """\n ... pass\n kwargs = cg" To assess the generation, we do not focus on the semantics of the imports used or whether they are part of the Pandas library or not, but in the language's correctness. In general, from a Python perspective, the outputs from GPT-2 are better. It includes line breaks, indentation, and fair use of multiple inputs in one sentence (except in one of the outputs). The AWD-LSTM and AWD-QRNN failed to auto-generate an import sentence appropriately, or at least, in the usual manner as regular users do. As a final comment on this, the other models trained failed on similar issues, plus they do not get enough semantic context related to the Pandas library. In the case of auto-completion, we observe that almost any auto-completion is right. The sentences that can be more accurate regarding the Python language are the ones in which the mask token has been replaced by white space or by a dot. They are not correct but are closer to be right comparing to the other ones. One interesting thing is that BERT assigns a very high score to a predicted mask, which is not correct, and very low scores to the other possible solutions (also incorrect). In the case of RoBERTa, it gives lower scores to all the solutions, yet also fails on the correctness: the second sentence predicted (score 0.222) can be closer to be right than the first one (score 0.622).

Discussion
It is easy to assert, considering the obtained results, that the tokenization model used profoundly affects the results when generating automated source code. It is consistent with the existing literature, as discussed during the introduction section 2019; 2016; 2016; 2016. Despite that, some interesting results must be discussed.
First of all, our overall results fit into the literature. Subword tokenization works better in the case of modeling source code as 2019 stated. Every result obtained is consistent in that sense. Even more, as 2016 envision, char tokenization probably should the best option to try by default when dealing with LMs and source code. Furthermore, according to the results achieved, models such as GPT-2 -using a tokenization model based on BPE over raw bytes-can outperform LSTM/QRNN models like those we tested to grasp better the internals of a source code language. As showcased during the results, even if GPT-2 was not the best model in terms of accuracy, it gave better-predicted outputs than the other ones selected in the comparison. Also, it may be noted that the comparison of the outputs generated by AWD-LSTM char, AWD-QRNN char, and GPT-2 can be polished to be more accurate if we continue the training until we get a better result in terms of the balance between training loss and validation loss (they seem to be underfitting, as we commented in the results section). As future work, it would be great to check, related to the previous consideration, if that improved performance in the case of GPT-2 is because it is a much bigger and more pre-trained model (163,037,184 parameters against 36,491 in the case of AWD-LSTM and AWD-QRNN models), or it is related to the dataset used for this research, if it is related to both causes or if it is related to other issues.
Regarding the pre-training and transfer learning, every pre-trained model (on English-based datasets) got better accuracy than its non-pre-trained counterparts except in word tokenization. It seems to be strongly related to the statements we introduced at the beginning of this paper, citing (Allamanis et al. 2015;Karampatsis and Sutton 2019) about the source code does not have the same restrictions in the sense of common words or neologisms. In this sense, the conclusion comes into our mind rapidly: if we consider source code words in "word" units, they probably will not fit in the fixed set of words used in a human language like the English. So, the LM's knowledge acquired during the pre-training is not entirely valid when we get out of that fixed set of words that compose a language. Most words in the source code language can be observed as neologisms by the LM, and thus, it needs to incorporate them and their relationships into the learned knowledge. In the case of the sub-word units, the LM can be less sensible to the neologisms. Even more, potentially, it could be more robust the more divided a word is since the set of bytes or chars is more straightforward than the chunks present in bigger constructions or information units. Some ways of going deeper into this research concerning the pre-training effect over LMs modeling source code could be related to research on what kind of language pretraining is more effective for general source code or specific programming languages.
About the tests made generating source code or filling in the blanks using the trained LMs, we think that, in general, the textual results obtained are not so good, yet they are informative of how LMs are working and how they can be improved. One of the things that can explain these results is the dataset used. In this case, we used a public dataset that was not designed to train language models per se. The dataset is also relatively small to train a big LM that could accomplish appropriately challenging tasks like generating source code or auto-completing it. If the dataset would be bigger in training big LMs focused on modeling the Python language, or even not so big but focused on some specific libraries or Python aspects, the results may be better. For sure, recent examples of LMs -for example GPT-3 (Brown et al. 2020)that claim to be able to produce quality textual outputs even in contexts in which they were not trained, use enormous datasets and some similar NN architectures so that those approaches can be relevant for a general context like a broad LM for Python language.
Related to evaluating the code generated or filled, we observed in the literature different approaches (Celikyilmaz, Clark, and Gao 2020). In the context of LMs modeling source code, many papers and software libraries devoted to the research on translating between source code languages typically evaluate text generation using methods like BLEU (Papineni et al. 2002) or variants like SacreBLEU (Post 2018). Other papers like (Tiwang, Oladunni, and Xu 2019), rely on the accuracy to assess the performance of an LM based on deep learning. Even more, some models can work on solving different tasks that are part of existing benchmarks (Weston et al. 2015) or are evaluated, also checking their perplexity (similarly to those that evaluate the model using the accuracy). We assessed the models using accuracy and evaluated the textual outputs from the models based on our prior knowledge, because, as we outlined in the results section, relying only on the accuracy was not accurate while also checking the textual output from the models. In terms of accuracy, the best models were not the best ones regarding their textual outputs, raising some concerns on evaluating LMs. We think that these kinds of assessments are not enough (even our procedure) for tasks like auto-completion or source code generation, so one of the future research lines is improving the evaluation of LMs for source code. Based on some existing ideas in broad NLP, there are many opportunities to explore in that sense. From behavioral testing introduced recently (Ribeiro et al. 2020), to a humancentric evaluation of the models (Celikyilmaz, Clark, and Gao 2020) with particular emphasis on reproducible and un-biased assessments, or combinations of both.

Conclusions
This paper compares how different approaches to tokenization models, deep neural network architectures, pre-trained models, and transfer learning affect the results from language models used to generate source code or auto-complete software pieces. During the research, we studied different NNs architectures like AWD-LSTM, AWD-QRNN, and Transformer to seek which kind of them work better with different tokenization models (word, unigram, BPE, and char). Also, we compared the pre-training effect on the results given by LMs after training them and fine-tuning them via transfer learning to work with other languages (English language to Python programming language). As a result of this work, we find that in small LMs (like our AWD-LSTM and AWD-QRNN models), the tokenization using char-sized chunks works better than using other tokenization models. In bigger models like the Transformer GPT-2, the accuracy was slightly worse, yet it raised better results on the source code generation tests (even using another tokenization approach like BPE over raw bytes). Regarding source code auto-completion, we tested some transformer models like BERT and RoBERTA. While their accuracy was above any of the other models, they did not perform very well when performing the tasks proposed in our tests. In general, we find that pre-trained models work better, even if they were not trained initially for a programming language like Python (our models were pre-trained using the English language). Finally, and related to the evaluation of tasks like automated source code generation and source code auto-completion, we raise some concerns about the methods used in the literature and propose some research lines to work on the future.