POP-ON: Prediction of Process Using One-Way Language Model Based on NLP Approach
Abstract
:1. Introduction
2. Related Work
2.1. Event Log
2.2. Process Mining for Manufacturing
2.3. Prediction of Process
2.4. Transformer
3. POP-ON
3.1. One-Way Language Model
3.2. Data Pre-Processing
3.2.1. One-Hot Encoding
3.2.2. Min-Max Scaling
3.3. Proposed Prediction Model
4. Performance Analysis
4.1. Experiment Environment
4.2. Datasets
4.3. Evaluation Metrics
- : True positive. Number of cases classified as class i, and the actual class is also i.
- : False positive. Number of cases classified as class i, but the actual class is not i.
- : True negative. Number of cases not classified as class i, and the actual class is also not i.
- : False negative. Number of cases not classified as class i, but the actual class i.
- l: Number of all classes.
- : Number of class i.
- n: Total number of datasets, thus .
4.4. Comparative Analysis
4.5. Hyperparameter Optimization
- ;
- ;
- ;
- .
4.6. The Need for an Attribute Linear Layer
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Nyhuis, P.; Wiendahl, H.P. Fundamentals of Production Logistics: Theory, Tools and Applications; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2008. [Google Scholar]
- Ebert, C.; Abrahamsson, P.; Oza, N. Lean software development. IEEE Softw. 2012, 5, 22–25. [Google Scholar] [CrossRef] [Green Version]
- Barad, M.; Sapir, D.E. Flexibility in logistic systems—Modeling and performance evaluation. Int. J. Prod. Econ. 2003, 85, 155–170. [Google Scholar] [CrossRef]
- Becker, T.; Intoyoad, W. Context aware process mining in logistics. Procedia Cirp 2017, 63, 557–562. [Google Scholar] [CrossRef]
- Van der Aalst, W.M.; Schonenberg, M.H.; Song, M. Time prediction based on process mining. Inf. Syst. 2011, 36, 450–475. [Google Scholar] [CrossRef] [Green Version]
- Grigori, D.; Casati, F.; Castellanos, M.; Dayal, U.; Sayal, M.; Shan, M.C. Business process intelligence. Comput. Ind. 2004, 53, 321–343. [Google Scholar] [CrossRef] [Green Version]
- Duan, L.; Da Xu, L. Business intelligence for enterprise systems: A survey. IEEE Trans. Ind. Inform. 2012, 8, 679–687. [Google Scholar] [CrossRef]
- Mun, J.; Jeong, J. Design and Analysis of RUL Prediction Algorithm Based on CABLSTM for CNC Machine Tools. In Proceedings of the 2020 7th International Conference on Soft Computing & Machine Intelligence (ISCMI), Stockholm, Sweden, 14–15 November 2020; pp. 83–87. [Google Scholar]
- Philipp, P.; Jacob, R.; Robert, S.; Beyerer, J. Predictive Analysis of Business Processes Using Neural Networks with Attention Mechanism. In Proceedings of the 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), Fukuoka, Japan, 19–21 February 2020; pp. 225–230. [Google Scholar]
- Philipp, P.; Georgi, R.X.M.; Beyerer, J.; Robert, S. Analysis of control flow graphs using graph convolutional neural networks. In Proceedings of the 2019 6th International Conference on Soft Computing & Machine Intelligence (ISCMI), Johannesburg, South Africa, 19–20 November 2019; pp. 73–77. [Google Scholar]
- Curtis, B.; Kellner, M.I.; Over, J. Process modeling. Commun. ACM 1992, 35, 75–90. [Google Scholar] [CrossRef]
- Bandara, W.; Gable, G.G.; Rosemann, M. Factors and measures of business process modelling: Model building through a multiple case study. Eur. J. Inf. Syst. 2005, 14, 347–360. [Google Scholar] [CrossRef] [Green Version]
- Van Der Aalst, W.; Adriansyah, A.; De Medeiros, A.K.A.; Arcieri, F.; Baier, T.; Blickle, T.; Bose, J.C.; Van Den Brand, P.; Brandtjen, R.; Buijs, J.; et al. Process mining manifesto. In Proceedings of the International Conference on Business Process Management, Clermont-Ferrand, France, 29 August 2011; Springer: Berlin/Heidelberg, Germany, 2011; pp. 169–194. [Google Scholar]
- Der Aalst, V.; Mining, W.P. Discovery, Conformance and Enhancement of Business Processes; Springer: Berlin/Heidelberg, Germany, 2011. [Google Scholar]
- Becker, T.; Lütjen, M.; Porzel, R. Process maintenance of heterogeneous logistic systems—A process mining approach. In Dynamics in Logistics; Springer: Berlin/Heidelberg, Germany, 2017; pp. 77–86. [Google Scholar]
- Evermann, J.; Rehse, J.R.; Fettke, P. Predicting process behaviour using deep learning. Decis. Support Syst. 2017, 100, 129–140. [Google Scholar] [CrossRef] [Green Version]
- Tax, N.; Verenich, I.; La Rosa, M.; Dumas, M. Predictive business process monitoring with LSTM neural networks. In Proceedings of the International Conference on Advanced Information Systems Engineering, Essen, Germany, 12–16 June 2017; Springer: Berlin/Heidelberg, Germany, 2017; pp. 477–492. [Google Scholar]
- Leontjeva, A.; Conforti, R.; Di Francescomarino, C.; Dumas, M.; Maggi, F.M. Complex symbolic sequence encodings for predictive monitoring of business processes. In Proceedings of the International Conference on Business Process Management, Rio de Janeiro, Brazil, 18–22 September 2016; Springer: Berlin/Heidelberg, Germany, 2016; pp. 297–313. [Google Scholar]
- Márquez-Chamorro, A.E.; Resinas, M.; Ruiz-Cortés, A.; Toro, M. Run-time prediction of business process indicators using evolutionary decision rules. Expert Syst. Appl. 2017, 87, 1–14. [Google Scholar] [CrossRef]
- Mehdiyev, N.; Evermann, J.; Fettke, P. A novel business process prediction model using a deep learning method. Bus. Inf. Syst. Eng. 2020, 62, 143–157. [Google Scholar] [CrossRef] [Green Version]
- Peters, M.E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; Zettlemoyer, L. Deep contextualized word representations. arXiv 2018, arXiv:1802.05365. [Google Scholar]
- Radford, A.; Narasimhan, K.; Salimans, T.; Sutskever, I. Improving Language Understanding by Generative Pre-Training. 2018. Available online: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf (accessed on 5 October 2020).
- Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog 2019, 1, 9. [Google Scholar]
- Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv 2018, arXiv:1810.04805. [Google Scholar]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017, 30, 5998–6008. [Google Scholar]
- Belinkov, Y.; Glass, J. Analysis methods in neural language processing: A survey. Trans. Assoc. Comput. Linguist. 2019, 7, 49–72. [Google Scholar] [CrossRef]
- Li, J.; Wang, H.J.; Bai, X. An intelligent approach to data extraction and task identification for process mining. Inf. Syst. Front. 2015, 17, 1195–1208. [Google Scholar] [CrossRef]
- Ethayarajh, K. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. arXiv 2019, arXiv:1909.00512. [Google Scholar]
- Kaplan, R.M. A method for tokenizing text. In Inquiries into Words, Constraints and Contexts; CSLI Publication: Stanford, CA, USA, 2005; pp. 55–64. [Google Scholar]
- Han, J.; Pei, J.; Kamber, M. Data Mining: Concepts and Techniques; Elsevier: Amsterdam, The Netherlands, 2011. [Google Scholar]
- Harris, D.; Harris, S. Digital Design and Computer Architecture; Morgan Kaufmann: Burlington, MA, USA, 2010. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 770–778. [Google Scholar]
- Ba, J.L.; Kiros, J.R.; Hinton, G.E. Layer normalization. arXiv 2016, arXiv:1607.06450. [Google Scholar]
- Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 2014, 15, 1929–1958. [Google Scholar]
- Van Dongen, B. BPI Challenge 2012. Dataset. 2012. Available online: https://www.win.tue.nl/bpi/doku.php?id=2012:challenge (accessed on 5 October 2020).
- Steeman, W. BPI Challenge 2013. Dataset. 2013. Available online: https://www.win.tue.nl/bpi/doku.php?id=2013:challenge (accessed on 5 October 2020).
- Verenich, I. BPI Challenge Helpdesk. Dataset. 2016. Available online: https://data.mendeley.com/datasets/39bp3vv62t/1 (accessed on 5 October 2020).
- Pasquadibisceglie, V.; Appice, A.; Castellano, G.; Malerba, D. Using convolutional neural networks for predictive process analytics. In Proceedings of the 2019 International Conference on Process Mining (ICPM), Aachen, Germany, 24–26 June 2019; pp. 129–136. [Google Scholar]
- Camargo, M.; Dumas, M.; González-Rojas, O. Learning accurate LSTM models of business processes. In Proceedings of the International Conference on Business Process Management, Vienna, Austria, 1–6 September 2019; Springer: Berlin/Heidelberg, Germany, 2019; pp. 286–302. [Google Scholar]
- Hinkka, M.; Lehto, T.; Heljanko, K. Exploiting Event Log Event Attributes in RNN Based Prediction. In Data-Driven Process Discovery and Analysis; Springer: Berlin/Heidelberg, Germany, 2018; pp. 67–85. [Google Scholar]
- Khan, A.; Le, H.; Do, K.; Tran, T.; Ghose, A.; Dam, H.; Sindhgatta, R. Memory-augmented neural networks for predictive process analytics. arXiv 2018, arXiv:1802.00938. [Google Scholar]
- Evermann, J.; Rehse, J.R.; Fettke, P. A deep learning approach for predicting process behaviour at runtime. In Proceedings of the International Conference on Business Process Management, Rio de Janeiro, Brazil, 18–22 September 2016; Springer: Berlin/Heidelberg, Germany, 2016; pp. 327–338. [Google Scholar]
- Di Mauro, N.; Appice, A.; Basile, T.M. Activity prediction of business process instances with inception cnn models. In Proceedings of the International Conference of the Italian Association for Artificial Intelligence, Rende, Italy, 19–22 November 2019; Springer: Berlin/Heidelberg, Germany, 2019; pp. 348–361. [Google Scholar]
Dataset | Number of Cases | Number of Events | Number of Activities | Number of useful Attributes |
---|---|---|---|---|
BPI’2012 A | 13,087 | 60,849 | 10 | 1 |
BPI’2012 O | 5015 | 31,244 | 7 | 1 |
BPI’2012 W Completed | 9658 | 72,413 | 6 | 1 |
BPI’2013 Incidents | 7554 | 65,533 | 13 | 8 |
BPI’2013 Closed Problems | 1487 | 6660 | 7 | 2 |
Helpdesk | 4580 | 21,348 | 14 | 1 |
Metrics | Formula |
---|---|
Accuracy | |
Precision | |
Recall | |
F-measure | |
MCC (Matthews Correlation Coefficient) |
Approach | BPI 2012 A | BPI 2012 W Complete | BPI 2012 O Closed Problems | BPI 2013 Incidents | BPI 2013 | Helpdesk | |
---|---|---|---|---|---|---|---|
Pasquadibisceglie et al. | [38] | 71.47 | 66.14 | 77.51 | 24.35 | 33.10 | 65.84 |
Tax et al. | [17] | 77.75 | 66.80 | 81.22 | 65.57 | 67.50 | 75.06 |
Camargo et al. | [39] | 78.82 | 65.19 | 85.13 | 60.62 | 68.01 | 76.51 |
Hinkka et al. | [40] | 79.27 | 67.24 | 85.51 | 61.14 | 77.95 | 77.90 |
Khan et al. | [41] | 75.62 | 75.91 | 84.48 | 55.57 | 64.34 | 69.13 |
Evermann et al. | [42] | 74.44 | 65.38 | 79.20 | 55.66 | 68.15 | 70.07 |
Mauro et al. | [43] | 78.09 | 65.01 | 81.52 | 56.97 | 71.09 | 74.77 |
POP-ON | 79.49 | 81.36 | 92.15 | 78.36 | 79.89 | 75.53 |
Dataset | Numerical Attributes | Categorical Attributes |
---|---|---|
BPI 2012 A | time:elapsed, case:AMOUNT_REQ | org:resource |
BPI 2012 W Complete | time:elapsed, case:AMOUNT_REQ | org:resource |
BPI 2012 O | time:elapsed, case:AMOUNT_REQ | org:resource |
BPI 2013 Closed Problems | time:elapsed | org:group org:role concept:name product impact resource country organization country organization involved |
BPI 2013 Incidents | time:elapsed | org:group org:role concept:name product impact resource country organization country organization involved |
Helpdesk | time:elapsed | - |
Hyperparameter | Description |
---|---|
Dimension of Embedding | |
Dimension of Feed-Forward Network | |
Dimension of Attribute Linear Layer | |
Number of Transformer Decoder Layers | |
Number of Attention Head |
Hyperparameter Set | Accuracy | |||||
---|---|---|---|---|---|---|
1 | 1 | 16 | 128 | 128 | 0.8296 | |
2 | 1 | 16 | 128 | 128 | 0.8593 | |
4 | 1 | 16 | 128 | 128 | 0.8984 | |
8 | 1 | 16 | 128 | 128 | 0.9064 | |
1 | 2 | 32 | 256 | 256 | 0.8934 | |
2 | 2 | 32 | 256 | 256 | 0.9039 | |
4 | 2 | 32 | 256 | 256 | 0.9121 | |
8 | 2 | 32 | 256 | 256 | 0.9169 | |
1 | 4 | 64 | 512 | 512 | 0.9051 | |
2 | 4 | 64 | 512 | 512 | 0.9110 | |
4 | 4 | 64 | 512 | 512 | 0.9171 | |
8 | 4 | 64 | 512 | 512 | 0.9194 | |
1 | 8 | 128 | 1024 | 1024 | 0.9125 | |
2 | 8 | 128 | 1024 | 1024 | 0.9196 | |
4 | 8 | 128 | 1024 | 1024 | 0.9197 | |
8 | 8 | 128 | 1024 | 1024 | 0.9217 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Moon, J.; Park, G.; Jeong, J. POP-ON: Prediction of Process Using One-Way Language Model Based on NLP Approach. Appl. Sci. 2021, 11, 864. https://doi.org/10.3390/app11020864
Moon J, Park G, Jeong J. POP-ON: Prediction of Process Using One-Way Language Model Based on NLP Approach. Applied Sciences. 2021; 11(2):864. https://doi.org/10.3390/app11020864
Chicago/Turabian StyleMoon, Junhyung, Gyuyoung Park, and Jongpil Jeong. 2021. "POP-ON: Prediction of Process Using One-Way Language Model Based on NLP Approach" Applied Sciences 11, no. 2: 864. https://doi.org/10.3390/app11020864
APA StyleMoon, J., Park, G., & Jeong, J. (2021). POP-ON: Prediction of Process Using One-Way Language Model Based on NLP Approach. Applied Sciences, 11(2), 864. https://doi.org/10.3390/app11020864