A Novel Framework for Natural Language Interaction with 4D BIM †
Abstract
1. Introduction
2. Literature Review
2.1. Role of AI in Construction Management
2.2. Natural Language Processing in BIM and Construction Scheduling
2.3. Real-Time Schedule Updating
2.4. Contributions of the VISA4D Framework
- Field-based interaction methods; unlike conventional BIM interfaces that rely exclusively on office-based interaction, VISA4D establishes direct communication channels between construction site operations and digital scheduling platforms by enabling on-site personnel to interact with the 4D BIM environment through intuitive natural language commands.
- Multimodal communication protocols; the implementation of both voice and text-based interaction distinguishes VISA4D from existing construction schedule interfaces. This dual-channel approach accommodates diverse operational environments, from construction sites where voice commands allow hands-free operation while performing physical tasks to office environments where text-based interaction might be preferred.
- Facilitation of 4D schedule manipulation; the system enables complete schedule management operations through natural language commands in a 4D BIM environment, such as task addition/deletion, modification of time and date parameters, and task status adjustments.
- Real-time synchronization with BIM; the instant integration of natural language commands with 4D BIM visualizations represents a significant advancement over traditional methodologies that typically involve substantial processing delays between field observations and schedule updates.
3. Research Methodology
3.1. Problem Identification
3.2. Design and Development
3.3. Implementation
3.4. Evaluation
4. System Architecture and Implementation Details
4.1. User Input Layer (gui.py)
4.2. Nlp_processor.py
4.3. Task_manager.py
- Task creation: this operation facilitates adding new tasks to the TimeLiner, the 4D scheduling tool in Navisworks.
- Status updates: this function allows for the modification of task statuses, enabling dynamic transitions between states such as “not started” and “in progress”.
- Date modifications: this capability provides the flexibility to reschedule tasks by modifying their start or end dates, ensuring accurate project timelines.
- Task deletion: this operation permits the removal of tasks from the system, effectively managing the project scope and maintaining data integrity.
4.4. Navisworks_api.py
4.5. NavisworksServer.cs
4.6. Navisworks TimeLiner Custom Plugin (Addin.cs)
5. Schedule Maintenance and Updating Using the VISA4D Tool
6. Validation and Verification of the VISA4D Tool
6.1. User Tests
6.2. Assessing the Reliability and Performance of the VISA4D Tool
- Task creation: to test the ability of the system to recognize requests to add new construction tasks.
- Status update: to verify accuracy in recognizing status change requests.
- Date update: to test the system’s ability to recognize different date patterns and distinguish between start and end date modifications
- Task deletion: to test the system’s ability to recognize task removal requests.
- Intent accuracy: percentage of commands where the predicted intent matched the expected intent.
- Task name extraction accuracy: percentage of commands where the extracted task name correctly matched the expected task name.
- “Update date” commands misclassified as “update status” (three cases);
- Unknown commands interpreted as “create task” (three cases);
- “Delete task” commands misclassified as “create task” (two cases);
- One “update status” command classified as unknown.
6.3. Discussion of Validation Results and Limitations
6.3.1. Sample Size and Participant Demographics
6.3.2. Speech Recognition Challenges
6.3.3. Technical Infrastructure Constraints and Integration Challenges
6.3.4. Intent Recognition Accuracy Implications
7. Discussion
8. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
Abbreviations
VISA4D | Voice-Integrated Scheduling Assistant for 4D BIM |
NLP | Natural Language Processing |
BIM | Building Information Modeling |
AEC | Architecture, Engineering, and Construction |
3D BIM | Three dimensional BIM |
4D BIM | Four Dimensional BIM |
AI | Artificial Intelligence |
LLMs | Large Language Models |
CPM | Critical Path Method |
LPS | Last Planner System |
PERT | Program Evaluation and Review Technique |
ML | Machine Learning |
KBES | Knowledge-Based Expert System |
NLU | Natural Language Understanding |
DL | Deep Learning |
GPT | Generative Pretrained Transformer |
BERT | Bidirectional Encoder Representations from Transformers |
DSR | Design Science Research |
API | Application Programming Interface |
JSON | JavaScript Object Notation |
GUI | Graphical User Interface |
APS | Autodesk Platform Services |
FEDS | Framework for Evaluation in Design Science |
IT | Information Technology |
Appendix A
Questions | Options/Response |
---|---|
Q1: Ease of use of VISA4D Tool | The tool was very easy to use |
The tool was easy to use | |
The tool was neither easy nor difficult to use | |
The tool was difficult to use | |
The tool was very difficult to use | |
Q2: Command recognition of the VISA4D Tool | The tool consistently recognized my commands accurately |
The tool often recognized my commands accurately | |
The tool sometimes recognized my commands accurately | |
The tool rarely recognized my commands accurately | |
The tool never recognized my commands accurately | |
Q3: Text Handling Effectiveness of the VISA4D Tool | The tool handled text inputs very effectively. |
The tool handled text inputs effectively. | |
The tool handled text inputs neither effectively nor ineffectively. | |
The tool handled text inputs ineffectively. | |
The tool handled text inputs very ineffectively. | |
Q4: Voice Handling Effectiveness of the VISA4D Tool | The tool handled voice inputs very effectively. |
The tool handled voice inputs effectively. | |
The tool handled voice inputs neither effectively nor ineffectively. | |
The tool handled voice inputs ineffectively. | |
The tool handled voice inputs very ineffectively. | |
Q5: VISA4D’s Response Time of Commands | The tool’s response time to commands was always satisfactory. |
The tool’s response time to commands was usually satisfactory. | |
The tool’s response time to commands was sometimes satisfactory and sometimes unsatisfactory. | |
The tool’s response time to commands was usually unsatisfactory. | |
The tool’s response time to commands was always unsatisfactory. | |
Q6: Errors in Command Execution by the VISA4D Tool | The tool executed commands without any errors. |
The tool rarely executed commands with errors. | |
The tool sometimes executed commands with errors. | |
The tool often executed commands with errors. | |
The tool always executed commands with errors. | |
Q7: User-Friendliness of the VISA4D Tool’s Interface | The tool’s interface is very user-friendly. |
The tool’s interface is generally user-friendly. | |
The tool’s interface is neither user-friendly nor user-unfriendly. | |
The tool’s interface is somewhat user-unfriendly. | |
The tool’s interface is very user-unfriendly. | |
Q8: Ease of Learning to Use the VISA4D Tool | Learning to use the tool was very easy. |
Learning to use the tool was easy. | |
Learning to use the tool was neither easy nor difficult. | |
Learning to use the tool was difficult. | |
Learning to use the tool was very difficult. | |
Q9: What do you like the most about VISA4D? | |
Q10: What challenges did you face while using VISA4D? | |
Q11: Do you have any general comments or feedback about VISA4D? | |
Q12: Are there any additional features you would like to see in VISA4D? | |
Q13: What improvements would you suggest for VISA4D? |
References
- Sacks, R.; Eastman, C.; Lee, G.; Teicholz, P. BIM Handbook: A Guide to Building Information Modeling for Owners, Managers, Designers, Engineers and Contractors, 3rd ed.; Wiley: Hoboken, NJ, USA, 2018. [Google Scholar]
- Staub-French, S.; Khanzode, A. 3D and 4D modeling for design and construction coordination: Issues and lessons learned. Jit Constr. 2007, 12, 381–407. [Google Scholar]
- Mazars, T.; Adel, F. Chronographical spatiotemporal dynamic 4D planning. Autom. Constr. 2020, 112, 103076. [Google Scholar] [CrossRef]
- Doukari, O.; Seck, B.; Greenwood, D. The efficient generation of 4D BIM construction schedules: A case study of the Nanterre 2 CESI project in France. Front. Built Environ. 2022, 8, 998309. [Google Scholar] [CrossRef]
- Milat, M.; Knezić, S.; Sedlar, J. Resilient scheduling as a response to uncertainty in construction projects. Appl. Sci. 2021, 11, 6493. [Google Scholar] [CrossRef]
- Kiziltas, S.; Akinci, B. The Need for Prompt Schedule Update by Utilizing Reality Capture Technologies: A Case Study. In Proceedings of the Construction Research Congress: Broadening Perspectives, San Diego, CA, USA, 5–7 April 2005. [Google Scholar]
- Das, S.; Das, D. Natural Language Processing (NLP) Techniques: Usability in Human-Computer Interactions. In Proceedings of the 2024 6th International Conference on Natural Language Processing (ICNLP), Xi’an, China, 22–24 March 2024. [Google Scholar]
- Boton, C.; Kubicki, S.; Halin, G. 4D/BIM Simulation for Pre-Construction and Construction Scheduling. Multiple Levels of Development within a Single Case Study. In Proceedings of the Creative Construction Conference, Kraków, Poland, 21–24 June 2015. [Google Scholar]
- Zhou, W.; Whyte, J.; Sacks, R. Construction safety and digital design: A review. Autom. Constr. 2012, 22, 102–111. [Google Scholar] [CrossRef]
- Sloot, R.N.F.; Heutink, A.; Voordijk, J.T. Assessing usefulness of 4D BIM tools in risk mitigation strategies. Autom. Constr. 2019, 106, 102881. [Google Scholar] [CrossRef]
- Trebbe, M.; Hartmann, T.; Dorée, A. 4D CAD models to support the coordination of construction activities between contractors. Autom. Constr. 2015, 39, 83–91. [Google Scholar] [CrossRef]
- Son, H.; Kim, C.; Kwon Cho, Y. Automated schedule updates using as-built data and a 4D building information model. J. Manag. Eng. 2017, 33, 4017012. [Google Scholar] [CrossRef]
- Wang, X.; Truijens, M.; Hou, L.; Wang, Y.; Zhou, Y. Integrating augmented reality with building information modeling: Onsite construction process controlling for liquefied natural gas industry. Autom. Constr. 2014, 40, 96–105. [Google Scholar] [CrossRef]
- Sciuto, A.; Saini, A.; Forlizzi, J.; Hong, J. Hey Alexa, What’s Up?: A Mixed-Methods Studies of In-Home Conversational Agent Usage. In Proceedings of the 2018 Designing Interactive Systems Conference (DIS ’18), Hong Kong, China, 9–13 June 2018. [Google Scholar]
- Hoy, M.B. Alexa, Siri, Cortana, and More: An Introduction to Voice Assistants. Med. Ref. Serv. Q. 2018, 37, 81–88. [Google Scholar] [CrossRef]
- Bello, S.; Oyedele, L.; Akanbi, L.; Bello, A.-L. Cloud computing for chatbot in the construction industry: An implementation framework for conversational-BIM voice assistant. Digit. Eng. 2025, 5, 100031. [Google Scholar] [CrossRef]
- Jaff, L.; Garg, S.; Guven, G. AI-Supported Real-time Schedule Updating and Maintenance. In Proceedings of the Joint Canadian Society for Civil Engineering Construction Specialty and Construction Research Congress (CSCE-CRC), Montreal, QC, Canada, 28–31 July 2025. [Google Scholar]
- Asadzadeh, A.; Arashpour, M.; Li, H.; Ngo, T.; Bab-Hadiashar, A.; Rashidi, A. Sensor-Based Safety Management. Autom. Constr. 2020, 113, 103128. [Google Scholar] [CrossRef]
- Gado, N.G. AI revolutionizes construction management “Building smarter, safer, and efficiently addressing industry challenges”. Eng. Res. J. 2024, 183, 330–344. [Google Scholar] [CrossRef]
- Khobragade, A.N.; Maheswari, N.; Sivagami, M. Analyzing the housing rate in a real estate informative system: A prediction analysis. Int. J. Civ. Eng. Technol. 2018, 9, 1156–1164. [Google Scholar]
- Poh, C.Q.X.; Ubeynarayana, C.U.; Goh, Y.M. Safety leading indicators for construction sites: A machine learning approach. Autom. Constr. 2018, 93, 375–386. [Google Scholar] [CrossRef]
- Ajayi, A.; Oyedele, L.; Owolabi, H.; Akinade, O.; Bilal, M.; Davila Delgado, J.M.; Akanbi, L. Deep learning models for health and safety risk prediction in power infrastructure projects. Risk Anal. 2020, 40, 2019–2039. [Google Scholar] [CrossRef]
- Amer, F.; Koh, H.Y.; Golparvar-Fard, M. Automated methods and systems for construction planning and scheduling: Critical review of three decades of research. J. Constr. Eng. Manag. 2021, 147, 2093. [Google Scholar] [CrossRef]
- Barreiro, J.; Boyce, M.; Do, M.; Frank, J.; Iatauro, M.; Kichkaylo, T.; Morris, P.; Ong, J.; Remolina, E.; Smith, T. EUROPA: A Platform for AI Planning, Scheduling, Constraint Programming, and Optimization. In Proceedings of the International Conference on Automated Planning and Scheduling, Atibaia, São Paulo, Brazil, 25–29 June 2012. [Google Scholar]
- Aljebory, K.M.; QaisIssam, M. Developing AI Based Scheme for Project Planning by Expert Merging Revit and Primavera Software. In Proceedings of the 16th International Multi-Conference on Systems, Signals & Devices (SSD), Istanbul, Turkey, 21–24 March 2019. [Google Scholar]
- Lin, W.Y. Prototyping a chatbot for site managers using building information modeling (BIM) and natural language understanding (NLU) techniques. Sensors 2023, 23, 2942. [Google Scholar] [CrossRef]
- Singh, J.; Anumba, C.J. Real-time pipe system installation schedule generation and optimization using artificial intelligence and heuristic techniques. J. Inf. Technol. Constr. 2022, 27, 173–190. [Google Scholar] [CrossRef]
- Ma, Y.; Qiao, E. Visualized Management System of Prefabricated Construction and Installation Engineering Based on BIM and Machine Learning. In Proceedings of the IEEE International Conference on Industrial Application of Artificial Intelligence (IAAI), Harbin, China, 24–26 December 2021. [Google Scholar]
- Zheng, J.; Fischer, M. Dynamic prompt-based virtual assistant framework for BIM information search. Autom. Constr. 2023, 155, 105067. [Google Scholar] [CrossRef]
- You, H.; Ye, Y.; Zhou, T.; Zhu, Q.; Du, J. Robot-enabled construction assembly with automated sequence planning based on ChatGPT: RoboGPT. Buildings 2023, 13, 1772. [Google Scholar] [CrossRef]
- Fernandes, D.; Garg, S.; Nikkel, M.; Guven, G. A GPT-powered assistant for real-time interaction with Building Information Models. Buildings 2024, 14, 2499. [Google Scholar] [CrossRef]
- Prieto, S.A.; Mengiste, E.T.; García de Soto, B. Investigating the Use of ChatGPT for the scheduling of construction projects. Buildings 2023, 13, 857. [Google Scholar] [CrossRef]
- Golparvar-Fard, M.; Peña-Mora, F.; Savarese, S. Automated progress monitoring using unordered daily construction photographs and IFC-based Building Information Models. J. Comput. Civ. Eng. 2015, 29, 4014025. [Google Scholar] [CrossRef]
- Liu, S.-S.; Shih, K.-C. Construction rescheduling based on a manufacturing rescheduling framework. Autom. Constr. 2009, 18, 715–723. [Google Scholar] [CrossRef]
- Rebolj, D.; Babič, N.Č.; Magdič, A.; Podbreznik, P.; Pšunder, M. Automated construction activity monitoring system. Autom. Constr. 2008, 22, 493–503. [Google Scholar] [CrossRef]
- Kim, C.; Son, H.; Kim, C. Automated construction progress measurement using a 4D building information model and 3D data. Autom. Constr. 2013, 31, 75–82. [Google Scholar] [CrossRef]
- Nabavi, A.; Ramaji, I.; Sadeghi, N.; Anderson, A. Leveraging natural language processing for automated information inquiry from building information models. J. Inf. Technol. Constr. 2023, 28, 266–285. [Google Scholar] [CrossRef]
- Singh, K.; Pal, A.; Kumar, P.; Lin, J.; Hsieh, S.-H. Prospects of integrating BIM and NLP for automatic construction schedule management. In Proceedings of the 40th International Symposium on Automation and Robotics in Construction, IAARC, Chennai, India, 5–7 July 2023. [Google Scholar]
- Peffers, K.; Tuunanen, T.; Rothenberger, M.; Chatterjee, S. A design science research methodology for information systems research. J. Manag. Inf. Syst. 2007, 24, 45–77. [Google Scholar] [CrossRef]
- Venable, J.R.; Baskerville, R. Eating our own cooking: Toward a more rigorous design science of research methods. Electron. J. Bus. Res. Methods 2012, 10, 141–153. [Google Scholar]
- Venable, J.; Pries-Heje, J.; Baskerville, R. FEDS: A framework for evaluation in design science research. Eur. J. Inf. Syst. 2016, 25, 77–89. [Google Scholar] [CrossRef]
- Hevner, A. A three cycle view of design science research. Scand. J. Inf. Syst. 2007, 19, 4. [Google Scholar]
- Brocke, J.; Winter, R.; Hevner, A.; Maedche, A. Accumulation and evolution of design knowledge in design science research—A journey through time and space. J. Assoc. Inf. Syst. 2020, 21, 520–544. [Google Scholar]
- Gregor, S.; Hevner, A. Positioning and presenting design science research for maximum impact. MIS Q. 2013, 37, 337–356. [Google Scholar] [CrossRef]
- Baskerville, R.; Baiyere, A.; Gregor, S.; Hevner, A.; Rossi, M. Design science research contributions: Finding a balance between artifact and theory. J. Assoc. Inf. Syst. 2018, 19, 358–376. [Google Scholar] [CrossRef]
- Sein, M.; Henfridsson, O.; Purao, S.; Rossi, M.; Lindgren, R. Action design research. MIS Q. 2011, 35, 37–56. [Google Scholar] [CrossRef]
- Sonnenberg, C.; Brocke, J. Evaluations in the Science of the Artificial—Reconsidering the Build-Evaluate Pattern in Design Science Research. In Lecture Notes in Computer Science LNCS; Springer: Berlin/Heidelberg, Germany, 2012; pp. 381–397. [Google Scholar]
- Prat, N.; Wattiau, I.; Akoka, J. Artifact evaluation in information systems design science research? A holistic view. In Proceedings of the Pacific Asia Conference on Information Systems, PACIS, Chengdu, China, 24–28 June 2014. [Google Scholar]
- Jaff, L.; Garg, S.; Guven, G. Voice-Integrated-Schdueling-Assistant-for-4D-BIM-VISA4D, GitHub. Available online: https://github.com/SGradient/Voice-Integrated-Schdueling-Assistant-for-4D-BIM-VISA4D- (accessed on 30 April 2025).
Feature | VISA4D | Nabavi et al. (2023) [37] | Singh et al. (2023) [38] | Aljebory and Issam (2019) [25] |
---|---|---|---|---|
Primary Technology | Natural language interface for 4D BIM | NLP for automated information inquiry from BIM | BIM and NLP integration for automatic scheduling | AI-based project planning by connecting BIM with Primavera |
Field Usability | Both site and office | Office-based only | Office-based only | Office-based only |
Real-time updates to schedule | Real-time synchronization with 4D BIM | N/A | Not focused on real-time operations | Not focused on real-time updates |
User Accessibility | Intuitive natural language interaction requiring minimal training | User-friendly but requires some training | User-friendly but limited in generating schedule | Requires expert knowledge to operate |
Integration with BIM | Direct integration with 4D BIM | Integration with BIM | Proposes integration but lacks implementation details | No direct integration with BIM |
Schedule Manipulation | Complete schedule operations through natural commands | Limited to information retrieval | Focuses on schedule generation rather than manipulation | Limited to schedule generation |
Multimodal Communication | Voice and text interaction | Voice and text interaction | Text-based interaction with potential voice interaction | Text-based interaction only |
Commands | Sample Queries | |
---|---|---|
Create task | Add Foundation Piling task | Can you create Electrical Wiring |
Create a new task for Grade Beams Pour #3 | Please add Plumbing Installation | |
Set up Elevator Raft | Add task Exterior Siding | |
I need a new task for Feature Wall Pours | Create a new task for Door 2000 Installation | |
Can you create Main Floor Glazing Installation | Set up Stair 1000 | |
Please add Second Floor Drywall Installation | I need a new task for Slab A | |
Add task Roof Door Installation | Can you create Structural Framing | |
Create a new task for Concrete Pouring | Please add Final Inspection | |
Set up HVAC Installation | Add task Grade Beams Pour #5 | |
I need a new task for Window Installation | Create a new task for Main Floor Door Installation | |
Update Status | Update the status of Foundation Piling to complete | Update the status of Electrical Wiring to in progress |
Change Grade Beams Pour #3 status to in progress | Change Plumbing Installation status to complete | |
Set Elevator Raft as on hold | Set Exterior Siding as in progress | |
Mark Feature Wall Pours complete | Mark Door 2000 Installation complete | |
Update Main Floor Glazing Installation to in progress | Update Stair 1000 to on hold | |
Can you update Second Floor Drywall Installation to on hold | Can you update Slab A to in progress | |
Please change Roof Door Installation status to complete | Please change Structural Framing status to complete | |
Update Concrete Pouring to in progress | Update Final Inspection to on hold | |
Set HVAC Installation as complete | Set Grade Beams Pour #5 as in progress | |
Mark Window Installation on hold | Mark Main Floor Door Installation complete | |
Delete Task | Delete Foundation Piling task | Delete the Electrical Wiring task |
Remove Grade Beams Pour #3 task | Could you please remove Plumbing Installation task | |
Delete task Elevator Raft from the TimeLiner | I’d like to remove Exterior Siding | |
Remove task painting | Please remove Door 2000 Installation | |
Delete Main Floor Glazing Installation | Would you delete Stair 1000 task for me | |
Can you remove Second Floor Drywall Installation | Can you remove task Slab A | |
Delete task Roof Door Installation | I need to terminate Structural Framing | |
Remove the Concrete Pouring task | Please remove the Final Inspection task | |
I would like you to remove HVAC Installation | Delete the task Grade Beams Pour #5 | |
Remove Window Installation | I want you to cancel Main Floor Door Installation | |
Update Date | Update start date for Foundation Piling to April 5 | Update start date for Electrical Wiring to March 28 |
Update finish date for Grade Beams Pour #3 to May 12 | Update finish date for Plumbing Installation to July 18 | |
Update start date for Elevator Raft to June 20 | Update start date for Exterior Siding to May 20 | |
Update end date for Feature Wall Pours to July 10 | Update end date for Door 2000 Installation to June 28 | |
Update start date for Main Floor Glazing Installation to March 15 | Update start date for Stair 1000 to April 10 | |
Update end date for Second Floor Drywall Installation to August 22 | Update finish date for Slab A to August 5 | |
Update start date for Roof Door Installation to April 30 | Update start date for Structural Framing to March 25 | |
Update finish date for Concrete Pouring to May 8 | Update end date for Final Inspection to September 30 | |
Update start date for HVAC Installation to June 15 | Update finish date for Grade Beams Pour #5 to April 22 | |
Update end date for Window Installation to September 3 | Update start date for Main Floor Door Installation to May 15 |
Intent | Precision (%) | Recall (%) | F1-Score | Support |
---|---|---|---|---|
Create Task | 77 | 100 | 87 | 17 |
Delete Task | 100 | 90 | 95 | 20 |
Unknown | 0 | 0 | 0 | 3 |
Update Date | 100 | 85 | 92 | 20 |
Update Date | 86 | 95 | 91 | 20 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Jaff, L.; Garg, S.; Guven, G. A Novel Framework for Natural Language Interaction with 4D BIM. Buildings 2025, 15, 1840. https://doi.org/10.3390/buildings15111840
Jaff L, Garg S, Guven G. A Novel Framework for Natural Language Interaction with 4D BIM. Buildings. 2025; 15(11):1840. https://doi.org/10.3390/buildings15111840
Chicago/Turabian StyleJaff, Larin, Sahej Garg, and Gursans Guven. 2025. "A Novel Framework for Natural Language Interaction with 4D BIM" Buildings 15, no. 11: 1840. https://doi.org/10.3390/buildings15111840
APA StyleJaff, L., Garg, S., & Guven, G. (2025). A Novel Framework for Natural Language Interaction with 4D BIM. Buildings, 15(11), 1840. https://doi.org/10.3390/buildings15111840