A Novel Framework for Natural Language Interaction with 4D BIM ^†

Abstract

Natural language interfaces can transform the construction industry by enhancing accessibility and reducing administrative workload in the day-to-day operations of project teams. This paper introduces the Voice-Integrated Scheduling Assistant for 4D BIM (VISA4D) tool that integrates speech recognition and Natural Language Processing (NLP) capabilities with Building Information Modeling (BIM) to streamline construction schedule updating and maintenance processes. It accepts voice and text inputs for schedule updates, facilitating real-time integration with Autodesk Navisworks, and eliminates the need for direct access to or advanced knowledge of BIM tools. It also provides visual progress tracking abilities through colour-coded elements within the 4D BIM model for communicating task status updates within the project teams. To demonstrate its capability to enhance schedule updating and maintenance efficiency, the VISA4D tool is implemented in an office building project in Canada and user testing is performed. An overall accuracy of 89% was observed in successfully classifying 71 out of 80 tested construction-specific commands, while the user surveys indicated high usability, with 92% of participants finding VISA4D easy to use and reporting consistent command recognition accuracy. This study advances the existing work on AI-enhanced construction management tools by tackling the challenges associated with their practical implementation in field operations.

1. Introduction

The architecture, engineering, and construction (AEC) sector is experiencing a significant technological transformation, where traditional methodologies increasingly intersect with advanced computational methods. Building Information Modeling (BIM) has emerged as one of the most cutting-edge and transformative practices in the AEC sector, defined as a combination of technologies, workflows, and guidelines that allows various stakeholders to collaboratively plan, build, and manage a facility within a shared digital environment [1]. The integration of time with three-dimensional (3D) BIM, known as four-dimensional (4D) BIM, has further transformed project delivery by connecting building elements to construction scheduling tasks, enabling visualization and optimization of construction sequences [2,3,4].

Despite these advancements, construction project management, particularly scheduling, remains one of the most challenging aspects of the industry, characterized by complexity, uncertainty, and the need for continuous adaptation to evolving site conditions [5]. Typically, the process of schedule creation, monitoring, and updating is largely manual, time-intensive, and disconnected from field operations [6]. The integration of artificial intelligence (AI) into construction processes represents a transformative opportunity to address these persistent challenges. Recent advancements in Natural Language Processing (NLP) and the development of Large Language Models (LLMs) have showcased significant improvements in understanding context, processing unstructured information, and facilitating human–computer interaction through intuitive interfaces [7]. These technological developments offer a unique potential to bridge the operational gap between construction site activities and digital schedule updates and maintenance systems.

Construction scheduling methodologies have evolved substantially, with traditional network-based techniques, like the Critical Path Method (CPM) and Program Evaluation and Review Technique (PERT) advancing alongside lean-based approaches like the Last Planner System (LPS). In parallel, the rise of digital technologies has led to the development of 4D visualization and simulation tools [8], which can be integrated with various planning systems, including LPS, to enhance reliability, communication, and co-ordination. By integrating time-related data with 3D models, the use of 4D BIM enables project teams to identify potential conflicts early, optimize construction sequences, and improve communication across teams by providing animated representations of the construction process linked to scheduling data [9]. This capability has proven particularly valuable in complex projects where conventional scheduling approaches are inadequate, allowing for thorough testing of construction plans and enhancing communication with team members [10]. However, the practical implementation of these advanced methodologies faces significant barriers, including the disconnect between field observations and digital documentation, the requirement for specialized technical expertise, and the substantial time investment needed for schedule maintenance and updates [11]. Project managers and superintendents typically dedicate between 30 to 50% of their productive time to documentation and administrative tasks related to progress monitoring and schedule updating [12]. This represents a substantial inefficiency in industry workflows, particularly considering the critical importance of site-based expertise in effective decision-making processes. Furthermore, the usual practice with BIM and scheduling software predominantly relies on office-based desktop interfaces, creating a functional separation between construction site operations and digital project management systems [13].

Recent empirical studies have demonstrated the significant potential of voice and natural language interfaces to enhance operational efficiency across various industries [14,15]. In the context of construction, these technologies offer promising opportunities to transform management practices by enabling more intuitive, accessible, and field-oriented interactions with sophisticated digital systems. An innovative approach combining conversational AI with BIM via cloud infrastructure has emerged based on input from various construction professionals [16]. Site workers using these integrated systems prioritize features like hands-free operation, quick system response, and automatic calculation capabilities when accessing digital building models in field conditions. The implementation of such voice-enabled construction systems offers several advantages, including better adherence to safety protocols, workforce efficiency gains through instant information access, enhanced project monitoring capabilities, and improved opportunities for on-the-job learning without disrupting workflows [16]. By facilitating direct communication between construction professionals and 4D BIM platforms through natural language, they can reduce documentation burdens, enhance data accuracy, and improve the timeliness of schedule updates. Building on these advancements, this study introduces Voice-Integrated Scheduling Assistant for 4-Dimensional BIM (VISA4D), a novel framework that leverages advanced NLP capabilities to establish more effective connections between field operations and digital scheduling systems. VISA4D represents a shift in construction schedule management, moving beyond standard computer interfaces to embrace more intuitive and accessible interaction methodologies that accommodate the operational realities of construction environments [17]. While 4D BIM has already demonstrated its value in enhancing safety management, improving risk mitigation strategies, and optimizing construction sequences [18], the integration of natural language interfaces with these systems presents an opportunity to further enhance their accessibility and practical value for construction professionals.

VISA4D is based on the idea that construction schedule management can be significantly improved through natural language interfaces that enable field personnel to interact directly with 4D BIM systems via voice and text commands. This approach acknowledges the irreplaceable value of human expertise in construction management while addressing inefficiencies associated with traditional documentation and schedule updating processes. Therefore, the main contribution of VISA4D is that it addresses the gap in linking site operations with digital project records by integrating natural language interfaces with 4D BIM. This enables field personnel to update schedules via voice or text without directly accessing the BIM model, reducing technical barriers and supporting timely, accurate progress tracking on site.

This paper presents a comprehensive investigation of VISA4D’s conceptual framework, technological implementation, and practical applications within the context of construction scheduling. The research demonstrates how natural language interfaces can transform construction schedule updating and maintenance by enhancing accessibility, reducing administrative workload, and enabling more effective integration between field observations and digital documentation systems. The findings contribute to the growing body of knowledge on AI-enhanced construction management methodologies and offer practical insights into the design and deployment of natural language interfaces within complex technological environments.

2. Literature Review

The integration of AI into BIM represents a significant advancement within the AEC industry. This literature review explores the applications and potential of AI in enhancing 4D BIM processes and construction management.

2.1. Role of AI in Construction Management

AI is transforming construction scheduling by automating complex processes and enhancing decision-making capabilities [19]. In recent decades, numerous studies have explored how AI and its subfields can address industry-specific challenges in construction [20,21,22]. One area that faces significant challenges is construction planning, as current systems largely depend on manual input and lack flexibility [23]. AI can significantly advance the automation of construction scheduling by learning from vast datasets of project records, thus reducing the reliance on manual setup and maintenance of scheduling systems. This advancement could lead to more robust, adaptable systems that learn and improve from each executed project, making the scheduling process more efficient and less prone to human error [23].

Several AI applications have been successfully integrated into construction scheduling with promising results. In [24], EUROPA provides a comprehensive platform for constraint-based temporal planning that has been successfully implemented in NASA projects. This framework offers capabilities in three key areas: representation, reasoning, and search. EUROPA’s modular architecture allows extensive customization through plugins and extensions, making it adaptable to various scheduling challenges, though it requires significant technical expertise to implement [24].

Building on this foundation, Knowledge-Based Expert System (KBES) has been developed in [25] that connects Autodesk Revit with Primavera. Their system automates the extraction of building components from BIM models and converts them into scheduled activities with proper sequencing. Unlike EUROPA, which requires specialized knowledge [24], this system creates a more accessible bridge between design and scheduling by leveraging existing commercial software familiar to industry professionals. The authors demonstrated through a simple house building case study that their approach eliminates many tedious steps in schedule creation, though field testing on complex projects remains limited [25].

A critical review of automated planning methods spanning three decades were conducted in [23], identifying key barriers to adoption as follows: (1) rigid knowledge representation in existing systems; (2) dependency on manually maintained work templates; (3) lack of automated learning methods; (4) limited validation on real-world projects; and (5) disconnection between planning and optimization research. Their analysis suggests that advancements in deep learning (DL) and NLP could facilitate automatic generation of dynamic work templates from existing project records, reducing the manual effort currently required.

While these approaches represent significant advancements in integrating AI with construction scheduling, they still face implementation challenges. The trend toward more intuitive interfaces is evidenced in recent research combining Natural Language Understanding (NLU) with BIM [26], facilitating voice-activated updates to improve accessibility for field personnel. The integration of machine learning (ML)-enhanced BIM models with natural language capabilities presents opportunities to improve schedule management efficiency and reduce technical barriers to adoption [26,27,28].

Despite these advancements in AI applications for construction scheduling, a significant gap remains in real-time, field-accessible interfaces that allow project managers to interact naturally with scheduling systems. While [23] highlighted the rigid knowledge representation and limited real-world validation of existing systems and both [24,25] demonstrated powerful AI planning capabilities, these solutions still require specialized technical knowledge and office-based operation. The VISA4D framework addresses this critical gap by leveraging AI to process natural language commands delivered through voice and text inputs directly from construction sites. Unlike the existing systems described above, VISA4D democratizes access to scheduling technology by enabling immediate on-site schedule updates through intuitive voice commands, thereby reducing the technical barriers that have historically limited the adoption of advanced scheduling tools among field personnel.

2.2. Natural Language Processing in BIM and Construction Scheduling

In recent years, NLP has found significant applications within BIM information systems, effectively reducing the need for extensive knowledge and technical expertise among users. Users can access and manipulate BIM data through natural language queries, facilitating interaction without requiring an in-depth understanding of the software’s functionalities [29]. Advanced LLMs, such as OpenAI’s Generative Pretrained Transformer (GPT) series and Bidirectional Encoder Representations from Transformers (BERT), have demonstrated remarkable efficiency in tasks involving comprehension, generation, and reasoning with natural language by leveraging DL techniques and large-scale datasets to develop relative representations of text [30]. This capability allows them to generate consistent and contextually appropriate responses, thereby enhancing user engagement with BIM platforms.

Leveraging existing technological frameworks, researchers have advanced several innovative NLP-enhanced BIM systems. For example, a BIM-GPT framework was introduced in [29], which effectively translates natural language queries into relevant segments of 3D BIM models. Similarly, a GPT-powered assistant named DAVE was designed for real-time multimodal interactions within BIM contexts [31]. This system integrates with Autodesk Revit, enabling users to engage through both text and voice for updates and inquiries, thus aiming to optimize complex BIM workflows. Additionally, a chatbot named JULIE was tailored specifically for construction site managers. JULIE employs NLU to identify user intent accurately, facilitating the delivery of actionable insights via conversational interfaces [26].

NLP has shown considerable promise in optimizing planning and scheduling processes within the construction sector. A comprehensive evaluation of the current landscape of automated planning and scheduling in construction is provided in [23]. The investigation primarily focused on identifying the main challenges that have caused the delay in the adoption of automated planning techniques and systems within the construction industry. A notable finding is the difficulty in transferring the expertise of human planners to automated systems, mainly due to the absence of standardized terminology and systematic knowledge frameworks in the construction domain. The authors advocate for the application of NLP to bridge the gap between short-term planning tasks, such as look-ahead scheduling, and long-term planning tasks, such as master schedules within the construction industry [23]. Furthermore, the study critiques existing methodologies for their heavy reliance on rigid manual templates and highlights the critical need for adaptable knowledge management systems. These insights underline the imperative to refine current practices by integrating flexible, automated solutions that can effectively leverage human expertise in construction planning [23].

In a recent study, researchers investigated the influence of GPT models on project management, specifically concerning scheduling and task allocation in construction projects [32]. The primary objective was to provide a comprehensive analysis of tasks generated by these models. The results demonstrated that GPT models could create a coherent and sequential task list across various scenarios. However, it was noted that some generated tasks did not fully align with the project’s overarching goals. The quantitative analysis revealed variability in the project timeline for different stakeholders relative to the baseline schedule. Furthermore, discrepancies were observed in the number of personnel required to complete the proposed tasks [32].

Despite these advances, current NLP implementations in construction scheduling still have limitations in real-time on-site application and voice command integration. The proposed VISA4D framework addresses this gap by enabling direct voice and text command inputs from construction sites to instantly update 4D BIM models, potentially transforming how construction schedules are managed and communicated across project teams.

2.3. Real-Time Schedule Updating

In construction management, real-time schedule updating represents a critical yet challenging aspect of project oversight. This process involves comparing actual construction data from the site with the initial plan to accurately adjust activity timelines. The scheduling of construction activities relies mostly on the expertise of superintendents and project managers, who conduct visual inspections and documentation based on their direct oversight and knowledge [33].

A common application of BIM in construction involves linking schedules to BIM models, creating 4D BIM, a combination of 3D models with time information. This integration enables visualization of construction simulations and provides essential insights into critical schedule aspects [6]. The process necessitates a thorough assessment of on-site project conditions and a comparison between current progress and baseline schedules [6,34].

To support this complex process, researchers have proposed various technological solutions. These include frameworks incorporating automated activity tracking, material tracking, and mobile computing communication [35], as well as methods integrating 4D BIM with 3D data for construction progress measurement [36]. In [36], a two-stage revision process was developed to address incomplete datasets and it achieved progress tracking accuracy of up to 100%. Building on these innovations, a system was created to streamline schedule updates by fusing 3D point-cloud data with 4D BIM, enhancing accuracy through automated registration techniques and effective object matching [12].

Despite advancements in technology, schedule updating and maintenance in construction still rely heavily on human expertise. The VISA4D framework acknowledges the indispensable role of human judgment within construction management. Rather than aiming for full automation, it focuses on streamlining the labour-intensive tasks of data collection and organization involved in progress monitoring and, therefore, enhancing documentation efficiency while preserving the value of expert insight.

2.4. Contributions of the VISA4D Framework

Despite significant technological advancements in BIM-based construction scheduling, substantial methodological gaps persist in real-time schedule updating and maintenance. VISA4D addresses these challenges through the innovative application of NLP capabilities within the 4D BIM domain. While existing construction schedule management methods often depend on conventional computer interfaces and office-based data entry, they can limit the immediacy of field-to-digital schedule updates and contribute to higher administrative workloads. Although specialized knowledge may still be needed to operate such systems effectively, including the current version of VISA4D, this approach represents a step toward streamlining routine updates and reducing inefficiencies in project workflows.

The VISA4D framework introduces a transformative tool through natural language interfaces that facilitate direct integration with 4D BIM systems. It contributes to the existing body of knowledge in several critical dimensions as follows:

• Field-based interaction methods; unlike conventional BIM interfaces that rely exclusively on office-based interaction, VISA4D establishes direct communication channels between construction site operations and digital scheduling platforms by enabling on-site personnel to interact with the 4D BIM environment through intuitive natural language commands.
• Multimodal communication protocols; the implementation of both voice and text-based interaction distinguishes VISA4D from existing construction schedule interfaces. This dual-channel approach accommodates diverse operational environments, from construction sites where voice commands allow hands-free operation while performing physical tasks to office environments where text-based interaction might be preferred.
• Facilitation of 4D schedule manipulation; the system enables complete schedule management operations through natural language commands in a 4D BIM environment, such as task addition/deletion, modification of time and date parameters, and task status adjustments.
• Real-time synchronization with BIM; the instant integration of natural language commands with 4D BIM visualizations represents a significant advancement over traditional methodologies that typically involve substantial processing delays between field observations and schedule updates.

VISA4D directly addresses critical limitations identified in current research regarding AI integration with construction scheduling. Recent studies emphasize the limitations of automated scheduling systems, particularly their reliance on rigid templates and inability to effectively capture human expertise [23]. VISA4D’s application of NLU establishes a more intuitive mechanism for translating human judgment into actionable scheduling parameters.

The system also addresses limitations identified in recent evaluations of LLMs in construction contexts. While research has demonstrated potential in generating construction task sequences [32], challenges remain in aligning outputs with project-specific objectives. VISA4D avoids these limitations by focusing on schedule updating and maintenance, keeping decision making in the hands of project personnel while streamlining the implementation process.

Finally, from a theoretical perspective, VISA4D contributes to the dialogue on human–computer interaction in construction management by demonstrating the potential of natural language interfaces to bridge the gap between field operations and digital systems. This approach aligns with current research emphasizing the importance of developing technologies that accommodate existing workflows rather than imposing artificial constraints. Overall, the development of VISA4D represents a significant advancement in AI integration with construction scheduling, addressing critical gaps in 4D BIM accessibility and efficiency.

Table 1. Comparison of VISA4D with existing methods.

Feature	VISA4D	Nabavi et al. (2023) [37]	Singh et al. (2023) [38]	Aljebory and Issam (2019) [25]
Primary Technology	Natural language interface for 4D BIM	NLP for automated information inquiry from BIM	BIM and NLP integration for automatic scheduling	AI-based project planning by connecting BIM with Primavera
Field Usability	Both site and office	Office-based only	Office-based only	Office-based only
Real-time updates to schedule	Real-time synchronization with 4D BIM	N/A	Not focused on real-time operations	Not focused on real-time updates
User Accessibility	Intuitive natural language interaction requiring minimal training	User-friendly but requires some training	User-friendly but limited in generating schedule	Requires expert knowledge to operate
Integration with BIM	Direct integration with 4D BIM	Integration with BIM	Proposes integration but lacks implementation details	No direct integration with BIM
Schedule Manipulation	Complete schedule operations through natural commands	Limited to information retrieval	Focuses on schedule generation rather than manipulation	Limited to schedule generation
Multimodal Communication	Voice and text interaction	Voice and text interaction	Text-based interaction with potential voice interaction	Text-based interaction only

provides a summary of a comparison of VISA4D with existing methods.

3. Research Methodology

This research employs a design science research (DSR) methodology [39] to develop and evaluate the VISA4D framework. DSR is particularly suitable for this study as it focuses on creating innovative artifacts to solve practical problems while making rigorous theoretical contributions to knowledge [39]. DSR is characterized by its emphasis on developing solutions to address problems in a domain [40], which aligns perfectly with the aim of improving construction schedule management through natural language interfaces to 4D BIM.

Following the DSR methodology guidelines established in [39] and extended in [41], this research was structured into four main phases: (1) problem identification and conceptualization, (2) design and development, (3) implementation, and (4) evaluation, as illustrated in Figure 1.

3.1. Problem Identification

The initial phase involved identifying the research problem through an extensive literature review, professional experience, and insights gathered from project management and site supervision staff (Figure 1a). This approach aligns with the principle that DSR projects should begin with a problem relevant to the application environment [42].

This phase established the need for improved methods to update and maintain construction schedules in 4D BIM environments. The literature review revealed that, while 4D BIM offers significant advantages for construction scheduling, the updating process remains complicated and slow, particularly for field personnel who may lack advanced training in specialized BIM tools. Furthermore, input from construction professionals highlighted the communication gap between field operations and office-based BIM specialists, leading to delays in schedule updates and potential discrepancies between planned and actual progress.

The problem identification phase was guided by principles from DSR [43], incorporating both theoretical gaps from the literature and practical challenges raised by industry professionals, to support relevance and potential contribution. Based on these findings, the research problem was conceptualized as a need for an intuitive interface that allows direct interaction with 4D BIM schedules through natural language commands. This conceptualization guided the development of system requirements and the overall architecture of VISA4D.

3.2. Design and Development

The design and development phase (Figure 1b) focused on creating the system architecture and determining the appropriate technologies and methodologies for implementation. This phase embodies what is described as the creative process of generating new artifacts to address identified problems [44].

The design process involved requirements analysis, system architecture design, technology selection, and algorithm design. Requirements analysis defined functional and nonfunctional requirements based on the identified problem and user needs. Key requirements included the ability to process natural language commands, integration with a commonly used 4D BIM software, such as Autodesk Navisworks 2024, support for both voice and text input, and real-time schedule updates.

System architecture design (Section 4) developed a modular system where each major function, the user interface, NLP processing, task management, and BIM integration was designed as a separate, dedicated component with its specific role. This approach follows the modularity principle adopted in DSR by [45], making the system more flexible and easier to maintain.

Technology selection involved choosing appropriate technologies for each system component, including Python 3.11.9 for client-side components due to its extensive NLP libraries, C# for server-side integration with Navisworks, and JavaScript Object Notation (JSON) for data interchange. Algorithm design included designing the NLP pipeline and algorithms for intent classification, entity extraction, and command validation. This involved developing ML models and rule-based patterns for construction-specific language processing.

The development process incorporated iterative refinement, which is a key characteristic of DSR as described in [46]. The system was implemented using modern software development practices, including object-oriented programming and asynchronous processing. Regular prototype testing was conducted throughout this phase to identify and address technical issues, as recommended for continuous evaluation in DSR projects [47].

3.3. Implementation

The implementation phase (Figure 1c) translated the design specifications into a functional system, embodying the emphasis on creating tangible artifacts as the primary contribution of DSR [39]. This phase involved several key activities, including component development, integration, NLP model training, user interface implementation, and system testing.

Component development implemented each system component according to the architectural design of the system, including the Graphical User Interface (GUI) layer, NLP processor, task manager, Navisworks Application Programming Interface (API), and custom Navisworks plugin. Through well-defined interfaces and protocols, as explained in more detail in Section 4, communication has been established between all components. This included implementing the HTTP server and RESTful API for interaction between the client application and Navisworks.

NLP model training involved training and fine-tuning the intent classification model using construction-specific training data generated from templated command structures. User interface implementation developed an intuitive interface that supports both text and voice inputs, with appropriate feedback mechanisms and error handling. System testing included conducting iterative testing to ensure functionality across various command types and scenarios. The testing strategy encompassed unit testing to validate individual components individually, followed by integration testing to assess the functionality of the complete system as an integrated system.

The implementation resulted in a functional prototype of the VISA4D framework capable of processing natural language commands and executing corresponding actions within the Navisworks environment.

3.4. Evaluation

The evaluation phase (Figure 1d) assessed the effectiveness, usability, and performance of the VISA4D framework through multiple approaches. This comprehensive evaluation strategy follows the Framework for Evaluation in Design Science (FEDS) [41], which emphasizes the importance of a multi-faceted evaluation approach for DSR artifacts.

As described in more detail in Section 6.1, user testing was conducted with 20 undergraduate and graduate civil engineering students who used the VISA4D tool for the case study of a real-world office building construction. Participants performed various tasks, including adding new tasks, deleting existing tasks, updating dates, and modifying task statuses using both voice and text commands. Qualitative and quantitative feedback were collected through anonymous surveys that included structured multiple-choice questions and open-ended questions to evaluate user experience, ease of use, and observed utility. This approach aligns with guidance on user-centered evaluation of information technology (IT) artifacts [48].

Performance testing (Section 6.2) involved developing a systematic testing framework to assess the reliability and accuracy of the NLP processor in interpreting construction-specific commands. This included testing 80 unique commands across five intent categories and analyzing the results using various performance metrics. Accuracy analysis computed and analyzed metrics such as intent accuracy, task name extraction accuracy, precision, recall, and F1-scores to identify strengths and weaknesses in the system’s NLP capabilities.

Finally, the evaluation phase provided comprehensive insights into the VISA4D tool’s performance and usability, validating its effectiveness while also highlighting areas for future improvement. This combination of naturalistic (i.e., user testing) and artificial (i.e., performance testing) evaluation strategies follows the balanced approach recommended in [41] for robust DSR evaluation.

4. System Architecture and Implementation Details

The implementation of the VISA4D tool follows a modular approach, with each component designed to fulfill specific functions within a voice and text-interactive framework. This section details the system architecture and code structure of the VISA4D tool that integrates conversational interfaces with Autodesk Navisworks. All source code for the VISA4D tool was custom-developed by the authors and is available in an open-source repository [49].

Modern software development practices, including object-oriented programming and asynchronous processing, have been utilized in this process to ensure a responsive user experience. Python programming language was used for the development of client-side components, as it offers extensive libraries and frameworks for NLP to perform text preprocessing, sentiment analysis, and named entity recognition. Additionally, Python offers various tools and widgets through its user interface libraries, such as Tkinter, which is ideal for building chat-bot-type interfaces.

On the server side, integration with Navisworks is achieved using C#, which directly interfaces with the Autodesk API stack through Microsoft Visual Studio. The codebase is organized into logical modules to ensure a clear separation of interfaces between different components.

As shown in Figure 2, the system comprises many interconnected components, each with specific functions; however, there are three primary components. The NLP processor component (Figure 2a) is the NLU engine, implemented in Python to leverage its extensive libraries for language understanding, pattern matching, and intent classification. The other two are NavisworksServer (Figure 2f) and VISA4D Plugin (custom Navisworks plugin) (Figure 2g), which are closely integrated components implemented in C# to interface directly with Autodesk Navisworks.

NavisworksServer component (Figure 2f) implements a local HTTP server using the .NET System.Net.HttpListener class, creating a bridge between external applications, such as the user interface, and Navisworks (Figure 2i), whereas the VISA4D Plugin (Figure 2g) component implements the custom Navisworks plugin, which integrates directly with Navisworks through the Autodesk API.

JSON serves as the primary data interchange format throughout the system (Figure 2d). It facilitates structured communication across system layers for transmitting parsed commands. HTTP serves as the primary communication protocol between system components (Figure 2e), implementing RESTful design principles and using HTTP methods, such as POST for creating new tasks in Navisworks TimeLiner, PUT for updating existing tasks, DELETE for removing tasks from the system, and GET for retrieving authentication status. The system implements OAuth 2.0 (Open Authorization) authentication for Autodesk Platform Services (APS), ensuring secure access to construction data. In the following section, each of the components is explained in more detail.

4.1. User Input Layer (gui.py)

The GUI layer (Figure 2c) serves as the primary interface between users and the VISA4D tool and facilitates both text- and voice-based interactions. This layer was implemented using the customtkinter library, which extends Python’s Tkinter capabilities to create a more refined and interactive interface. The dual-input modality is a key implementation feature of VISA4D, which allows users to interact through both conventional text inputs and voice commands (Figure 3). The voice input functionality, particularly valuable for enabling hands-free operation for site superintendents, was implemented using SpeechRecognition library, utilizing Google’s Speech-to-Text API as its transcription backend.

Upon activation through the microphone button (as shown in the system sequence diagram in Figure 4), the system initiates a sequence of operations that begins with microphone calibration using SpeechRecognition, followed by audio recording with appropriate timeout mechanisms. The recorded audio undergoes processing and conversion to text, and is treated as a text command moving forward, which is routed to the NLP system for analysis and task execution. If the input is a text command, then the processing is the same, except for the microphone calibration.

Asynchronous speech processing was applied, and speech recognition operations are run in separate threads to prevent user interface blocking or freezing during audio processing operations. Thread synchronization was managed using Tkinter’s after method to safely update user interface elements from worker threads without causing race conditions. Additionally, the implementation incorporates on-demand component initialization to reduce application startup time.

Comprehensive error handling was integrated throughout the GUI layer (Figure 2c) to manage possible issues such as connection failure or unrecognized speech patterns. To enhance usability across different environments and user preferences, the interface implements a dynamic theming system supporting both light and dark visual modes.

The OAuth 2.0 authentication mechanism was implemented to secure integration with the Navisworks API. Once authenticated, the status indicator in the header changes to reflect the connection state (e.g., Connected, as in Figure 3), providing continuous visual feedback about system connectivity. Moreover, the interface actively checks authentication status before attempting protected operations, ensuring system security and preventing unauthorized access.

The GUI layer connects directly to the NLP processor class and Task Manager class components to forward natural language inputs (both typed and transcribed) to appropriate subsystems for transformation into structured commands. Overall, the GUI layer achieves a balance between visual appeal and functional robustness, creating a system capable of reliable operation in real-world construction environments.

4.2. Nlp_processor.py

The nlp_processor.py module (Figure 2a) integrates state-of-the-art NLP tools with domain-specific knowledge to interpret and process construction-related commands. Its primary objective is to convert unstructured natural language input into structured, actionable tasks suitable for construction management systems. This module combines multiple techniques such as ML, rule-based pattern matching, and semantic similarity measures to ensure precise and reliable interpretation of user inputs in a construction context.

The NLP pipeline begins with text preprocessing, where raw user input is normalized. Stop words are removed and lemmatization is applied to reduce noise. For example, in the command “Update the foundation pouring task to complete”, tokenization and part-of-speech tagging are handled using the spaCy library, which identifies the syntactic roles of words such as “foundation” (noun) and “pouring” (verb).

The intent classification phase is designed to accurately categorize user commands based on their specific operational intent. This includes commands for creating tasks, updating statuses or dates, and deleting tasks. The classification system encompasses four primary intent categories: create_task, update_status, update_date, and delete_task. Any command that does not conform to these defined categories is classified as unknown.

To support intent recognition, the system uses TF-IDF vectorization combined with a RandomForestClassifier. The TF-IDF model emphasizes construction-specific terms such as “pour”, “foundation”, and “inspection”, enabling the classifier to distinguish between domain-relevant variations. Training data for this classifier are generated programmatically using templated command structures to ensure coverage of key variations in phrasing. Once the intent is identified, the system performs entity extraction to delineate task names, statuses, and relevant dates.

Entity recognition is accomplished through a hybrid approach that primarily relies on direct pattern matching with domain-specific templates. If patterns fail to match, it falls back on dependency parsing and noun phrase extraction, leveraging spaCy’s syntactic analysis. In cases of ambiguous expressions, semantic similarity is evaluated using SentenceTransformer, a BERT-based model that maps text into vector space for comparison. This allows the system to interpret commands like “start concrete pouring” and “begin concrete work” as semantically similar.

The nlp_processor.py also incorporates a command validation phase to ensure completeness before execution. It assesses the presence of essential parameters and analyzes the confidence scores associated with identified intents and entities. When the confidence score falls below a predetermined threshold (typically 60%), the system prompts the user for clarification; if between 60–85%, then the system performs additional validation checks before executing the command. For instance, if it is an “update status” command, the system ensures it has status information. Furthermore, if the confidence score exceeds 85%, the processor proceeds directly to execute the task. This decision framework is detailed in Figure 4, which maps out the end-to-end flow from raw input to task manager execution.

4.3. Task_manager.py

The task_manager.py (Figure 2d) module serves as a bridge between the NLP processor component and the Navisworks application. Upon interpreting user commands and extracting intents and entities, nlp_processor.py forwards this information to task_manager.py. This module processes the data and transforms these into a structured JSON format that aligns with the requirements of the Navisworks API.

The module implements four fundamental operations:

• Task creation: this operation facilitates adding new tasks to the TimeLiner, the 4D scheduling tool in Navisworks.
• Status updates: this function allows for the modification of task statuses, enabling dynamic transitions between states such as “not started” and “in progress”.
• Date modifications: this capability provides the flexibility to reschedule tasks by modifying their start or end dates, ensuring accurate project timelines.
• Task deletion: this operation permits the removal of tasks from the system, effectively managing the project scope and maintaining data integrity.

Communication with Navisworks is handled via a RESTful API using the Python requests library for executing HTTP operations. This interaction is secured using OAuth 2.0 for authentication and employs JSON formatting for payloads to facilitate data exchange. Additionally, URL encoding is utilized to handle special characters in task names, along with comprehensive validation and error-handling mechanisms.

A synchronization pattern is implemented where updates are initially applied to a local state before being pushed to Navisworks. The local state is sustained through two JSON-based data structures: (1) task_statuses and (2) task_dates. This local storing mechanism minimizes the frequency of API calls and enhances overall response times.

Finally, task_manager.py also implements comprehensive logging throughout the entire process for effective monitoring and troubleshooting. Logs are stored in a visa4d.log file where information about operation attempts with parameters, successful completions, error conditions with details, and changes in authentication status are stored. These logs provide visibility into system operations and serve as an audit trail for construction schedule modifications.

4.4. Navisworks_api.py

The navisworks_api.py module (Figure 2h) facilitates direct interaction between the underlying system and Autodesk Navisworks. It acts as an intermediary that converts user-driven commands into executable tasks.

Upon initialization, the module sets up a base URL (http://localhost:5000) for interfacing with the Navisworks service. It also defines the requisite standard HTTP headers to facilitate JSON communication and configures endpoint mappings for essential operations, including task creation, updates, deletions, and user authentication. The authentication workflow initiates with the authenticate method that retrieves access tokens via client credentials. These tokens, along with their expiration timestamps, are stored securely to ensures their integrity. To maintain token validity, expiration is actively tracked with time buffers. This class implements three fundamental task operations, as discussed previously: create_task, update task, and delete task. The navisworks_api.py leverages the Python requests library for HTTP operations, using appropriate request types for different operations. Each request is constructed with appropriate headers, JSON payloads, and error handling to ensure reliable communication with the Navisworks system. Figure 5 shows the navisworks_api.py steps in detail.

4.5. NavisworksServer.cs

The NavisworksServer.cs class (Figure 2f) serves as a local HTTP server that facilitates communication between external applications and the custom Navisworks plugin. Operating on localhost at port 5000, it leverages the HttpListener class from the System.Net namespace. This setup creates a bridge that allows external applications to interact with Navisworks’ TimeLiner functionality through standard web protocols.

The server implementation employs asynchronous programming patterns using C#’s async/await functionality, ensuring efficient handling of multiple requests without blocking the main application thread.

The TimeLiner task management system exposes three core REST API endpoints for handling tasks: creation, updating, and deletion. These endpoints are designed to accept JSON payloads for initiating new tasks, processing update requests, and triggering task deletion. The server utilizes Newtonsoft.Json for serialization and deserialization, ensuring reliable data exchange between the client and server.

The server maintains a reference to the main custom VISA4D plugin instance, allowing it to delegate task operations to the appropriate plugin methods. This integration ensures that all operations performed through the API directly affect the Navisworks environment.

4.6. Navisworks TimeLiner Custom Plugin (Addin.cs)

The VISA4D Plugin (Figure 2g), the custom Navisworks TimeLiner Plugin, represents the final component of the integration between the NLP system and Autodesk Navisworks. This custom plugin was implemented in C# using Visual Studio 2022. This plugin serves as the interface between the NLP-driven command layer and the Navisworks environment, allowing natural language inputs to trigger direct actions within the 4D environment. The plugin is designed as an AddInPlugin, utilizing three fundamental components of the Autodesk API: Autodesk.Navisworks.Api for document access and manipulation, Autodesk.Navisworks.Api.Plugins for interface integration, and Autodesk.Navisworks.Api.TimeLiner for scheduling capabilities.

A key feature of the plugin is its initialization of a local HTTP server (NavisworksServer.cs), which listens for incoming commands from the external client application. This client, implemented in Python, handles natural language input and connects to the plugin via the HTTP interface. This architecture enables full-cycle command execution, from language input to 4D model manipulation, supporting task creation, updates, and deletions within Navisworks. Visual feedback is provided through a colour-coding scheme that represents task statuses directly in the 3D model. The plugin uses the colour assignment to represent different task states: gray for “not started”, blue for “in progress”, red for on hold, and green for completed. These colour indications allow users to visually assess construction progress at a glance by inspecting the associated model elements. To ensure data consistency, the plugin’s update mechanism duplicates the original task before any changes are made. This approach protects nontargeted task attributes, ensuring that only the specified parameters are modified. Appendix A shows the overall communication data flow for VISA4D.

5. Schedule Maintenance and Updating Using the VISA4D Tool

As explained above, VISA4D enables field personnel to interact with 4D BIM schedules using natural language voice or text commands, streamlining progress tracking and reducing the need for direct manipulation of complex scheduling tools. The tool supports a range of schedule-related operations, including updating task start and finish dates, changing task statuses, and adding or removing activities. Figure 6 shows the 4D BIM schedule of a sample project within the Navisworks environment updated using the VISA4D tool. In this example, using the VISA4D tool, a user updated a task’s completion date by issuing a command such as “Hey VISA4D, update the actual finish date for slabs to September 9”. In response, VISA4D automatically adjusted the recorded finish date to 9 September 2024, synchronized the change with the Navisworks schedule, and updated the associated Gantt chart. Visual indicators within the Navisworks environment are simultaneously updated to reflect task status, with completed elements turning green, as shown in Figure 6.

In cases where co-ordination issues arise, users can suspend a task by issuing commands such as “set roof mechanical equipment and all door installations on hold”. In the example in Figure 6, VISA4D reflects this status in both the Gantt chart and the visual model, changing the task’s colour to red. The tool also allows progress updates for tasks that are underway. For example, the command “Update the status of flooring to in progress” in the example in Figure 6 resulted in appropriate schedule updates and a visual change (e.g., blue colouring) indicating the task is in progress (Figure 6).

VISA4D also accommodates real-time schedule modifications. In Figure 6, the user added overlooked tasks, such as window 100 and window 200, through voice commands. If needed, invalid entries can be deleted easily, for example, using a command like “Hey VISA4D, delete window 100 task”.

These interactions illustrate the system’s ability to facilitate fast, intuitive, and accurate updates to the project schedule directly from the field. The updated Navisworks model uses a colour-coded system where green is for completed, red is for on hold, blue is for in progress, and grey is for not started. This allows project managers to quickly assess project status and prioritize critical issues (Figure 6). These features were validated through user testing, during which participants successfully performed similar actions using the VISA4D prototype on a real-world office building project. This process is explained in detail in the following section.

6. Validation and Verification of the VISA4D Tool

6.1. User Tests

As part of the validation process, the VISA4D tool was tested by 20 undergraduate and graduate civil engineering students at the University of Manitoba who volunteered to participate in this study. A real-world office building construction was used as the case study building for the tests, for which a construction schedule was created in Navisworks and linked to its model elements to create the 4D BIM of the project. The user tests included testing multiple activities and primarily focused on adding new tasks, deleting existing tasks, assigning actual dates to tasks, updating or adding task start and finish dates, and modifying the status of tasks. Participants were instructed to engage with at least four random features of the VISA4D tool. During the user testing, all participants tested both voice and text commands.

After the hands-on testing, an anonymous survey that included structured multiple-choice questions as well as open-ended questions (presented in

Table A1. Validation questions for participants.

Questions	Options/Response
Q1: Ease of use of VISA4D Tool	The tool was very easy to use
	The tool was easy to use
	The tool was neither easy nor difficult to use
	The tool was difficult to use
	The tool was very difficult to use
Q2: Command recognition of the VISA4D Tool	The tool consistently recognized my commands accurately
	The tool often recognized my commands accurately
	The tool sometimes recognized my commands accurately
	The tool rarely recognized my commands accurately
	The tool never recognized my commands accurately
Q3: Text Handling Effectiveness of the VISA4D Tool	The tool handled text inputs very effectively.
	The tool handled text inputs effectively.
	The tool handled text inputs neither effectively nor ineffectively.
	The tool handled text inputs ineffectively.
	The tool handled text inputs very ineffectively.
Q4: Voice Handling Effectiveness of the VISA4D Tool	The tool handled voice inputs very effectively.
	The tool handled voice inputs effectively.
	The tool handled voice inputs neither effectively nor ineffectively.
	The tool handled voice inputs ineffectively.
	The tool handled voice inputs very ineffectively.
Q5: VISA4D’s Response Time of Commands	The tool’s response time to commands was always satisfactory.
	The tool’s response time to commands was usually satisfactory.
	The tool’s response time to commands was sometimes satisfactory and sometimes unsatisfactory.
	The tool’s response time to commands was usually unsatisfactory.
	The tool’s response time to commands was always unsatisfactory.
Q6: Errors in Command Execution by the VISA4D Tool	The tool executed commands without any errors.
	The tool rarely executed commands with errors.
	The tool sometimes executed commands with errors.
	The tool often executed commands with errors.
	The tool always executed commands with errors.
Q7: User-Friendliness of the VISA4D Tool’s Interface	The tool’s interface is very user-friendly.
	The tool’s interface is generally user-friendly.
	The tool’s interface is neither user-friendly nor user-unfriendly.
	The tool’s interface is somewhat user-unfriendly.
	The tool’s interface is very user-unfriendly.
Q8: Ease of Learning to Use the VISA4D Tool	Learning to use the tool was very easy.
	Learning to use the tool was easy.
	Learning to use the tool was neither easy nor difficult.
	Learning to use the tool was difficult.
	Learning to use the tool was very difficult.
Q9: What do you like the most about VISA4D?
Q10: What challenges did you face while using VISA4D?
Q11: Do you have any general comments or feedback about VISA4D?
Q12: Are there any additional features you would like to see in VISA4D?
Q13: What improvements would you suggest for VISA4D?

in Appendix A) was conducted to gather feedback from the participants, which served as the basis for the validation of the tool. The survey revealed positive outcomes, with 95% of respondents (19 out of 20) reporting that the tool was very easy or easy to use. Additionally, all participants affirmed that the VISA4D tool consistently or often recognized commands accurately, while 85% reported consistent accuracy. Regarding text handling, all participants found the tool effective, with 85% rating it as very effective.

Voice command handling also yielded favourable results, with 80% of respondents rating this feature as very effective or effective. Furthermore, the tool’s response time was considered satisfactory by all participants, with 90% describing it as always satisfactory. The user-friendliness of the interface received high approval, with 85% labeling it as very user-friendly. Learning to operate the tool was deemed very easy by 90% of the testers.

Participants appreciated the simplicity and intuitiveness of the user interface, effective voice control, and visual clarity provided by colour-coding tasks. However, challenges noted included occasional crashes of the Autodesk Navisworks software (version 2024), predominantly associated with computational limitations and the robustness of voice command recognition across diverse accents. Suggested improvements highlighted enhancing stability to prevent software crashes and improving voice command capabilities. Challenges have been further explained in Section 6.3.

6.2. Assessing the Reliability and Performance of the VISA4D Tool

The reliability and performance of the VISA4D tool were investigated and verified through a systematic testing approach focusing on intent classification capabilities. A custom testing framework was developed in Python to assess the precision of the NLP processor class in interpreting and responding to a range of construction-specific commands.

To achieve broad coverage and assess the reliability of the test framework, each of the five intent categories (i.e., create task, delete task, update date, update status, and unknown) was evaluated with a set of 20 unique commands, resulting in a total of 80 commands tested.

Table 2. Sample questions categorized by their perspective commands.

Commands	Sample Queries
Create task	Add Foundation Piling task	Can you create Electrical Wiring
	Create a new task for Grade Beams Pour #3	Please add Plumbing Installation
	Set up Elevator Raft	Add task Exterior Siding
	I need a new task for Feature Wall Pours	Create a new task for Door 2000 Installation
	Can you create Main Floor Glazing Installation	Set up Stair 1000
	Please add Second Floor Drywall Installation	I need a new task for Slab A
	Add task Roof Door Installation	Can you create Structural Framing
	Create a new task for Concrete Pouring	Please add Final Inspection
	Set up HVAC Installation	Add task Grade Beams Pour #5
	I need a new task for Window Installation	Create a new task for Main Floor Door Installation
Update Status	Update the status of Foundation Piling to complete	Update the status of Electrical Wiring to in progress
	Change Grade Beams Pour #3 status to in progress	Change Plumbing Installation status to complete
	Set Elevator Raft as on hold	Set Exterior Siding as in progress
	Mark Feature Wall Pours complete	Mark Door 2000 Installation complete
	Update Main Floor Glazing Installation to in progress	Update Stair 1000 to on hold
	Can you update Second Floor Drywall Installation to on hold	Can you update Slab A to in progress
	Please change Roof Door Installation status to complete	Please change Structural Framing status to complete
	Update Concrete Pouring to in progress	Update Final Inspection to on hold
	Set HVAC Installation as complete	Set Grade Beams Pour #5 as in progress
	Mark Window Installation on hold	Mark Main Floor Door Installation complete
Delete Task	Delete Foundation Piling task	Delete the Electrical Wiring task
	Remove Grade Beams Pour #3 task	Could you please remove Plumbing Installation task
	Delete task Elevator Raft from the TimeLiner	I’d like to remove Exterior Siding
	Remove task painting	Please remove Door 2000 Installation
	Delete Main Floor Glazing Installation	Would you delete Stair 1000 task for me
	Can you remove Second Floor Drywall Installation	Can you remove task Slab A
	Delete task Roof Door Installation	I need to terminate Structural Framing
	Remove the Concrete Pouring task	Please remove the Final Inspection task
	I would like you to remove HVAC Installation	Delete the task Grade Beams Pour #5
	Remove Window Installation	I want you to cancel Main Floor Door Installation
Update Date	Update start date for Foundation Piling to April 5	Update start date for Electrical Wiring to March 28
	Update finish date for Grade Beams Pour #3 to May 12	Update finish date for Plumbing Installation to July 18
	Update start date for Elevator Raft to June 20	Update start date for Exterior Siding to May 20
	Update end date for Feature Wall Pours to July 10	Update end date for Door 2000 Installation to June 28
	Update start date for Main Floor Glazing Installation to March 15	Update start date for Stair 1000 to April 10
	Update end date for Second Floor Drywall Installation to August 22	Update finish date for Slab A to August 5
	Update start date for Roof Door Installation to April 30	Update start date for Structural Framing to March 25
	Update finish date for Concrete Pouring to May 8	Update end date for Final Inspection to September 30
	Update start date for HVAC Installation to June 15	Update finish date for Grade Beams Pour #5 to April 22
	Update end date for Window Installation to September 3	Update start date for Main Floor Door Installation to May 15

presents the sample queries categorized by their functional perspectives. This collection was designed to encompass a wide array of the prototype’s potential use cases, facilitating a thorough evaluation of its functional accuracy.

Each intent category was tested with tailored command types:

• Task creation: to test the ability of the system to recognize requests to add new construction tasks.
• Status update: to verify accuracy in recognizing status change requests.
• Date update: to test the system’s ability to recognize different date patterns and distinguish between start and end date modifications
• Task deletion: to test the system’s ability to recognize task removal requests.

These commands were stored in a structured CSV file, with command sets designed using varied phrasings, orders, and construction-specific terminology to reflect real-world language use and test the robustness of the NLP processor class. To handle such variations, VISA4D employs several techniques: the NLP processor uses semantic similarity evaluation to interpret differently phrased commands with similar meanings, as shown in Section 4.2, where phrases like “start concrete pouring” and “begin concrete work” are understood as equivalent. For ambiguous expressions, the system applies a BERT-based model to map text into a vector space for comparison and, when confidence scores fall below set thresholds, it prompts the user for clarification, ensuring accurate interpretation even when construction professionals use unclear terms.

The framework analyzed each command by comparing the predicted task name and intent with the expected intent and expected task name. Then, each command was fed into the NLP processor class, and the responses were captured and analyzed, extracting predicted intents and task names.

Finally, the accuracy metrics were computed by comparing predicted values against expected values. Python libraries, including pandas for data manipulation, sklearn.metrics for classification metrics, and matplotlib with seaborn for visualization of results, were used. Various performance metrics were computed and visualized:

• Intent accuracy: percentage of commands where the predicted intent matched the expected intent.
• Task name extraction accuracy: percentage of commands where the extracted task name correctly matched the expected task name.

Figure 7 shows the accuracy results across command types. While most intent categories demonstrated high performance, task creation and date updates showed slightly lower task accuracy, pointing to some confusion in those areas.

A total of nine errors were identified, as detailed in Figure 8. The most common misclassifications included:

• “Update date” commands misclassified as “update status” (three cases);
• Unknown commands interpreted as “create task” (three cases);
• “Delete task” commands misclassified as “create task” (two cases);
• One “update status” command classified as unknown.

Figure 8. Error distribution.

Figure 8. Error distribution.

To facilitate comprehensive analysis, a confusion matrix was employed, as shown in Figure 9, providing a clear representation of both correctly and incorrectly classified instances.

The confusion matrix visualization clearly differentiates correct classifications, represented by green diagonal cells, from incorrect classifications, highlighted in red. The intensity of the colours reflects the frequency of occurrences to provide a visual insight into areas of strength and weakness of the tool. The confusion matrix revealed specific error patterns, such as three instances (15%) of “update_date” command incorrectly classified as “update_status”, three ambiguous or “unknown” commands misclassified as “create_task”, and one instance (5%) where the “update_status” command was incorrectly recognized as “unknown”.

The system achieved an overall accuracy of 89%, successfully classifying 71 out of the 80 tested commands. Performance varied across intent types, highlighting specific strengths and challenges. The detailed evaluation given in

Table 3. Detailed evaluation of intent.

Intent	Precision (%)	Recall (%)	F1-Score	Support
Create Task	77	100	87	17
Delete Task	100	90	95	20
Unknown	0	0	0	3
Update Date	100	85	92	20
Update Date	86	95	91	20

shows the precision, recall, F1-score, and support metrics, which further clarified the system’s performance across individual intent types.

The “create_task” intent demonstrated perfect recall (100%) but lower precision (77%), primarily due to the confusion and misclassification of ambiguous commands, resulting in an F1-score of 87%. Conversely, the “delete_task” intent exhibited flawless performance with 100% precision, recall, and an F1-score of 100%, reflecting its high reliability. The “unknown” intent faced significant challenges, yielding zero precision and recall, indicating the system’s inability to correctly classify ambiguous commands, resulting in an F1-score of 0%.

For the “update_date” intent, the system achieved perfect precision (100%) but slightly reduced recall (85%), leading to an F1-score of 92%, influenced by occasional misclassification as “update_status”. Lastly, the “update_status” intent showed robust performance, with high recall (95%) and considerable precision (86%), resulting in an F1-score of 91%.

In conclusion, while the system demonstrates strong overall capability in intent recognition, the detailed analysis highlighted key areas for further improvement. Specifically, future developments should focus on enhancing the system’s ability to manage ambiguous inputs and more accurately distinguish closely related intents such as “update_date” and “update_status”. These refinements will further increase accuracy, reliability, and operational effectiveness in practical applications.

6.3. Discussion of Validation Results and Limitations

While the validation results presented in Section 6.1 and Section 6.2 demonstrate promising potential for the VISA4D tool, several important limitations and challenges must be acknowledged for a comprehensive evaluation of the system’s capabilities and constraints. The future work that emerges from these identified limitations is detailed in Section 7.

6.3.1. Sample Size and Participant Demographics

The user testing involved 20 undergraduate and graduate civil engineering students from the University of Manitoba, which presents certain limitations. First, the relatively small sample size may not fully represent the broader population of potential users. Second, all participants come from an academic background with similar educational training, which may not accurately reflect how construction professionals with varied expertise and technical backgrounds would interact with the tool. Construction professionals, especially those with extensive field experience but limited exposure to advanced digital tools, might face different usability challenges than those encountered by engineering students who are typically more familiar with digital interfaces and BIM software.

Additionally, the testing was conducted in a controlled academic setting, which differs substantially from actual construction project environments. Real-world deployment would introduce additional variables that were not captured in the validation study.

6.3.2. Speech Recognition Challenges

As evidenced in both the user testing feedback and technical evaluation, speech recognition presented notable challenges. Participants with diverse accents reported occasional difficulties with command recognition, which aligns with the limitations identified in our technical assessment. The current speech recognition system exhibited significant recognition gaps for construction-specific terminology such as “grade beams”, “HVAC”, “MEP systems”, and specialized identifiers like “concrete pour #3”, which fall outside everyday language datasets on which most speech recognition models are trained.

Although our implementation of a construction-specific vocabulary enhancement within the NLP processor component improved recognition accuracy, this approach has inherent limitations, as it requires manual definition of construction terminology and does not automatically adapt to project-specific terminology. Additionally, construction terminology varies significantly across regions and projects, creating recognition challenges that would require continual system refinement.

The environmental conditions of typical construction sites also present practical challenges for speech recognition. Construction environments are often extremely loud, which would significantly impact speech recognition accuracy and may require users to repeat commands or prefer text input in noisy conditions over voice interaction.

6.3.3. Technical Infrastructure Constraints and Integration Challenges

The VISA4D implementation faces technical constraints despite its effectiveness in controlled testing environments. Approximately 15% of test sessions experienced Navisworks crashes due to API limitations, as the software was not originally designed for external access. The system’s reliance on trial API functionality creates persistent interoperability challenges, though users typically recovered by restarting the application.

Scalability concerns would emerge when deploying beyond individual sessions, as the local HTTP server may encounter performance bottlenecks with multiple simultaneous users. Construction projects require numerous team members to concurrently access and update schedule information, necessitating additional concurrency controls to manage conflicting modifications.

Moreover, the system’s reliance on OAuth for Autodesk Platform Services introduces dependency on internet connectivity for authentication and token renewal. While the Task Manager component implements local state caching to provide some offline functionality, extended periods without connectivity would render the system unresponsive. This limitation could be particularly problematic in remote construction sites with limited or intermittent internet access.

Finally, long-term viability requires continuous maintenance to preserve compatibility with evolving Navisworks platforms and API changes, representing an ongoing development challenge for the system.

6.3.4. Intent Recognition Accuracy Implications

The technical evaluation revealed specific weaknesses in intent recognition that warrant consideration for practical deployment. The “create_task” intent demonstrated perfect recall (100%) but lower precision (77%), indicating that the system occasionally misclassified other commands as task creation requests. In a real-world construction management context, such misclassifications could lead to unintended task additions, potentially introducing errors into project schedules that might go unnoticed until they impact workflow or resource allocation.

The system also performed poorly with “unknown” intents, yielding zero precision and recall, which reflects an inability to properly flag ambiguous commands. This limitation could lead to confusion in practical settings, where users might believe a command was successfully executed when, in fact, it was misinterpreted by the system. The confusion between closely related intents such as “update_date” and “update_status” (where three “update_date” commands were misclassified as “update_status”) could result in incorrect schedule information being recorded.

7. Discussion

The VISA4D framework presents a transformative step in the integration of conversational AI with 4D scheduling to support construction schedule updates and maintenance. The developed tool leverages NLP techniques and enables users to interact with the 4D BIM environment through voice or text commands, reducing the need for specialized technical expertise. This allows on-site personnel to update task details, such as dates, statuses, and interdependencies, directly from mobile devices. The interface is designed to be intuitive and user-friendly, particularly for users with limited experience using tools like Navisworks. When Navisworks is reopened, updates are automatically synchronized, ensuring that the project files remain up-to-date. This reduces administrative effort, supports real-time decision making, and improves communication among project team members. Moreover, the tool’s accessibility helps broaden participation in schedule management and reduces the learning curve, typically associated with BIM tools. In addition, its modular and scalable structure supports VISA4D’s application across a range of project sizes and complexities, making it a practical contribution to digital construction workflows.

The validation and verification results further support these benefits. User tests demonstrated high usability ratings, with 95% finding the tool easy to use and all participants confirming accurate command recognition. Additionally, the systematic verification testing revealed an overall system accuracy of 89% across different command types, with particularly strong performance in task deletion (100% accuracy) and task status updates (91% F1-score). These findings validate the practical effectiveness of the VISA4D framework for construction schedule updating and maintenance processes.

While the current implementation primarily interfaces with Navisworks’ TimeLiner functionality, VISA4D’s modular architecture is designed with extensibility in mind. This approach would allow for adaptation to support different scheduling methodologies, including pull scheduling and other lean construction techniques. The system’s remote access capabilities and ability to facilitate operational adjustments to project schedules make it particularly well suited for integration with lean construction tools.

As outlined in Section 6.3, the current version of the VISA4D framework has several limitations that necessitate further investigation and refinement. Future work should focus on addressing these limitations through several key initiatives. First, conducting expanded validation with industry professionals would provide valuable insights into performance and usability in real-world construction settings. Similarly, extending capability beyond Autodesk Navisworks to other 4D BIM platforms would enhance the tool’s applicability across different project environments. Developing adaptive learning techniques for the NLP component could improve recognition of construction-specific terminology without the need for manual configuration. This includes expanding the command vocabulary, improving context-awareness, and enabling automatic incorporation of new terms.

Future development should also address the scalability challenges by implementing robust concurrency controls and optimizing the system architecture for multi-user scenarios. Improvements to error handling and recovery mechanisms would also increase system resilience, especially during crashes. Finally, implementing privacy-preserving techniques, along with noise-resistant speech recognition algorithms, would mitigate security risks and improve performance in the noisy environments typical of construction sites.

8. Conclusions

This paper presents VISA4D, a novel framework that uses voice and text commands and AI algorithms to streamline real-time construction schedule updating and maintenance within 4D BIM environments. At its core, VISA4D integrates an AI-driven chatbot with Autodesk Navisworks, thereby enabling users to efficiently and practically generate, update, and maintain construction schedules. It addresses the limitations of traditional schedule updating processes in construction projects by integrating conversational AI with a 4D BIM tool, and bridges the gap between field operations and digital systems in the context of construction schedule updating and maintenance.

The VISA4D framework, as demonstrated through user testing, has the potential to simplify the complexities associated with 4D BIM operations. By making construction schedule creation and updating more accessible and intuitive, the system can empower a wider range of construction professionals to engage with 4D BIM tools. Its user-friendly approach promotes a more interactive and flexible method for managing construction schedules, ultimately supporting better collaboration and communication within construction project teams.

With the ongoing digital transformation of the industry, employing such advanced technologies is critical for improving the efficiency of construction projects while fostering collaboration and enhancing the overall quality of the project outcomes. In that sense, VISA4D represents a substantial leap forward in the realm of digital construction technologies and illustrates the potential for AI integration in construction. By facilitating direct, real-time engagement with 4D BIM models, it establishes a new benchmark for construction scheduling and project management.