PCLLM: An Integrated LLM-Driven System for Automating Desktop Operations via Direct Mouse and Keyboard Control

Abstract

The of personal computer (PC) tasks represents a systems-level challenge that integrates natural language processing, visual perception and mouse–keyboard action control. While existing approaches mainly focus on the application programming interface (API)-based or terminal-based automation, which are incompatible with the majority of applications for the lack of accessible interface. In this article, we propose PCLLM, a novel end-to-end system that automates PC operations by integrating large language models (LLMs) with computer vision techniques to directly control the mouse and keyboard. First, a software knowledge-based prompt engineering method is developed to comprehend software architecture and operational sequences. Second, template matching techniques are integrated for precise element localization, allowing the system to accurately identify and interact. Third, a dual-LLM pipeline is designed to automatically generate the test data, where a questioner LLM generates diverse task commands and the PCLLM executes these tasks, the corresponding process data are recorded automatically for performance evaluation. Finally, PCLLM is further validated through three typically PC applications (Notepad, Wordpad and Calculator), demonstrating its flexible and robust performance towards intelligent PC automation. To evaluate the proposed system, we adopt task completion rate as the primary metric. Experimental results show that PCLLM achieves the highest completion rates of 98.59%, 95.77%, and 52.11% on Notepad for basic, intermediate, and advanced tasks respectively when powered by GPT-4o, outperforming the CogAgent baseline. These results demonstrate the effectiveness of our approach for PC task automation.

1. Introduction

Driven by the growing complexity of software systems, the automation of personal computers (PCs) has become an increasingly important research topic [1] and offers multiple advantages such as improving efficiency. The automation of PCs refers to an algorithm or model that interacts with the software for task completion, thus improving user efficiency by offloading routine tasks [2]. Consequently, PC task automation holds significant potential for enhancing individual and organizational performance.

In the early stages of research, PC task automation relied on script-based or rule-based methods designed for specific tasks [3]. With the advancement of artificial intelligence, smarter voice assistants such as Siri and Alexa emerged, capable of following human instructions to perform simple assignments on mobile devices. Consequently, interactive environments have been predominantly developed for mobile scenarios [4,5] or web applications [6,7,8,9,10]. However, the voice assistants rely on predefined intents and parameter extraction from user queries which limits their adaptability to novel linguistic patterns or unanticipated contextual nuances. Furthermore, these environments are typically built on the Android ecosystem and designed for relatively simple and narrowly defined tasks.

With the advent of ChatGPT, large language model (LLM)-based automation methods have been broadly explored for tasks automation [11,12,13,14,15,16]. Additionally, human feedback reinforcement learning has been investigated to refine LLM behavior, leading to more accurate and context-aware interactions in complex scenarios [17]. While some benchmarks [18,19,20,21] have been extended to include more diverse operations, existing methods remain insufficient in capturing the full complexity and task diversity inherent in desktop computing environments.

Humans can interact with PCs using only a mouse and keyboard to accomplish a wide variety of complex tasks across multiple software environments. Therefore, equipping LLMs with the ability to accurately control mouse and keyboard represents a more effective and intelligent approach to PC automation. Specifically, a mouse-and-keyboard-based solution offers several advantages, including alignment with human intuition and applicability across virtually all software systems. Gao et al. [22] pioneered an LLM-based actor–critic framework named ASSISTGUI, which enables the automation of complex desktop tasks by parsing graphical user interface (GUI) and employing advanced reasoning techniques. However, ASSISTGUI employed several large neural networks for subtasks, requiring heavy resources while yielding suboptimal performance.

PC operation through mouse and keyboard control requires the model to develop two capabilities. First, the model possesses knowledge of the software being operated, including the hierarchical structure, the function of a specific button and the operation sequence to complete a given task. Second, the control model should be capable of accurately locating the target button or text elements in the GUI environment to perform precise mouse clicks. In this article, we propose a PC automation end-to-end system called PCLLM based on LLM technology. Specifically, the approach uses an LLM to parse user commands and generate corresponding action sequence in text format. Template matching is then employed to accurately locate GUI elements on the PC screen. Finally, the action sequence is converted into actual mouse and keyboard operations using a PC automation library like PyAutoGUI. In addition to PCLLM, we develop a novel method to automate the process of generating test data, which utilizes two LLMs acting as a “questioner” and an “executor”. The questioner generates diverse task commands, while the executor performs the tasks and records the corresponding mouse and keyboard actions. Through automated question–execute cycles, the two LLMs jointly produce test data.

This study addresses the following research questions. First, can an LLM-based system effectively automate PC operations by generating mouse and keyboard actions? Second, how do different LLMs (GPT-4o, GPT-4o-mini, Gemini-1.5-Pro, and Gemini-2.0-flash-exp) perform across tasks of varying complexity? The independent variable is the choice of LLM and task complexity (basic/intermediate/advanced), while the dependent variable is the task completion rate. Our hypothesis is that more capable LLMs (e.g., GPT-4o) will achieve higher completion rates, and that task complexity will negatively impact performance across all models.

The contributions of our article are concluded as follows:

• We propose a novel end-to-end PC task automation framework that leverages LLMs to operate the mouse and keyboard. The experimental results demonstrate the framework’s robust performance across three PC applications which are Wordpad, Notepad and Calculator, The experimental results demonstrate the framework’s robust performance across Notepad, Wordpad, and Calculator, achieving task completion rates of up to 98.59%, 91.55%, and 84.43% respectively when powered by GPT-4o.
• We present an advanced prompt engineering methodology that systematically integrates three software-related components which are software hierarchical architecture, button functional descriptions and few-shot examples. This prompt significantly enhances the LLM ability on understanding and executing PC tasks.
• A dual-LLM pipeline architecture is presented, in which one model generates diverse task descriptions while another executes and records the corresponding operations, demonstrating the feasibility of fully automated test data generation for PC automation tasks.
• We conduct extensive experimental validation using three standard PC applications (Notepad, WordPad, and Calculator), with quantitative results showing superior performance compared to existing approaches.

2. Related Work

Algorithm-driven PC task automation is an exciting and promising research topic that aims to enhance human productivity by freeing users from complex desktop environments. Existing approaches to PC task automation can be broadly categorized into three branches which are terminal-based methods using tools such as NL2Bash for Ubuntu [23,24], application programming interface (API)-based automation as seen in DroidBot-GPT along with AutoDroid [21,25], and keyboard-and-mouse operation automation exemplified by ASSISTGUI and CogAgent [22,26].

Terminal-based methods offer an advantage, as the input and output of a terminal take the form of text, which is compatible with the language processing ability of LLMs naturally. Lin et al. [23] proposed NL2Bash which translates natural language instructions into Linux command-line operations using LLMs, improving command accuracy and execution efficiency. Similarly, Liu et al. [24] developed a benchmark platform and employed reinforcement learning algorithms to adapt agents to diverse terminal tasks, achieving high completion rates and flexibility. However, terminal-based automation schemes impose limitations on the range of tasks such as file operations and system configuration, and most PC software are designed to operate in a desktop environment, making it impossible to obtain app information in a system terminal.

API-based automation methods directly interact with software-specific functions and publicly available API services to accomplish more complex tasks. Especially with the advent of ChatGPT, API-based methods have received significant attention due to the capability of LLMs to logically organize multiple API calls. Schick et al. [27] developed Toolformer, which was a pioneer of LLM-driven API-based automation models. The authors systematically developed methods for generating training data containing API calls and enabling the model to execute the API calls effectively. Similarly, several studies [21,28] also proposed language models with API calling capabilities. Notably, Qin et al. [28] introduced ToolLLM, which is capable of handling over 16,000 APIs. In general, the API-based automation methods effectively address the limitations of terminal-based methods by enabling direct access to application functions. However, the API-based approaches face inherent limitations due to API availability constraints, as most desktop software vendors prioritize GUI over programmatic interfaces for security and commercial reasons.

Compared with the previous two branches, GUI-based automation methods offer the most versatility, as they mimic human operation on a PC desktop via keyboard and mouse to accomplish a wide range of complex tasks. For example, Nakano et al. [29] proposed WebGPT which has the ability to simulate human mouse–keyboard operations to interact with a specially designed browser. Gao et al. [22] introduced ASSISTGUI, where an actor–critic framework is used to parse and manipulate GUIs. The experimental results demonstrated that ASSISTGUI achieved significant improvements over previous methods. However, WebGPT and ASSISTGUI demonstrate operational capabilities only within constrained desktop environments. Recently, Hong et al. [26] put forward CogAgent based on visual language models for PC automation. Then Claude 3.5 Sonnet put forward a “computer use” function for UI interactions, exemplifying the growing scholarly engagement with LLM-powered GUI agents. Soon OpenAI proposed a Computer-Using Agent and achieved state-of-the-art performance. However, it should be noted that Claude 3.5 Sonnet and the Computer-Using Agent are not open-source. In addition, many other enterprises have also proposed agents with similar computer-use capabilities, such as OK-Computer from Kimi, Wukong from Alibaba, WorkBuddy from Tencent. The aforementioned GUI models extend the capabilities of automation beyond terminal- and API-based methods by directly engaging with graphical elements, making well-trained model applicable to a broader range of desktop tasks.

The existing approaches mentioned above have advanced PC automation considerably, but several limitations remain. First, the terminal-based methods only support the text paradigm and are unavailable for GUI-driven workflows. Second, due to vendor restrictions, the application scope of API-based approaches is also limited. Finally, open-source models such as CogAgent [26] and ASSISTGUI [22] exhibit suboptimal performance, whereas closed-source models like Claude 3.5 Sonnet and Computer-Using Agent lack technical transparency. To address the challenges mentioned above, we propose PCLLM for PC task automation. The PCLLM leverages the capabilities of text-based LLMs while maintaining architectural simplicity.

3. Methodology

Figure 1 presents the comprehensive system architecture of PCLLM, highlighting its structure with three critical modules which are prompt engineering, LLM and terminal toolkit. First, the prompt engineering is specifically designed to encode software-related information into the LLM. Second, the LLM aims at two primary tasks: understanding user queries and generating action sequences. The action sequences adhere to a rigorously defined syntactic structure to facilitate automated translation into executable keyboard and mouse operations through programmatic interpretation. Then the template matching algorithm is employed to obtain accurate coordinates of mouse click targets. Finally, the system utilizes PyAutoGUI to automatically perform the corresponding mouse operations at precise screen coordinates, thereby completing the translation from linguistic commands to physical interactions.

3.1. Prompt Engineering

Prompt engineering serves as a critical technique to optimize the output quality and operational efficacy of LLMs in the context of PC task automation. In PCLLM, prompt engineering consists of three core components: software hierarchical architecture, button functional descriptions and few-shot examples.

Software Hierarchical Structure. The software hierarchical structure indicates which function would expand when a button is clicked. In PCLLM, the software hierarchical structure provides two advantages: providing prior knowledge of software and helping the LLM navigate the GUI. Take Notepad as an example, the LLM learns that activating the “Save As” function in Notepad requires first clicking on the “File” menu. In our design, the hierarchy is organized into a Python dictionary format to ensure the structure is easily parsed by the LLM.

Button Functionality and Hotkeys. In our work, button functionality and hotkeys are critical in enhancing LLM performance on PC automation tasks. Specifically, button functionality provides a short description of each software component. The hotkey refers to the keyboard shortcut that can directly invoke corresponding functions. This design brings a couple of benefits. First, the detailed button functionality description helps distinguish between similar functions, such as “Save” and “Save as” in Notepad. In addition, hotkeys offer shortcuts to directly invoke related functions thus improving operational efficiency significantly.

Few-Shot Examples. Few-shot examples are efficient methods for improving LLM performance in relevant downstream tasks. Based on the few-shot examples, the LLM is capable of completing similar tasks with high accuracy. In our work, we provide examples of some simple tasks. Each example provides a natural language instruction paired with the corresponding action sequence (e.g., “Search for ’invoice’ in the document” - hotkey(Ctrl+f)-input(invoice)-press(Enter)). Experimental results demonstrate that few-shot examples most significantly enhance model performance on test tasks.

Other Content. Finally, to better parse the action sequence into individual operation steps, the LLM is required to generate strictly formatted output. In our work, the prompt requires all actions to conform to a predefined form such as click(File) and hotkey(Ctrl+S). Each pattern comprises an action type and associated parameters of action as specified in

Table 1. The action space of an LLM model.

Function	Parameter	Instance
moveTo	button	MoveTo (File)
click	button	Click (File)
doubleclick	button	Doubleclick (File)
scroll	times	Scroll (10)
write	message	Write (Hello)
keyDown	key	KeyDown (UP)
keyUp	key	KeyUP (UP)
hotkey	* keys	Hotkey (Ctrl + c)

. Then all actions should be separated by a “-” symbol to indicate the sequence (e.g., click(File)-click(Save As)-input(report.txt)). Additionally, the prompt explicitly excludes natural language explanations or extraneous text from the output to ensure the result is directly parsed and executed by automation scripts.

3.2. GUI Element Localization via Template Matching

After obtaining the action sequence of a specific task, the next objective is to accurately locate GUI elements. For simplicity, we utilize the template matching technique to directly obtain corresponding coordinates of the clickable targets.

Template Matching. For non-text UI components, a template matching approach is employed based on normalized cross-correlation. Given template image T and screen region S, the similarity is shown in Equation (1), where $T(x^{\prime },y^{\prime })$ is the template image, $I(x,y)$ is the source image and $R(x,y)$ is the correlation result matrix. The detection confidence score at position $(x,y)$ is given by Equation (2). Then the confidence score is compared with a user-defined threshold $\tau$. If the confidence score exceeds the threshold, $C(x,y)$ is returned by template matching:

$$R(x,y)=-{\displaystyle \frac{{\sum }_{x^{\prime },y^{\prime }}\left(T(x^{\prime },y^{\prime })·I(x+x^{\prime },y+y^{\prime })\right)}{\sqrt{{\sum }_{x^{\prime },y^{\prime }}T{(x^{\prime },y^{\prime })}^2·{\sum }_{x^{\prime },y^{\prime }}I{(x+x^{\prime },y+y^{\prime })}^2}}}$$

(1)

$$C(x,y)=max\left(R\right(x,y\left)\right)\in [0,1]$$

(2)

3.3. Two LLMs for Automated Test Data Generation

To minimize manual effort in generating test data, a dual-LLM architecture is proposed to streamline the generation of test data as shown in Figure 2. The framework consists of two specialized LLMs, a questioner model that generates diverse task descriptions and an executor model that performs the corresponding GUI operations. The collaborative interaction between the two models establishes a closed-loop testing system that efficiently produces evaluation data.

In detail, the questioner is responsible for generating specific instructions in natural language. In our work, the tasks range from basic operations to complex tasks. Take Notepad as an example, where the basic operations include creating a new document, opening a new tab, etc., and the complex tasks include file management and text formatting problems. In addition, the questioner is encouraged to introduce systematic variations in task parameters to ensure comprehensive test coverage.

The executor receives task descriptions generated by the questioner and outputs corresponding action sequences. All sequences strictly comply with the prescribed structure to maintain interoperability with the automated execution framework. During execution, the monitoring system captures detailed procedural records containing all mouse and keyboard actions with precise step numbers and visual information through sequential screenshots.

3.4. Materials

All experiments were conducted on a Windows 11 operating system. The screen resolution was fixed at 1920 × 1080 with 100% scaling to ensure consistent GUI layout. The automation framework was implemented in Python 3.10. Key libraries included PyAutoGUI (version 0.9.54) for mouse and keyboard control and OpenCV-Python (version 4.8.1) for template matching. All large language model inferences were performed via official cloud application programming interfaces; no local graphics processing unit was required. The specific LLMs and their API versions are detailed in Section 4.2. The three target applications were the native Windows 11 versions of Notepad, WordPad, and Calculator.

4. Experiment

4.1. Experimental Environment

Our experiment employs Notepad, WordPad, and Calculator as representative cases for comprehensive performance analysis of PCLLM. Specifically, Notepad and WordPad are mainly used to evaluate adaptability and generalizability of our method. The Calculator is used to assess the logical ability of the LLM because the memory-related function requires the model to fully understand how to use the memory function to store and record the intermediate values of the computation. An example of the running process of our system is shown in Figure 3.

All template images are manually pre-captured for each software application in this work. For each clickable UI element, one template image is stored. During execution, if template matching fails (confidence below threshold $\tau$, which is set as 0.95 in our experiment), the system skips the current action and proceeds to the next one.

4.2. Baseline and Comparison

In this part, experiments are designed to evaluate capability of LLMs in understanding human instruction and performing complex desktop operations. In addition, another target is to select the optimal base model for our framework. Specifically, we choose four LLMs: GPT-4o, GPT-4o-mini, Gemini-1.5-Pro and Gemini-2.0-flash-exp to evaluate our scheme. These models are selected for three reasons. OpenAI and Google are two of the leading LLM providers, making their models representative of the state of the art. The selected models span different capability levels, from flagship to lightweight and efficiency-oriented designs, allowing a systematic evaluation across diverse LLM backbones. Additionally, all selected models provide public APIs, which ensures the reproducibility of our experiments. It should be noted that each model is evaluated under the same task scenarios to ensure the consistency and comparability of experimental results and. For each LLM and each application, we conducted 216 tasks in total, with 72 tasks for each complexity level (basic, intermediate, and advanced).

The test tasks are stratified into three complexity levels to systematically evaluate model performance across varying difficulty spectra: basic, intermediate, and advanced. For Notepad and WordPad, basic tasks typically include straightforward actions such as opening a new tab, saving the current file with a new filename, or pasting the clipboard content into the cursor position in the software. Intermediate tasks involve the combination of two basic tasks such as “open a new tab and save it as ’new-file.txt’.” The advanced tasks are the combination of three to five basic operations across multiple basic tasks. For the Calculator application, basic tasks involve fundamental arithmetic operations on binomial expressions (e.g., “151 plus 31 = ?”), while intermediate tasks require evaluating composite arithmetic expressions where at least one term is an arithmetic operation rather than a single number (e.g., “15 plus (101 multiplied by 21) = ?”). Advanced tasks involve composite arithmetic expressions where each term is an expression such as “(13 plus 21) divided by (116 multiplied by 13) = ?” or “(13 plus 21) divided by (116 multiplied by 13) multiplied by (2 minus 5) = ?”. The intermediate and advanced tasks of the calculator require the use of memory-related operations, which can be challenging even for human users.

We choose the task completion rate as an evaluation metric to assess model performance. The task completion rate is defined as the number of tasks completed divided by the total number of tasks. In addition, for different software, different methods are utilized to calculate the task completion rate. Specifically, for applications like Notepad and WordPad, a manual annotation method is applied. The human evaluators manually assess whether the task is completed by observing the action sequence generated by the LLM and corresponding screenshot transitions. For the Calculator, a two-stage evaluation method is developed to automatically judge whether the task is finished correctly. First, the mathematical expression is calculated by a high-precision calculator API. Then the last screenshot from the sequence is recognized using the optical character recognition (OCR) technique to get the mathematical result produced by the LLM action sequence. Finally, the success rate can be calculated by the percentage of trials where OCR-extracted results match API-based output within the preset floating-point tolerance.

The corresponding experimental results on the three software applications are shown in Figure 4, Figure 5 and Figure 6. First, the results demonstrate a negative correlation between task complexity and LLM operation accuracy. For example, the success rate of GPT-4o falls from 0.98 on basic tasks to 0.52 on advanced tasks on Notepad as shown in Figure 4. On the Calculator Gemini-1.5-pro-latest drops from 0.90 on basic tasks to 0.49 on advanced tasks as shown in Figure 5. The results suggest that as the length of operation sequence grows, the ability of the LLM to accurately predict the subsequent action diminishes, revealing limitations in handling complex operational workflows. Second, GPT-4o achieves the highest success rate on Notepad and WordPad as shown in Figure 4 (e.g., 0.96 basic, 0.87 intermediate, 0.28 advanced on Notepad for Gemini-1.5-pro vs. 0.98/0.95/0.52 for GPT-4o), whereas Gemini-1.5-pro exhibits superior performance on Calculator as shown in Figure 5 (0.89 on basic, 0.80 on intermediate, 0.49 on advanced, versus 0.84/0.50/0.56 of GPT-4o). The phenomenon suggests that the capabilities of LLMs are dependent on task-specific domains. GPT-4o excels in daily tasks on PC systems and Gemini-1.5-pro demonstrates stronger logical reasoning ability.

To verify that the observed performance differences are not due to random chance, we conducted Kruskal–Wallis tests on the task-level success/failure data (216 tasks per model per application). Details are given in

Table 2. Statistical analysis results of pairwise model comparisons.

Application	Comparison	H-Test p	MW p	Sig.
Notepad	GPT-4o vs. GPT-4o-mini	<$0.001$	<$0.001$	***
	GPT-4o vs. Gemini-1.5-Pro		<$0.001$	***
	GPT-4o vs. Gemini-2.0-flash-exp		$0.002$	**
	GPT-4o-mini vs. Gemini-1.5-Pro		$0.011$	*
	GPT-4o-mini vs. Gemini-2.0-flash-exp		<$0.001$	***
	Gemini-1.5-Pro vs. Gemini-2.0-flash-exp		$0.357$	n.s.
Calculator	GPT-4o vs. GPT-4o-mini	<$0.001$	<$0.001$	***
	GPT-4o vs. Gemini-1.5-Pro		<$0.001$	***
	GPT-4o vs. Gemini-2.0-flash-exp		$0.002$	**
	GPT-4o-mini vs. Gemini-1.5-Pro		<$0.001$	***
	GPT-4o-mini vs. Gemini-2.0-flash-exp		<$0.001$	***
	Gemini-1.5-Pro vs. Gemini-2.0-flash-exp		<$0.001$	***
WordPad	GPT-4o vs. GPT-4o-mini	$0.003$	<$0.001$	***
	GPT-4o vs. Gemini-1.5-Pro		$0.018$	*
	GPT-4o vs. Gemini-2.0-flash-exp		$0.008$	**
	GPT-4o-mini vs. Gemini-1.5-Pro		$0.204$	n.s.
	GPT-4o-mini vs. Gemini-2.0-flash-exp		$0.330$	n.s.
	Gemini-1.5-Pro vs. Gemini-2.0-flash-exp		$0.768$	n.s.

*** $p<0.001$, ** $p<0.01$, * $p<0.05$, n.s. = not significant.

. The results showed significant differences among the four LLMs across all three applications (p < 0.001 for Notepad and Calculator; p = 0.003 for WordPad). Post hoc Mann–Whitney U tests with Bonferroni correction revealed that GPT-4o significantly outperformed all other models on Notepad and WordPad (p < 0.05), and also outperformed GPT-4o-mini and Gemini-2.0-flash-exp on Calculator (p < 0.01). Gemini-1.5-Pro showed competitive performance on Calculator, while no significant difference was found between Gemini-1.5-Pro and Gemini-2.0-flash-exp on Notepad and WordPad (p > 0.05). These results confirm that the performance gaps are statistically reliable.

In addition to the closed-source LLMs like GPT, several Qwen-series open-source language models are also tested as the core in our framework. The open-source language models from the Qwen family offer benefits such as lightweight design and lower operation cost, which make them suitable for use in limited resource environments. However, the open-source models’ performance on complex tasks is inferior to closed-source models such as GPT series and Gemini Series. Hence, the trade-off between model capacity and hardware performance should be carefully considered. Specifically, the Qwen models selected for our experiments are Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct and Qwen2.5-32B-Instruct. And the Qwen models are tested on Notepad and Calculator respectively. The experimental results are shown in

Table 3. Performance of different Qwen models on Notepad tasks.

Model	Task Complexity	Success Rate (%)
GPT-4o	basic	98.59
	intermediate	95.77
	advanced	52.11
Qwen2.5-32B-Instruct	basic	91.55
	intermediate	69.01
	advanced	45.07
Qwen2.5-14B-Instruct	basic	85.91
	intermediate	64.79
	advanced	36.62
Qwen2.5-7B-Instruct	basic	56.34
	intermediate	53.52
	advanced	28.17

and

Table 4. Performance of different Qwen models on Calculator tasks.

Model	Task Complexity	Success Rate (%)
Gemini-1.5-pro-latest	basic	89.53
	intermediate	80.19
	advanced	48.62
Qwen2.5-32B-Instruct	basic	70.21
	intermediate	19.84
	advanced	16.90
Qwen2.5-14B-Instruct	basic	42.23
	intermediate	12.68
	advanced	8.51
Qwen2.5-7B-Instruct	basic	87.32
	intermediate	12.67
	advanced	2.82

.

Two conclusions can be drawn from Table 3 and Table 4. First, the performance of closed-source models significantly outperforms open-source models, especially on advanced tasks. Specifically, the task completion rates of GPT-4o on Notepad are 98.59%, 95.77%, and 52.11% from basic to significantly outperforming open-source models, especially on advanced tasks. In comparison, the success rates of Qwen2.5-32B-Instruct drop to 91.55%, 69.01% and 45.07% respectively. Similarly, the same trend is observed on Calculator application between Gemini-1.5-pro-latest and Qwen-series models in Table 4. In addition, the performance improves significantly as the parameter size of the models increases in the Qwen2.5 family, which is easily observed from Table 3 and Table 4. In conclusion, the results highlight the trade-off between resource efficiency and model capability on complicated user commands.

4.3. Ablation Study

An ablation study is also conducted on the different parts of the prompts. The experiment is designed to identify how the inclusion of each part of the prompt impacts the performance of LLMs. The original prompt used in our scheme includes four parts: the hierarchical structure of the software, the functionality of each clickable button, the hotkey usage and the few-shot examples for specific tasks. In the ablation study, the experiment is set up with four configurations: (1) removing the hierarchical structure of software, (2) omitting the few-shot examples for task operation, (3) excluding the functionality of the clickable button, and (4) using only the task description, without any supplementary prompts. Regarding the experimental setup, two LLMs that possess the best performance in the baseline experiments on Notepad are evaluated to ensure the reliability of the ablation study. Each ablation evaluation includes 72 individual test tasks.

Based on the experimental results shown in

Table 5. Task completion rate for different ablation parts and models.

Model	Ablation Part	Task Complexity	Complication Rate (%)
GPT-4o	full prompt	basic	98.59
		intermediate	95.77
		advanced	52.11
	remove architecture	basic	94.37
		intermediate	67.61
		advanced	21.13
	no button function	basic	98.59
		intermediate	97.18
		advanced	49.30
	no hotkey	basic	97.22
		intermediate	90.27
		advanced	48.61
	no few-shot example	basic	91.55
		intermediate	59.15
		advanced	15.49
	only basic desc	basic	47.89
		intermediate	7.04
		advanced	0.00
Gemini-2.0-flash-exp	full prompt	basic	92.96
		intermediate	87.32
		advanced	28.17
	remove architecture	basic	92.96
		intermediate	76.06
		advanced	5.63
	no button function	basic	94.37
		intermediate	78.87
		advanced	22.54
	no hotkey	basic	91.66
		intermediate	77.78
		advanced	25.00
	no few-shot example	basic	95.77
		intermediate	77.46
		advanced	12.68
	only basic desc	basic	35.21
		intermediate	4.23
		advanced	0.00

, two key conclusions are drawn regarding the model performance. First, prompt engineering plays an essential role in enhancing the LLM ability to interact with software, as the performance drops sharply when only basic task descriptions are utilized. For GPT-4o, the task completion rate plummeted to 47.89%, 7.04%, and 0% for basic, intermediate, and advanced tasks respectively without supplementary prompts, compared to 98.59%, 95.77%, and 52.11% with the full prompt. The same phenomenon is observed for Gemini-2.0-flash-exp. Second, the hierarchical software architecture and detailed operational steps (few-shot examples) are of comparable importance, while button functionalities and hotkey descriptions play a relatively minor role. For instance, removing the hierarchical structure caused the performance of GPT-4o on advanced tasks to drop sharply from 52.11% to 21.13%, and omitting operational steps led to an even more dramatic decline to 15.49%. In contrast, excluding button functionalities has a mild impact, with the advanced task rate decreasing only slightly from 52.11% to 49.30%. The same pattern is observed for Gemini-2.0-flash-exp. The hierarchical structure ablation reduced advanced task performance from 28.17% to 5.63%, while the absence of operational steps lowered it to 12.68%. Button functionality and hotkey usage methods exclusion resulted in a smaller reduction to 22.54%. The comprehensive ablation study results are systematically presented in Table 5, demonstrating the contribution of each component.

From the experimental results presented above, it can be concluded that the specially designed prompt significantly enhances LLM performance on PC automation downstream tasks. In addition, the results also provide valuable insights for refining prompt engineering techniques for future research, aiming at helping LLMs to better interact with desktop environments in complex GUI scenarios.

4.4. Comparison with CogAgent

A comparison experiment is conducted between PCLLM and CogAgent which is a leading open-source model in the field of GUI-based PC task automation. Specifically, we adopt the CogAgent-9B variant. The evaluation uses the same 216 Notepad tasks (72 per complexity level: basic, intermediate, and advanced) as in the main experiments. CogAgent is deployed on a GPU server (NVIDIA H800) and accessed via its API. Task completion is determined by the final screen state following the same protocol used for PCLLM. In our experiments, CogAgent is integrated into the same test environment to evaluate its performance alongside the LLM-based automation system proposed in this paper. Details of the experimental results are shown in

Table 6. Task success rates on Notepad for different schemes.

Scheme	Task Type	Success Rate (%)
CogAgent	Basic	45.07
	Intermediate	8.45
	Advanced	0.00
Our scheme, (GPT-4 as decision model)	Basic	98.59
	Intermediate	95.77
	Advanced	52.11
Our scheme, (Qwen2.5-7B-Instruct as decision model)	Basic	56.34
	Intermediate	53.52
	Advanced	28.17

.

The results show that PCLLM outperforms CogAgent by 53.52% on basic tasks. It should be noted that CogAgent has 9B parameters. Even when integrating the Qwen2.5-7B-Instruct into our framework, performance of PCLLM still exceeds CogAgent across diverse task types, which is attributed to our innovative approach in prompt engineering and the integration of template matching techniques for accurate GUI interaction. In contrast, CogAgent’s performance decreases significantly as the task complexity increases. Consequently, although CogAgent is a robust baseline solution, our experimental findings highlight how flexible system architecture and prompt design can deliver superior reliability and accuracy in automating complex tasks across varied desktop environments.

5. Discussion

Our experimental results demonstrate three key findings. First, LLM-based PC automation is feasible, with PCLLM achieving high task completion rates across Notepad, WordPad, and Calculator. Second, model capability significantly impacts performance: GPT-4o consistently outperforms other LLMs, particularly on complex tasks. Third, prompt engineering plays a critical role as evidenced by the ablation study where removing prompt components caused substantial performance drops.

Regarding our research questions, we find that: (1) an LLM-based system can effectively automate PC operations by generating mouse and keyboard actions, and (2) task complexity negatively impacts all models, though more capable LLMs (e.g., GPT-4o) maintain higher success rates on complex tasks. These results confirm our hypothesis.

For users and practitioners, PCLLM offers a practical solution for automating repetitive desktop tasks without requiring API access or terminal interfaces. The system can benefit knowledge workers, accessibility tools, and software testing scenarios. However, current limitations—manual prompt engineering, template sensitivity to resolution changes, and inability to handle pop-up dialogs—mean that deployment is best suited for controlled environments with stable UI layouts.

Despite achieving high accuracy, our approach has several limitations. First, it requires manually crafted prompts and pre-captured button templates for each application, lacking generalizability across different software. Second, NCC-based template matching is sensitive to resolution changes and window scaling, with no dynamic template update mechanism. Third, the LLM generates the entire action sequence upfront, preventing it from handling unexpected screen changes such as pop-up dialogs. Fourth, the approach struggles with complex applications and web browsing tasks where prompt engineering becomes infeasible.

To address these limitations, future work will focus on two directions. First, for robust GUI localization, we plan to replace NCC-based matching with multi-scale matching, adaptive thresholds, and lightweight object detectors such as YOLOv8 [30] and YOLO-TCS [31]. Feature-based retrieval methods, such as feature-aware multi-head self-attention hashing [32] and adaptive pyramid vision transformer (Adaptive-PVT) [33] also offer promising directions. Second, to enable step-by-step decision-making, we plan to integrate multimodal LLMs that can perceive screen states and handle unexpected changes.

6. Conclusions

PC automation is an important research topic and has the potential to free human effort for more innovative work. In this article, we propose a novel PC automation framework named PCLLM that consists of three parts: prompt engineering, LLM and PC automation tools like PyAutoGUI. Specifically, the prompt engineering provides software-related knowledge to the LLM to improve model performance. The LLM generates correct action sequences to accurately accomplish user instructions. Finally, the PC automation tools are responsible for converting action sequences into mouse and keyboard operations on the computer. We test PCLLM on three PC software applications. Experimental results demonstrate the feasibility of our scheme. Specifically, PCLLM achieves task completion rates of up to 98.6%, 95.8%, and 52.1% on Notepad for basic, intermediate, and advanced tasks respectively when powered by GPT-4o, outperforming both other LLMs and the CogAgent baseline. Our future work will focus on enhancing the accuracy and robustness of the PCLLM framework.