Search Results (22)

Multi-agent reinforcement learning has demonstrated excellent performance in complex decision-making tasks such as electronic games, power grid management, and autonomous driving. However, its vulnerability to adversarial attacks may impede its widespread application. Currently, research on adversarial attacks in reinforcement learning primarily focuses on single-agent scenarios, while studies in multi-agent settings are relatively limited, especially regarding how to achieve optimized attacks with fewer steps. This paper aims to bridge the gap by proposing a heuristic exploration-based attack method named the Search for Key steps and Key agents Attack (SKKA). Unlike previous studies that train a reinforcement learning model to explore attack strategies, our approach relies on a constructed predictive model and a T-value function to search for the optimal attack strategy. The predictive model predicts the environment and agent states after executing the current attack for a certain period, based on simulated environment feedback. The T-value function is then used to evaluate the effectiveness of the current attack. We select the strategy with the highest attack effectiveness from all possible attacks and execute it in the real environment. Experimental results demonstrate that our attack method ensures maximum attack effectiveness while greatly reducing the number of attack steps, thereby improving attack efficiency. In the StarCraft Multi-Agent Challenge (SMAC) scenario, by attacking 5–15% of the time steps, we can reduce the win rate from 99% to nearly 0%. By attacking approximately 20% of the agents and 24% of the time steps, we can reduce the win rate to around 3%. Full article

With the development of value decomposition methods, multi-agent reinforcement learning (MARL) has made significant progress in balancing autonomous decision making with collective cooperation. However, the collaborative dynamics among agents are continuously changing. The current value decomposition methods struggle to adeptly handle these dynamic changes, thereby impairing the effectiveness of cooperative policies. In this paper, we introduce the concept of latent interaction, upon which an innovative method for generating weights is developed. The proposed method derives weights from the history information, thereby enhancing the accuracy of value estimations. Building upon this, we further propose a dynamic masking mechanism that recalibrates history information in response to the activity level of agents, improving the precision of latent interaction assessments. Experimental results demonstrate the improved training speed and superior performance of the proposed method in both a multi-agent particle environment and the StarCraft Multi-Agent Challenge. Full article

In the context of real-time strategy video games like StarCraft II, strategic decision-making is a complex challenge that requires adaptability and precision. This research creates a mixed recommendation system that uses causal models and deep neural networks to improve its ability to suggest the best strategies based on the resources and conditions of the game. PySC2 and the official StarCraft II API collected data from 100 controlled matches, standardizing conditions with the Terran race. We created fake data using a Conditional Tabular Generative Adversarial Network to address data scarcity situations. These data were checked for accuracy using Kolmogorov–Smirnov tests and correlation analysis. The causal model, implemented with PyMC, captured key causal relationships between variables such as resources, military units, and strategies. These predictions were integrated as additional features into a deep neural network trained with PyTorch. The results show that the hybrid system is 1.1% more accurate and has a higher F1 score than a pure neural network. It also changes its suggestions based on the resources it has access to. However, certain limitations were identified, such as a bias toward offensive strategies in the original data. This approach highlights the potential of combining causal knowledge with machine learning for recommendation systems in dynamic environments. Full article

In high-speed crafts, whisker spray increases viscous resistance by enlarging the wetted surface near the stagnation line. Spray rails (SRs) mitigate this issue by redirecting water flow, reducing the wetted surface, and lowering overall resistance. This study investigates the effect of SRs on the hydrodynamic performance of the C1 hull of Naples Systematic Series (NSS), focusing on the systematic variations in size, number, and placement. Numerical simulations, validated with towing tank results, were conducted using STAR CCM+ 2306. Mesh independence analysis was also performed to optimize computational efficiency. Key findings highlight the critical role of SR design in performance optimization. Wider SRs (e.g., three per side, 0.96% LWL) reduced resistance by up to 8.5% at high speeds (${Fr}_{\nabla }=$ 3.26), but slightly increased the resistance at lower speeds (~2%) due to a larger wetted surface. Narrower SRs (e.g., three per side, 0.48% LWL) achieved resistance reductions of up to 4.6%, while configurations with multiple SRs (e.g., three per side, 0.72% LWL) outperformed single-rail designs by reducing resistance up to 4%. Placement near the chine proved more effective than near the keel, offering a 4% additional reduction in resistance. Additionally, SRs generated lift, raising the hull, and reducing immersion. The study underscores the importance of optimizing SR size, number, and placement to enhance hydrodynamic efficiency, particularly for high-speed operations. Full article

Multi-agent reinforcement learning (MARL) has proven to be effective and promising in team collaboration tasks. Knowledge transfer in MARL has also received increasing attention. Compared to knowledge transfer in single-agent tasks, knowledge transfer in multi-agent tasks is more complex due to the need to account for coordination among agents. However, existing knowledge transfer-based methods only focus on strategies or agent-level knowledge in a single task, and the transfer of knowledge in such a specific task to new and different types of tasks is likely to fail. In this paper, we propose a multitask-based training framework termed MTT in cooperative MARL, which aims to learn shared collaborative knowledge across multiple tasks simultaneously and then apply it to solve other related tasks. However, models obtained from multitask learning may fail on other tasks because the gradients from different tasks may conflict with each other. To obtain a model with shared knowledge, we provide conflict-free updates by ensuring a positive dot product between the final update and the gradient of each specific task. It also maintains a consistent optimization rate for all tasks. Experiments conducted in two popular environments, StarCraft II Multi-Agent Challenge and Google Research Football, demonstrate that our method outperforms the baselines, significantly improving the efficiency of team collaboration. Full article

Background: This article introduces SC-Phi2, a fine-tuned StarCraft II small language model. Small language models, like Phi2, Gemma, and DistilBERT, are streamlined versions of large language models (LLMs) with fewer parameters that require less computational power and memory to run. Method: To teach Microsoft’s Phi2 model about StarCraft, we create a new SC2 text dataset with information about StarCraft races, roles, and actions and use it to fine-tune Phi-2 with self-supervised learning. We pair this language model with a Vision Transformer (ViT) from the pre-trained BLIP-2 (Bootstrapping Language Image Pre-training) model, fine-tuning it on the StarCraft replay dataset, MSC. This enables us to construct dynamic prompts that include visual game state information. Results: Unlike the large models used in StarCraft LLMs such as GPT-3.5, Phi2 is trained primarily on textbook data and contains little inherent knowledge of StarCraft II beyond what is provided by our training process. By using LoRA (Low-rank Adaptation) and quantization, our model can be trained on a single GPU. We demonstrate that our model performs well at build order prediction, an important StarCraft macromanagement task. Conclusions: Our research on the usage of small models is a step towards reducing the carbon footprint of AI agents. Full article

The Hunter–Prey Optimization (HPO) algorithm represents a novel population-based optimization approach renowned for its efficacy in addressing intricate problems and optimization challenges. Photovoltaic (PV) systems, characterized by multi-peaked shading conditions, often pose a challenge to conventional maximum power point tracking (MPPT) techniques in accurately identifying the global maximum power point. In this research, an MPPT control strategy grounded in an improved Hunter–Prey Optimization (IHPO) algorithm is proposed. Eight distinct shading scenarios are meticulously crafted to assess the feasibility and effectiveness of the proposed MPPT method in capturing the maximum power point. A performance evaluation is conducted utilizing both MATLAB/simulation and an embedded system, alongside a comparative analysis with alternative power tracking methodologies, considering the diverse climatic conditions across different seasons. The simulation outcomes demonstrate the capability of the proposed control strategy in accurately tracking the global maximum power point, achieving a commendable efficiency of 100% across seven shading conditions, with a tracking response time of approximately 0.2 s. Verification results obtained from the experimental platform illustrate a tracking efficiency of 98.75% for the proposed method. Finally, the IHPO method’s output performance is evaluated on the StarSim Rapid Control Prototyping (RCP) platform, indicating a substantial enhancement in the tracking efficiency of the photovoltaic system while maintaining rapid response times. Full article

Due to the advantages of a centralized critic to estimate the Q-function value and decentralized actors to optimize the agents’ policies, counterfactual multi-agent (COMA) stands out in most multi-agent reinforcement learning (MARL) algorithms. The sharing of policy parameters can improve sampling efficiency and learning effectiveness, but it may lead to a lack of policy diversity. Hence, to balance parameter sharing and diversity among agents in COMA has been a persistent research topic. In this paper, an effective training method for a COMA policy network based on a differential evolution (DE) algorithm is proposed, named DE-COMA. DE-COMA introduces individuals in a population as computational units to construct the policy network with operations such as mutation, crossover, and selection. The average return of DE-COMA is set as the fitness function, and the best individual of policy network will be chosen for the next generation. By maintaining better parameter sharing to enhance parameter diversity, multi-agent strategies will become more exploratory. To validate the effectiveness of DE-COMA, experiments were conducted in the StarCraft II environment with 2s_vs_1sc, 2s3z, 3m, and 8m battle scenarios. Experimental results demonstrate that DE-COMA significantly outperforms the traditional COMA and most other multi-agent reinforcement learning algorithms in terms of win rate and convergence speed. Full article

This paper addresses an alternative perspective on tourism success, emphasising sustainability over traditional quantitative metrics such as arrival numbers. It explores the impact of fiction films and TV series on individuals’ mental representations of destinations featured on screen, as well as the capacity of film discourse to construct a brand aligned with local stakeholders’ interests. Qualitative methods have been employed, conducting a literature review on sustainable film tourism and destination branding. Local news and an interview with the head of the Peñíscola Film Office complemented academic insights. The primary goal is to examine the “Peñíscola de Cine” project as a paradigm of success, initiated by the city council of Peñíscola, Spain. This project positions the municipality as a natural film set through productions like Game of Thrones (2011–2019), illustrating how film can contribute to destination branding and community engagement. The study highlights the positive contribution of film tourism to sustainability by diversifying and de-seasonalising a territory’s offerings. It also attracts a more educated and environmentally conscious audience. However, it cautiously discusses the potential risks, as evidenced by misapplications in Goathland, England, and Skellig Michael, Ireland, following their appearances in Heartbeat (1992–2010) and Star Wars (1977–2019), respectively. The paper concludes by suggesting film-friendly measures for destination management organizations (DMOs), emphasising the pivotal role of film commissions and film offices in crafting effective marketing strategies and capturing the interest of audiovisual production companies. Full article

To deploy Unmanned Aerial Vehicles (UAVs) inside heterogeneous GPS-denied confined (potentially unknown) spaces, such as those encountered in mining and Urban Search and Rescue (USAR), requires the enhancement of numerous technologies. Of special interest is for UAVs to identify collision-freeSafe Flight Corridors (SFC${}^{+}$) within highly cluttered convex- and non-convex-shaped environments, which requires UAVs to perform advanced flight maneuvers while exploiting their flying capabilities. Within this paper, a novel auxiliary occupancy checking process that augments traditional 3D flight corridor generation is proposed. The 3D flight corridor is established as a topological structure based on a hand-crafted path either derived from a computer-generated environment or provided by the human operator, which captures humans’ preferences and desired flight intentions for the given space. This corridor is formulated as a series of interconnected overlapping convex polyhedra bounded by the perceived environmental geometries, which facilitates the generation of suitable 3D flight paths/trajectories that avoid local minima within the corridor boundaries. An occupancy check algorithm is employed to reduce the search space needed to identify 3D obstacle-free spaces in which their constructed polyhedron geometries are replaced with alternate convex polyhedra. To assess the feasibility and efficiency of the proposed SFC${}^{+}$ methodology, a comparative study is conducted against the Star-Convex Method (SCM), a prominent algorithm in the field. The results reveal the superiority of the proposed SFC${}^{+}$ methodology in terms of its computational efficiency and reduced search space for UAV maneuvering solutions. Various challenging confined-environment scenarios, each with different obstacle densities (confined scenarios), are utilized to verify the obtained outcomes. Full article

A mathematical model for predicting the maneuvering motion of a ship is constructed, using a planing craft with dual waterjet propulsion as the object of study. This model is based on the standard approach of the MMG (Maneuvering Modeling Group) and uses the Runge–Kutta algorithm to solve the differential equations. For the simulation of the turning and Z-shape maneuvering motion, the RANS equation is first solved using the program STAR-CCM + and then the PMM motion of the hull is simulated using the overlapping grid approach to derive the hydrodynamic derivative. The established method for predicting the ship’s maneuverability is feasible, as shown by the calculated results, which agree well with those obtained using data from the sea trials. This method was used to simulate the rudder rotation and Z-shape motion of the planing craft at medium and high speeds to predict the maneuverability index. Full article

The latest achievements in the field of reinforcement learning have encouraged the development of vision-based learning methods that compete with human-provided results obtained on various games and training environments. Convolutional neural networks together with Q-learning-based approaches have managed to solve and outperform human players in environments such as Atari 2600, Doom or StarCraft II, but the niche of 3D realistic games with a high degree of freedom of movement and rich graphics remains unexplored, despite having the highest resemblance to real-world situations. In this paper, we propose a novel testbed to push the limits of deep learning methods, namely an OpenAI Gym-like environment based on Dark Souls III, a notoriously difficult role-playing game, where even human players have reportedly struggled. We explore two types of architectures, Deep Q-Network and Deep Recurrent Q-Network, providing the results of a first incursion into this new problem class. The source code for the training environment and baselines is made available. Full article

Electricity demands are increasing significantly and the traditional power grid system is facing huge challenges. As the desired next-generation power grid system, smart grid can provide secure and reliable power generation, and consumption, and can also realize the system’s coordinated and intelligent power distribution. Coordinating grid power distribution usually requires mutual communication between power distributors to accomplish coordination. However, the power network is complex, the network nodes are far apart, and the communication bandwidth is often expensive. Therefore, how to reduce the communication bandwidth in the cooperative power distribution process task is crucially important. One way to tackle this problem is to build mechanisms to selectively send out communications, which allow distributors to send information at certain moments and key states. The distributors in the power grid are modeled as reinforcement learning agents, and the communication bandwidth in the power grid can be reduced by optimizing the communication frequency between agents. Therefore, in this paper, we propose a model for deciding whether to communicate based on the causal inference method, Causal Inference Communication Model (CICM). CICM regards whether to communicate as a binary intervention variable, and determines which intervention is more effective by estimating the individual treatment effect (ITE). It offers the optimal communication strategy about whether to send information while ensuring task completion. This method effectively reduces the communication frequency between grid distributors, and at the same time maximizes the power distribution effect. In addition, we test the method in StarCraft II and 3D environment habitation experiments, which fully proves the effectiveness of the method. Full article

Leveraging global state information to enhance policy optimization is a common approach in multi-agent reinforcement learning (MARL). Even with the supplement of state information, the agents still suffer from insufficient exploration in the training stage. Moreover, training with batch-sampled examples from the replay buffer will induce the policy overfitting problem, i.e., multi-agent proximal policy optimization (MAPPO) may not perform as good as independent PPO (IPPO) even with additional information in the centralized critic. In this paper, we propose a novel noise-injection method to regularize the policies of agents and mitigate the overfitting issue. We analyze the cause of policy overfitting in actor–critic MARL, and design two specific patterns of noise injection applied to the advantage function with random Gaussian noise to stabilize the training and enhance the performance. The experimental results on the Matrix Game and StarCraft II show the higher training efficiency and superior performance of our method, and the ablation studies indicate our method will keep higher entropy of agents’ policies during training, which leads to more exploration. Full article

Environmental information plays an important role in deep reinforcement learning (DRL). However, many algorithms do not pay much attention to environmental information. In multi-agent reinforcement learning decision-making, because agents need to make decisions combined with the information of other agents in the environment, this makes the environmental information more important. To prove the importance of environmental information, we added environmental information to the algorithm. We evaluated many algorithms on a challenging set of StarCraft II micromanagement tasks. Compared with the original algorithm, the standard deviation (except for the VDN algorithm) was smaller than that of the original algorithm, which shows that our algorithm has better stability. The average score of our algorithm was higher than that of the original algorithm (except for VDN and COMA), which shows that our work significantly outperforms existing multi-agent RL methods. Full article