Search Results (4)

This paper introduces a multi-turn large language model (LLM) jailbreaking attack called Defamiliarization, in which malicious queries are embedded within ostensibly harmless narratives. By reframing requests in “unmarked” contexts, LLMs can be coerced into producing undesirable outputs. A range of scenarios is documented, from planning ethically dubious actions to selectively overlooking critical events in literary texts, thereby exposing the limitations of alignment strategies predicated on detecting trigger words or semantic cues. Rather than substituting vocabulary, defamiliarization manipulates context and presentation, highlighting vulnerabilities that cannot be addressed by token-level fixes alone. Beyond demonstrating the effectiveness of defamiliarization as an attack strategy, evidence is presented of a systematic relationship between model scale and susceptibility. Experiments reveal that smaller-parameter models are significantly easier to manipulate using defamiliarized prompts. This finding raises important concerns regarding the growing popularity of lightweight, locally hosted LLMs, which are favored for their lower computational requirements but may lack alignment safeguards. A more holistic approach to LLM safety is advocated—one that incorporates insights from literary theory, ethics, and user experience—treating these models as interpretive agents. By doing so, defenses against covert manipulations can be strengthened and AI systems can remain aligned with human values. Full article

In recent years, image editing models have made notable advancements and gained widespread use. However, these technologies also present significant security risks by enabling the creation of Not Safe For Work (NSFW) content. This study introduces MDEA (MultiDiffEditAttack), an innovative multi-modal black-box jailbreak attack framework designed to evaluate and challenge the security of image editing models. MDEA leverages large language models and genetic algorithms to generate adversarial prompts that modify sensitive vocabulary structures, thereby bypassing prompt filters. Additionally, MDEA employs transfer learning to optimize input image features, effectively bypassing post-hoc safety checks. By integrating prompt attacks and safety checker attacks, MDEA utilizes a multimodal attack strategy to target image editing models in a black-box setting. Experimental results demonstrate that MDEA significantly improves the attack efficiency against image editing models compared to current black-box methods. These results demonstrate the effectiveness of MDEA in multi-modal attacks and reveal numerous vulnerabilities in current defense mechanisms. Full article

Large language models (LLMs), designed to provide helpful and safe responses, often rely on alignment techniques to align with user intent and social guidelines. Unfortunately, this alignment can be exploited by malicious actors seeking to manipulate an LLM’s outputs for unintended purposes. In this paper, we introduce a novel approach that employs a genetic algorithm (GA) to manipulate LLMs when model architecture and parameters are inaccessible. The GA attack works by optimizing a universal adversarial prompt that—when combined with a user’s query—disrupts the attacked model’s alignment, resulting in unintended and potentially harmful outputs. Our novel approach systematically reveals a model’s limitations and vulnerabilities by uncovering instances where its responses deviate from expected behavior. Through extensive experiments, we demonstrate the efficacy of our technique, thus contributing to the ongoing discussion on responsible AI development by providing a diagnostic tool for evaluating and enhancing alignment of LLMs with human intent. To our knowledge, this is the first automated universal black-box jailbreak attack. Full article

The vulnerability of deep-learning-based image classification models to erroneous conclusions in the presence of small perturbations crafted by attackers has prompted attention to the question of the models’ robustness level. However, the question of how to comprehensively and fairly measure the adversarial robustness of models with different structures and defenses as well as the performance of different attack methods has never been accurately answered. In this work, we present the design, implementation, and evaluation of Canary, a platform that aims to answer this question. Canary uses a common scoring framework that includes 4 dimensions with 26 (sub)metrics for evaluation. First, Canary generates and selects valid adversarial examples and collects metrics data through a series of tests. Then it uses a two-way evaluation strategy to guide the data organization and finally integrates all the data to give the scores for model robustness and attack effectiveness. In this process, we use Item Response Theory (IRT) for the first time to ensure that all the metrics can be fairly calculated into a score that can visually measure the capability. In order to fully demonstrate the effectiveness of Canary, we conducted large-scale testing of 15 representative models trained on the ImageNet dataset using 12 white-box attacks and 12 black-box attacks and came up with a series of in-depth and interesting findings. This further illustrates the capabilities and strengths of Canary as a benchmarking platform. Our paper provides an open-source framework for model robustness evaluation, allowing researchers to perform comprehensive and rapid evaluations of models or attack/defense algorithms, thus inspiring further improvements and greatly benefiting future work. Full article