Coordinating Drag-Based Structure Editing and Reference Style Transfer in Diffusion Models for Anime Images

Reference guided anime editing is challenging when the target requires both rendering style transfer and local structural change. Existing diffusion stylization methods that do not require training usually assume a fixed content layout, while drag-based editors deform local structures without enforcing a separate style reference. Directly combining them is unstable: reference attention can disrupt handle tracking during dragging, whereas stylization after dragging can weaken the edited structure. This paper proposes AnchorHandoff, a temporally coordinated diffusion framework for joint drag and style editing. Drag optimization is performed with style injection disabled, followed by a short interval without style injection that lets the edited structure stabilize. A predicted clean sample from this state after dragging is then used as an anchor: content queries are refreshed from the anchor, and reference style keys and values are replayed on the edited layout. Soft correspondences from intermediate attention features guide style injection toward compatible regions without parsers or segmentation labels. On a curated anime benchmark, controlled comparisons, ablations, and a blind study with 36 participants show that AnchorHandoff reduces residual tracking error and feature structure distortion while maintaining comparable distribution level style alignment. The method remains limited under very large structural changes, but the results highlight temporal handoff as an important factor in joint anime structure and style editing.