Abstract
Visual malware recognition classifies byte-plot images of executables and reports near-perfect accuracy, yet it is rarely evaluated the way a deployed detector would be. We present ByteScope, a framework that adds a cosine-margin family classifier, a supervised contrastive embedding for rejecting unknown families, and a class-balanced malware detection head calibrated at a fixed false-positive rate to an image backbone, together with a leakage-audited, open-world evaluation protocol, and we evaluate it on MaleVis against nine convolutional neural network (CNN), Vision Transformer and hybrid backbones with three seeds. A perceptual-hash audit, confirmed by pixel-level, embedding-based and visual inspection, shows that 50.6% of the test images and 70.0% of the malware have a same-class near-duplicate in the training split. On Swin-T, ByteScope reaches the highest mean 26-class accuracy (89.3%) and de-duplicated accuracy (78.4%), matches the best baseline in unknown-family detection (area under the receiver operating characteristic curve, AUROC, 0.925 against 0.922), and halves the evasion rate of its backbone under image-domain benign padding at a validation-calibrated threshold (15.2% against 36.0%) by making the operating point more stable, an effect that we trace to the normalized cosine classifier; at matched test false-positive rates, robustness is similar. At a 1% false-positive rate, neither ByteScope nor any of the nine baselines detects more than 3% of malware on average, whereas several variants of our hybrid CNN–Transformer architecture exceed this level (6.8 to 48.1%, with a large variance across seeds), and roughly 52 benign validation images cannot certify any rate below 5.6%. We release the code, the near-duplicate list and the splits.