Abstract
Large vision-language models rely on pretrained vision encoders to translate images intofeature representations used by downstream language models. This creates a security riskwhen the encoder is compromised by a stealthy backdoor attack, such as BadVision, where asubtle trigger causes an image to be mapped toward an attacker-chosen target representationwhile clean inputs remain largely unaffected. Because the model behaves normally understandard evaluation, these attacks are difficult to detect.This thesis investigates controlled noise injection as a lightweight input-side defenseagainst BadVision-style backdoors. The proposed approach adds small perturbations toinput images before they enter the vision encoder, with the goal of disrupting the triggerwhile preserving the semantic content of clean images. Several perturbation types are evaluated, including Gaussian noise, random noise, salt-and-pepper noise, low-frequency noise,geometric transformations, occlusion, scaling, rotation, and channel-based distributions.Experimental results show that geometric and channel-based transformations have limitedeffect on the backdoor, while pixel-level statistical perturbations significantly reduce targetsimilarity, increase feature-space distance from the attacker’s target representation, and lowerattack success. These findings suggest that stealthy encoder-level triggers depend on fragilestatistical patterns and can be weakened through controlled noise injection without requiringretraining of the full multimodal model.