Abstract
Network intrusion detection is a significant component of cybersecurity, essential for safeguarding digital infrastructures against both internal and external threats. While conventional Intrusion Detection Systems (IDS) perform well against known attack signatures, they fall short when confronted with previously unseen or zero-day threats – exposing a critical vulnerability in real-world deployment. To address this, we propose a generative multiagent framework designed specifically for Open-World Intrusion Detection and responding to threats based on their significance and their targets. This framework integrates modified generative transformers into a multi-agent architecture, enabling the system to detect both known and novel intrusions in dynamic environments. This approach represents a shift from static signature-based detection to a more adaptive, open-set recognition paradigm capable of proactively identifying emerging threats. This dissertation’s five key contributions are: (i) a comprehensive evaluation of current deep learning-based Open Set Recognition(OSR) models in the context of network intrusion, revealing critical performance gaps; (ii)the development of varMax, a robust open-set classifier tailored to high-dimensional network data; (iii) the introduction of UPacketLabel, an enhanced transformer-based IDS equipped with LLM-driven explainability for interpreting unknown threats; (iv) implementation of PacketGuard, adversarial defense mechanisms and test-time resilience techniques; and (v)the integration of an Adversarial Risk Assessment (ARA) framework for quantifying OSRmodel risk under adversarial attack scenarios. Experimental results show that our approach is a forward-looking solution capable of securing networks in adversarial and evolving threat landscapes.