# Securing AI Systems ###### **Introduction** David Sully, CEO of Advai, explores the critical need for Securing AI Systems. David dives into the highs and lows of AI, unpacking the reasons behind both its potential and its flaws. He unveils the security threats posed by AI, while also highlighting its ability to identify patterns that could be ignored by humans. David tackles the crucial question: How can we ensure the safe and responsible development of AI? Join this thought-provoking discussion on securing the future of AI! David Sully talking about Securing AI Systems ###### **Unmasking AI: A Deep Dive into Adversarial Challenges with Advai** AI has long been heralded as a revolutionary force, with smooth, seamless performances that captivate audiences and promise boundless potential. Yet, beneath the surface lies a world of vulnerabilities and challenges, as revealed in an insightful talk by the CEO of Advai during one of CyNam’s Headline event back in 2024. Here’s a glimpse into how Advai is breaking new ground by exposing the weaknesses in AI systems and ensuring their robustness against future adversaries. ###### **Advai: The AI Adversary Specialists** Founded three years ago, Advai challenges the traditional perception of AI-focused companies. While many celebrate the wonders of AI innovation, Advai positions itself as the necessary skeptic—the entity that breaks AI systems to uncover their flaws. As the CEO puts it, they are “that really annoying person in the corner, basically going, ‘Yeah, but…’” This philosophy stems from a fundamental understanding of AI’s imperfections. Highlighting the now-famous “dancing robots” videos, the CEO pointed out how even controlled environments—with optimised lighting, carefully placed obstacles, and countless takes—still result in AI failure. This, they argue, is emblematic of AI’s broader challenges: it may excel in controlled scenarios, but its real-world application often reveals significant shortcomings. Atlas \| Partners in Parkour ###### **AI’s Achilles Heel: Adversarial Attacks** Adversarial AI—the ability to manipulate or deceive AI systems—is at the core of Advai’s work. From altering pixels in an image to tricking a computer vision model into misidentifying objects, to exploiting vulnerabilities in language models to generate inappropriate outputs, adversarial attacks expose the fragility of these systems. One striking example shared was the infamous “stop sign attack,” where stickers placed on a stop sign caused an autonomous vehicle to perceive it as a 40mph sign. The human eye dismissed the stickers as graffiti, but the AI interpreted them as a directive to accelerate—a stark reminder of the risks posed by adversarial manipulation. ###### **The Broader Implications of AI Vulnerabilities** While adversarial AI makes for captivating demonstrations, its implications are far-reaching. AI systems are no longer confined to niche applications; they are increasingly embedded in critical decision-making processes across industries. Yet, organisations often fail to address fundamental governance, risk, and compliance (GRC) requirements. For instance, large enterprises relying on AI-powered tools like ChatGPT might struggle to audit or justify AI-driven decisions years later due to the lack of version control or traceability. The CEO’s stark warning: “Imagine the regulator walking in three years from now and asking how a decision was made. If you can’t even identify the version of the AI system used, how can you justify its actions?” ###### **AI’s Evolving Threat Landscape** The integration of AI systems introduces unique security challenges. Automated systems are now both the target and the weapon in cyberattacks. From deepfakes and misinformation campaigns targeting humans to automated adversarial attacks on AI systems, organisations are ill-equipped to address these multi-faceted threats. Compounding the issue is the lack of a standardised approach to AI development and deployment. Unlike software engineering, which benefits from test-driven development and rigorous standards, AI lacks comparable frameworks, leaving developers to grapple with “black box” systems that are difficult to explain, test, or defend. ###### **Towards a Robust AI Future** Advai’s mission is not merely to expose AI’s flaws but to drive the industry toward better practices. The CEO emphasised the need for a structured approach to AI deployment, starting with: - **Defining the Business Use Case:** Understanding the intended purpose and limitations of the AI system. - **Implementing GRC Measures:** Establishing clear governance, risk management, and compliance frameworks. - **Developing Robust Testing Protocols:** Designing tests that evaluate the AI system’s performance, bias, and ethical considerations. - **Ensuring Continuous Monitoring:** Maintaining vigilance over the AI’s outputs and interactions to catch issues early. As an example, Advai showcased facial verification systems, highlighting how even well-constrained AI applications can reveal biases and inconsistencies under rigorous testing. By formalizing testing and evaluation processes, organisations can uncover hidden weaknesses and build more reliable systems. ##### **The Call to Action** AI’s promise is immense, but so are its challenges. As adversarial threats grow and regulatory scrutiny increases, the need for a disciplined, methodical approach to AI development has never been more critical. Advai’s work serves as a reminder that the path to AI excellence lies not in ignoring its flaws but in confronting them head-on. As the CEO concluded, “The AI industry needs to shift from chasing perfection to embracing accountability, transparency, and resilience. Only then can we truly unlock its potential.” This article is based on a presentation delivered by David Sully at CyNam 24.2. The full session was recorded and is available to watch here: [Securing AI Systems](/content/resources/cynam-24-1-david-sully-securing-ai-systems/index.html). You can also find it on our resources page. [Advai](https://www.advai.co.uk/company/) works at the frontier of Artificial Intelligence. **We discover where AI will fail – and why**. We enable businesses to deploy Robust, Fair and Responsible AI.