Microchip with ‘AI’ in the centre, connected to circuits and a blue hand that taps on it.

Artificial Intelligence & IT Security

Artificial intelligence is fundamentally transforming IT security on both sides. Attackers are using AI to amplify and refine their cyberattacks. At the same time, AI opens up new possibilities for early threat detection and prevention. Thus, companies face a twofold challenge: they must protect themselves against AI-powered attacks while also securing their own AI systems.

Attack tool Secure implementation FAQ

AI & Security: Understanding risks. Protecting systems. Leveraging innovation safely.

The speed and complexity of cyber threats are increasing noticeably. Attacks are becoming more varied, targeted, and difficult to detect. At the same time, the IT landscape is expanding due to the use of AI, increasing the demands on security and control. Many traditional security concepts are not designed to handle these dynamics.

The crucial question, therefore, is: How can you maintain an overview and use AI in a controlled and secure manner?

AI as an attack tool: When cyber attacks become smarter

Attackers are using artificial intelligence to launch attacks that are more targeted, faster, and scalable. AI automates many steps that used to be manual and time-consuming, from gathering information to executing complex attacks.

The result? Attacks are becoming more efficient, more credible, and significantly harder to detect.

AI-generated phishing

AI-generated phishing emails are deceptively real and highly personalized, free of the typical errors found in classic phishing emails. The content is tailored specifically to the recipients, making these attacks significantly harder to detect.

Deepfakes and identity theft

Voices, images, and videos can be faked to look incredibly realistic. Attackers can appear to be executives, colleagues, or business partners, taking social engineering to a whole new level.

Read the article

Automated attack campaigns

Attacks are no longer controlled manually but are dynamically generated and continuously adapted by AI. This significantly increases their speed, reach, and success rate.

Intelligent analysis of target systems

AI systematically scans available data sources, detects patterns, and identifies vulnerabilities. As a result, attacks are targeted, well-prepared, and highly impactful.

AI as a Vulnerability: When Intelligent Systems Themselves Become a Risk

The productive use of AI introduces new vulnerabilities that are often difficult to detect within modern IT architectures. Unlike traditional systems, AI models process, interpret, and generate new information from data, as well as derive decisions from it. This flexibility opens up new attack vectors. These risks arise along the entire AI pipeline, from training data and model behavior to integration into business processes.

The following overview highlights key attack points where AI systems are particularly vulnerable today:

Prompt injection

Manipulated inputs cause the model to ignore rules or perform unwanted actions.

The risk: Disclosure of internal information or misuse of functions.

Typical measures: Clear system boundaries, strict context separation, input/output filters

Disclosure of sensitive information

The model inadvertently outputs personal data, trade secrets, or training content.

The risk: Data protection and compliance risks.

Typical measures: Data sanitization, access restrictions, clear usage guidelines

Supply chain risks

Insecure or tampered third-party models, adapters, or datasets can compromise integrity.

The risk: Hidden backdoors and licensing issues.

Typical measures: Supplier audits, SBOMs, model signatures, version control

Data and model poisoning

Poisoned training or retrieval data can influence outputs or embed backdoors.

The risk: Distortion of knowledge and decisions.

Typical measures: Source verification, versioning, red teaming, anomaly detection

Insecure output handling

LLM outputs are passed to systems, such as browsers, databases, and shells, without being validated.

The risk: Enables XSS, SQL injection, or code execution.

Typical measures: Context-dependent output encoding, prepared statements, zero-trust.

Excessive agency

AI agents are permitted to perform too many actions without sufficient oversight.

The risk: Data loss and misuse of privileged actions.

Typical measures: Least privilege, reduced tool functionality, human-in-the-loop

(Source: OWASP Top 10 for LLMs)

AI as a shield: Rethinking security

However, the same technologies can effectively combat threats. Artificial intelligence enables the analysis of large amounts of data in real time, the identification of patterns, and the automatic response to incidents.

AI scales and strengthens modern security. With AI-powered security solutions, security teams can make faster, more informed decisions in scenarios such as the following:

Early threat detection

AI can identify potential security risks at an early stage, allowing for proactive countermeasures before damage occurs. AI continuously analyzes behavioral and telemetry data (e.g., UEBA) to detect deviations in user and system behavior in a context-based manner — not just rule-based, but adaptive and learning.

Automated analysis and prioritization

Security incidents are assessed in real time and prioritized based on their actual risk to the organization, directing focus toward critical events. AI combines current event data with threat and exploit intelligence (e.g., EPSS) to dynamically prioritize vulnerabilities and incidents based on their actual likelihood of exploitation.

Support for security teams

AI eases the burden on security teams by processing complex data sets and providing a solid foundation for decision-making in day-to-day operations.

Faster Incident Response

Thanks to automated workflows and clear action recommendations, security incidents can be contained and resolved much more quickly.

Governance & compliance: Safely managing AI

Without clear guidelines, AI can quickly pose a risk to businesses. Therefore, the safe use of AI requires binding governance structures, transparent processes, and consistent compliance with regulatory requirements. A crucial aspect of this is designing AI systems, data flows, and decisions in a way that is traceable and controllable during ongoing operations, both organizationally and technically.

Key areas of action

Establishing company-wide AI governance structures
Controlling data access and flows
Ensuring transparency and traceability of AI decisions
Integrating AI into existing compliance and security models
Defining clear operational and control mechanisms for AI systems (e.g., approvals, monitoring, auditing during ongoing operations)
Establishing technical safeguards and control points throughout the AI lifecycle (e.g., before, during, and after model use)

In addition, European guidelines define clear requirements for the secure and responsible use of AI and IT systems.

EU AI Act

Ban on high-risk AI applications

Prohibits systems that pose unacceptable risks, such as social scoring. Status: Effective as of February 2025

Obligations for General-Purpose AI (GPAI)

Establishes transparency, security, and documentation requirements for general-purpose AI models. Status: Effective since August 2025

NIS2 Directive

Strengthening IT security management and governance

Establishes cyber resilience and clear responsibilities at the executive level. Implementation: In force since December 6, 2025

Mandatory registration and reporting requirements

Affected companies must register with the BSI and report security incidents. Status: Effective as of March 6, 2026

Secure implementation: Getting AI right from the start

Whether AI systems create long-term value or generate new risks depends largely on how early security, governance, and control are integrated into their architecture and use. AI should be considered part of the broader IT and security landscape rather than an isolated technology.

What companies should do now

AI inventory & risk assessment

All AI systems in use, including unofficially approved applications (shadow AI), should be fully inventoried, classified, and assessed on a risk-based basis, ideally taking regulatory requirements into account.

Governance & policies

Define clear rules that govern the use of AI within the organization, especially when handling sensitive data, external models, and generative systems.

Technical security (security by design)

Secure AI systems with appropriate technical safeguards, such as access controls, data filters, secure authentication, and protection against misuse or manipulation.

Continuous monitoring & control

AI systems must be continuously monitored, both in terms of their inputs and outputs and with regard to unusual access patterns or unexpected system behavior.

Success Factors for Secure AI Implementation

Sustainable AI deployment requires an interplay of technology, processes, and organization.

Security by besign & default security: Security is integrated into the architecture and systems from the very beginning.
Identity & access management (IAM): Strict control over who is allowed to access which AI systems, data, and functions.
Data control & minimization: Only necessary data is processed. Sensitive information is protected or excluded.
Transparency & traceability: AI decisions and outputs must be explainable, verifiable, and documentable.
Integration into existing security structures: AI systems should be embedded in existing security, risk, and compliance frameworks rather than operating in isolation.
Operational monitoring & incident response: AI systems must be actively integrated into security processes, including alerting, responding, and escalating in the event of anomalies.

No secure AI without security — no modern security without AI

AI is fundamentally transforming IT security as a tool for attackers, as a vulnerability within organizations, and as a core technology for modern defense.

For companies, this means security can no longer be viewed in isolation; it must be an integral part of every AI strategy, from initial architectural decisions to production. A holistic approach that consistently integrates technology, governance, and operational security is crucial. Only in this way can risks be managed and the potential of AI be leveraged sustainably.

Find more articles

Stay informed about AI & security – in the it-sa 365 Community

Artificial intelligence is rapidly transforming cybersecurity. New attack methods, regulatory requirements, and innovative defense strategies are emerging almost daily. To stay on top of these developments, you need up-to-date knowledge and practical insights, as well as the opportunity to exchange ideas with experts.

Connect with IT security, data, and compliance professionals in the it-sa 365 Community.

As a member, you’ll benefit from:

Exchanges with IT security experts and AI specialists Aktuellen
Up-to-date insights on threats, technologies, and best practices
Personalized content on AI & cybersecurity
Direct access to relevant solution providers and use cases

Computer keyboard featuring a highlighted red key labeled “Community”

FAQ on AI & Security

While traditional IT security remains an important foundation, it no longer fully addresses AI-specific risks. AI systems introduce new attack surfaces, data flows, and dependencies that require specialized security and governance approaches.

In cyber defense, the key value of AI lies in the real-time analysis of large volumes of data. Systems in the SIEM and EDR/XDR domains use AI to correlate security events, detect anomalies, and reduce false positives.

This allows security teams to identify threats in near real time and respond more quickly to critical incidents, increasingly at a pace that matches automated attack patterns.

Responsibility for AI security is typically shared among IT, security, data, and compliance teams, rather than resting with a single role. Having a clearly defined governance structure that unambiguously outlines responsibilities and holds senior management accountable is crucial.

In cybersecurity, GenAI primarily accelerates analysis and decision-making processes. For instance, GenAI automates report creation, translates technical log data into actionable recommendations, and assists development teams in building secure applications (Security-as-Code).

In the field of security testing (e.g., red teaming), GenAI is also used to simulate realistic attack scenarios such as phishing or exploit code. This allows for the specific testing of systems' defensive capabilities.

Securing generative AI requires specialized security approaches and new standards. Established frameworks include the OWASP Top 10 for Large Language Models and the NIST AI Risk Management Framework.

Technically, tools such as API gateways with AI-specific security policies, prompt injection detection mechanisms, and adversarial testing tools are used to verify the robustness and integrity of models.

The challenges lie at both the technical and organizational levels. Key factors include the shortage of professionals with combined AI and security expertise, the quality and availability of training data, and the growing complexity of hybrid IT and AI architectures. Added to this are regulatory requirements, particularly in the context of the EU AI Act, as well as unresolved liability and accountability issues surrounding automated decisions.

Generative AI introduces new security risks, especially when handling sensitive data. Critical issues include data leakage via prompts, prompt injection, and unintentionally disclosing confidential information.

Therefore, securing these systems requires a comprehensive architectural concept featuring private or sovereign cloud approaches, consistent identity and access management (zero trust), and continuous monitoring of all AI interfaces.

Shadow AI occurs when employees use AI tools without central approval. Managing this requires a combination of technical safeguards, clear guidelines, and the provision of secure, approved AI solutions as an alternative.

It describes a targeted attack on the training data or dataset of an AI model. In this attack, manipulated data is injected to influence the model’s behavior. For example, certain detections may be suppressed, or false results may be triggered. This poses a significant threat to the reliability and integrity of AI-based security systems.

Read the article:Supply Chain Attacks: Poisoned AI

AI is becoming a central component of modern cybersecurity architectures. There is a clear trend moving toward autonomous systems in which AI agents detect and assess threats and, in some cases, independently initiate countermeasures. This shifts the focus from reactive incident response to proactive, continuous resilience and automated defense.

Protecting yourself from deepfakes requires a combination of technical validation mechanisms, verified communication channels, and targeted awareness measures that go beyond traditional training.

Zero Trust is a core principle for securing AI systems. All access to data, models, and AI functions must be continuously verified, regardless of the environment from which it originates.

Yes. Depending on their application, AI systems can be classified as critical or business-critical IT resources. Therefore, they are subject to the NIS2 Directive's requirements, particularly those related to security measures, reporting obligations, and management-level responsibilities.