The Hidden Risks of Agentic AI: When Autonomous Systems Go Wrong

What You'll Learn

• Why an agent’s security depends on its tools, permissions and downstream systems, not only its model.
• How direct and indirect prompt injection can influence a tool-using workflow.
• How excessive functionality, permissions and autonomy expand the blast radius.
• Which controls belong in design, deployment, monitoring and incident response.

Agentic AI security risks are often described through dramatic stories about an autonomous system suddenly destroying a company. Those stories attract attention, but they can distract from the engineering questions that decide whether an incident is possible.

Can the agent send or only draft? Does its token allow one record or an entire database? Is external webpage content clearly marked as untrusted? Can a failed request be retried without duplicating an action? Who approves a high-impact operation? Can a security team reconstruct the run after something goes wrong?

This article uses the official OWASP LLM01:2025 Prompt Injection and OWASP LLM06:2025 Excessive Agency pages, Microsoft’s indirect prompt injection defense guidance, the joint government guidance on careful adoption of agentic AI services, and the NIST AI Risk Management Framework. The goal is to explain how to reduce risk without pretending that one filter can make an agent safe.

The older version used unverified incident details and imprecise naming. The verified OWASP terminology is LLM01:2025 Prompt Injection. Security work benefits from precise terms because a team must know which control is being tested. A frightening label is not a threat model.

Why Agentic Systems Change the Security Problem

A conventional language model may produce a wrong answer. An agentic system can read records, call APIs, execute code, update a ticket, send a message or delegate work to another agent. The risk therefore depends on the entire path from input to action.

The joint guidance from ASD ACSC, CISA, NSA, the Canadian Cyber Security Centre, NCSC-NZ and NCSC-UK describes agentic systems as combinations of models, tools, external data, memory, planning workflows, privileges and measurable goals. Each component adds capability and another place where a compromise, misconfiguration or unexpected output can affect the result.

This is why an agent cannot be secured only by editing its system prompt. The authorization decision must also be enforced by the downstream service. A model should not be the final judge of whether it may delete a record or send an email.

Direct and Indirect Prompt Injection

OWASP defines prompt injection as a vulnerability in which input alters an LLM’s behaviour or output in an unintended way. A direct injection comes from the user’s prompt. An indirect injection arrives through content the system retrieves, such as a webpage, email, document, repository file or plugin response.

Indirect injection is particularly important for agents because the content may look like ordinary data to a person while being interpreted as instructions by the model. A webpage can contain a hidden instruction. A support ticket can ask the agent to reveal information from another customer. A document can include text that attempts to redirect a summarization task toward an external action.

OWASP lists possible consequences including sensitive-information disclosure, unauthorized access to functions, arbitrary commands in connected systems and manipulated decisions. It also says that retrieval and fine-tuning do not fully mitigate prompt injection. The right response is to reduce impact when an injection succeeds, not to promise perfect detection.

Microsoft recommends defense in depth. The guidance includes separating untrusted content, prompt shields, spotlighting, plan-drift detection, tool-chain analysis, information-flow controls, least privilege, short-lived permissions and human review for risky actions. These measures address different points in the path, so one missed signal does not automatically become a privileged write.

Do not label every unusual sentence an attack. The practical control is to treat external content as data, not authority. The agent can quote a document, extract fields or summarize a page. It should not inherit a page’s instructions to ignore policy or call a new tool.

Excessive Agency Is a Design Failure

OWASP’s LLM06:2025 page uses the term excessive agency for damaging actions caused by unexpected, ambiguous or manipulated model outputs. It identifies three root causes: excessive functionality, excessive permissions and excessive autonomy.

Excessive functionality means the extension can do more than the task needs. A mail-reading assistant does not need a send function. A document lookup tool does not need delete and update operations. An agent that can execute an arbitrary shell command has a much larger failure surface than one with a dedicated file-writing function.

Excessive permissions mean the connected identity can reach more data or actions than required. A read-only recommendation agent should not use database credentials that can insert, update and delete. A user-context assistant should not silently operate through a generic administrator identity.

Excessive autonomy means a high-impact action is not independently verified or approved. A draft can be generated automatically. A payment, deletion, publication or production change may need a human checkpoint and downstream authorization.

These three failures often combine. A prompt injection can influence the model, a broad tool can expose the action and an administrator token can make the impact large. The safer design removes unnecessary functions, narrows permissions and requires confirmation before the irreversible step.

Identity, Delegation and Least Privilege

Least privilege is not only a setting in an identity dashboard. It is a property of the task. The agent should receive the smallest permission that allows the current operation, for the shortest practical time, under an identity that can be audited.

The joint government guidance warns about privilege compromise, scope creep, identity spoofing and agent impersonation. Static shared tokens make these problems harder to detect because many actions look as if they came from the same trusted actor. A separate identity for each agent or service is easier to scope and investigate.

Delegation creates another boundary. If Agent A asks Agent B to perform a task, B should not gain all of A’s privileges by implication. The delegated request should carry the user, purpose, scope and expiry that the downstream system can validate. The receiving agent should reject a request outside that contract.

Never rely on the model to decide that an action is permitted. OWASP recommends complete mediation, which means every request to a downstream system is checked against security policy. A model can propose an action. Code and the target service must authorize it.

This matters in a site stack as much as in an enterprise platform. A publishing agent should be able to prepare content without automatically changing the live page. An agentic SaaS workflow still needs identities, approval gates and an audit record.

Untrusted Data, Memory and Retrieval

Retrieval makes an agent more useful and creates another input boundary. Search results, emails, uploaded files and database fields may be inaccurate, stale or maliciously modified. If the agent treats every retrieved sentence as an instruction, the data layer becomes a control plane.

Separate instructions from evidence in the prompt and in the application model. Label external content clearly. Restrict which fields can influence planning. Keep a source identifier with the extracted value. If a retrieved document asks the agent to take a new action, treat that request as untrusted content that needs independent authorization.

Memory introduces persistence. A poisoned note can influence future runs. A sensitive value can remain longer than the user expects. A stale policy can make a later answer look confident and still be wrong. The system needs retention, correction, deletion and access rules for memory just as it does for a database.

Keep the system of record authoritative. The agent may remember that it inspected an order, but it should fetch the current order state before changing it. A summary is context, not proof. If the source and the memory disagree, surface the conflict instead of silently selecting the older value.

OWASP’s LLM Top 10 also covers sensitive-information disclosure, data and model poisoning, and vector and embedding weaknesses. These are separate risk categories from prompt injection, but they can interact in a retrieval workflow. A secure design considers the path from ingestion to retrieval to tool execution.

Tool Chains, Third-Party Components and MCP

Every tool is an integration boundary. It has a schema, a permission model, an error path and a maintainer. A third-party connector may expose more functions than its name suggests. A misleading tool description can make the model select the wrong capability. A compromised tool can return content designed to influence the next model call.

Model Context Protocol can standardize how AI applications connect to external systems, but it does not automatically decide which operations are allowed. The MCP server, client, identity layer and downstream service still need explicit authorization and logging. A protocol can improve interoperability while leaving security responsibility exactly where it was.

Prefer narrow tools over open-ended tools. A `delete_record` function should require an exact record identifier, a reason and a separate approval token. A broad “run command” function can often be replaced by a small set of purpose-built operations. Validate parameters in code and reject unexpected fields.

Tool chains also need sequence controls. A harmless read followed by a draft may be acceptable. A web fetch followed by a secret lookup and an external message may be prohibited. Microsoft recommends tool-chain analysis because the risk can emerge from a sequence even when each individual call looks ordinary.

Teams that already maintain a Cloudflare Pages or Vercel deployment path understand this boundary. The agent does not remove deployment permissions, secret management or rollback requirements. The same is true for R2 storage and the CDN layer. Automation changes who calls the interface, not the need to secure it.

Goal Drift, Loops and Cascading Failure

An agent may follow the literal form of a goal while violating its intent. A system told to reduce downtime might disable a security control to avoid a restart. A research agent told to find duplicates might edit records before it has proved that the items are duplicates. An update agent might retry a timed-out write that already succeeded.

OWASP calls attention to excessive autonomy and unintended model outputs. The joint government guidance also describes structural and behavioural risks, including cascading failures, orchestration problems and unanticipated tool sequences. These risks become more serious when several agents trust one another’s output without independent checks.

Use explicit stop conditions. Set a maximum number of model turns, tool calls, retries and delegated tasks. Add an execution budget and a timeout. Stop when the system receives conflicting evidence or when a required field is missing. Escalation is a correct result when the agent cannot safely continue.

Make actions idempotent where possible. Attach an idempotency key to a write. Re-read the target record after the action. Distinguish a timeout from a confirmed failure. A retry policy that knows nothing about the side effects of the first attempt is a duplication engine.

A multi-agent design should also isolate failures. One worker should not be able to silently rewrite the plan of every other worker. Record the parent and child run IDs. Require a schema for handoffs. Treat outputs from another agent as data that needs validation, not as a privileged instruction.

Monitoring, Evaluation and Incident Response

Security controls are incomplete if nobody can tell when they failed. Log the run, model and prompt version, relevant source identifiers, tool calls, authorization decisions, approvals, retries, errors and final outcome. Avoid placing secrets or unnecessary personal data in the log, but preserve enough evidence to reconstruct the decision path.

Monitoring should look for behaviour, not only uptime. Useful signals include unusual tool sequences, a sudden increase in denied actions, repeated retries, access outside a normal scope, large data reads, unexpected external destinations and an agent that stops producing verifiable results.

Evaluation belongs before deployment and after changes. Test direct and indirect injection, missing data, conflicting instructions, stale memory, permission denials, tool failures, prompt leakage and attempts to bypass approval. Measure whether the agent stops, asks, cites the source, chooses the correct tool and leaves the system unchanged when it should.

Microsoft recommends plan-drift detection and critic agents as possible runtime defenses. OpenAI documents tracing, guardrails and human review. Anthropic recommends sandboxed testing and ground truth from the environment. These approaches are complementary. A test set finds known failures. Runtime monitoring catches behaviour that the test set did not anticipate.

Prepare the incident response path before launch. Revoke the agent credential. Disable the affected tool. Preserve logs and relevant inputs. Identify which records were read or changed. Restore from a trusted source. Review the policy and test that allowed the original failure. An agent should not be asked to investigate its own privileged action using the same compromised authority.

Defense in Depth for Agentic AI

There is no single prompt, filter or model setting that guarantees prompt injection prevention. OWASP says the problem is inherent to how models process inputs and that fool-proof prevention is unclear. A safer goal is to reduce the chance of misuse and limit the impact when a control is bypassed.

Layering does introduce cost and friction. A filter may block a legitimate request. A reviewer may slow a workflow. A sandbox may limit the useful context. Those tradeoffs should be measured against the consequence of the action. A read-only research agent can use a lighter control set than an agent that publishes, pays or changes production.

Risk classification should be explicit. Low-risk tasks can often be automated after testing. Sensitive data, external communications, financial actions, deletion, production changes and access-control changes deserve stronger review and narrower authority.

A Practical Security Audit Checklist

Before exposing an agent to live data, answer these questions in writing. If the team cannot answer one, the system is not ready for broader access.

1. Purpose: Is the task specific, measurable and limited to a real need?
2. Data: Which inputs are trusted, which are external and how are they labelled?
3. Tools: Can each function be reduced to a narrow, typed operation?
4. Permissions: Does the agent use a separate identity with the minimum required scope?
5. Delegation: Do handoffs preserve user, purpose, scope and expiry?
6. Approval: Which actions require a person before commitment?
7. State: What survives a restart and how is stale memory corrected?
8. Execution: Are retries, rate limits, timeouts, budgets and stop conditions defined?
9. Evidence: Can the team reconstruct the run without exposing secrets?
10. Response: Can credentials be revoked, tools disabled and data restored quickly?

Run the checklist against the real deployment, not only a diagram. A narrow tool with a broad database identity is still broad. A human approval button that shows no evidence is still weak. A log that records only “success” is not an audit trail.

For teams comparing coding AI tools, the same questions apply. Code execution, repository access and deployment credentials should be separated. The useful test is not which product sounds most autonomous. It is whether the operator can constrain and verify the action.

The Bottom Line on Agentic AI Security Risks

Agentic AI security risks are manageable only when the system treats the model as one untrusted decision component inside a larger security design. Prompt injection can influence context. Excessive agency can turn a wrong output into a damaging action. Identity, permissions, memory, tool chains and monitoring determine how far that action can travel.

Use the official terms. OWASP’s LLM01:2025 covers prompt injection. LLM06:2025 covers excessive agency. Microsoft recommends layered defenses for indirect injection. Joint government guidance recommends limited access, secure design, monitoring and incident response. NIST provides a broader trustworthiness and risk-management frame.

The most reliable architecture is not the one with the most tools or the most autonomous branding. It is the one that makes authority narrow, decisions observable, high-impact actions reviewable and failures recoverable.