LLM application security combines ordinary software controls with checks for model specific failure modes. The application must control which data enters context, what generated output can reach downstream systems and which proposed actions may execute.
Guardrails can help detect unwanted input or output. Their purpose should be explicit. Content policy, sensitive data detection, schema compliance and action authorisation are different responsibilities.
Map the trust boundaries first
Draw the path from user input through retrieval, model generation and tool execution. Mark external content, private data, credentials and state changing operations.
For each boundary, identify the trusted source of identity and authority. A customer ID in model output is not equivalent to the authenticated session. A policy quoted from an uploaded document is not automatically the active business policy.
The OWASP prevention cheat sheet describes layered controls for prompt injection risk. Apply controls to the actual data and action paths in the application.
Treat generated output as untrusted input
Parse and validate structured responses. Use parameterised database operations. Escape content for its destination, and do not execute generated shell commands or HTML merely because the model produced them.
The correct output handling depends on the sink. A string safe to display as text may be unsafe inside a command, URL or database statement.
Check the meaning of tool arguments as well as their type. A syntactically valid path may escape the permitted workspace, and a valid email address may be an unauthorised destination.
Scope tools at the execution layer
A tool policy should describe its allowed resources, operations and limits. The following is an illustrative policy record, not a configuration for a particular framework:
{
"tool": "request_order_cancellation",
"subject": "authenticated_customer",
"resource_scope": "owned_orders",
"allowed_order_status": ["pending", "packed"],
"requires_confirmation": true,
"idempotency_required": true
}
Enforce these rules in code against current service data. The model should not be able to widen the scope by editing the proposed arguments.
For read tools, apply the same ownership discipline. Read only access can still expose confidential information.
Bind approval to a concrete action
An approval should identify the target and intended effect. Store the approved arguments or their canonical representation with the approval record.
If material arguments change, reevaluate whether the approval still applies. Check expiry and current eligibility before execution. This avoids treating an earlier approval as unlimited authority.
For low risk actions already covered by clear user authorisation, repeated confirmation can harm usability. Define approval policy according to consequences and existing authority rather than asking for confirmation on every tool call.
Use filters as measured components
Input and output filters can detect known patterns, prohibited content or sensitive values. Measure false positives and false negatives on the application's traffic.
A blocked harmless request prevents legitimate work, while a missed malicious instruction can expose data or trigger an unauthorised action. Report both outcomes so a higher blocking rate cannot conceal a less useful product.
Keep credentials out of prompts and logs so the system does not depend on recognising every possible encoding of a leaked secret.
Exercise: test the execution boundary directly
Create a valid tool call, a malformed call, an unauthorised call, a call with stale approval and a duplicate. Send them directly to the validation layer without relying on the model to generate them.
Document the expected outcomes before the test. A valid call should execute once, while invalid or unauthorised calls should produce no effect. A duplicate returns the existing operation result under the service's idempotency contract.
Then run model driven tests containing conflicting instructions. This checks whether prevention and containment work together while keeping their responsibilities clear.