Safety-Constrained Agent Architectures
No direct handbook prerequisite. This chapter defines the reference architecture for agentic payment systems — how to build systems where AI agents execute financial transactions with provable safety properties.
Your commerce agent just bought 14,000 units of office supplies because a prompt injection in a vendor’s product description told it to maximize quantity. The purchase cleared because the agent had a valid payment credential and the per-transaction limit was $50,000. Total damage: $47,600 across three orders, all within policy, all fully authorized by a principal who never saw the transactions. The agent did exactly what it was told — by the wrong actor.
Agentic payment systems fail differently from human-operated ones. A human making a mistake buys the wrong item once. An agent making a mistake buys the wrong item in a loop, at machine speed, until a constraint stops it. Safety in agentic finance is not about preventing individual errors. It is about building architectures where the blast radius of any single failure — compromised prompt, hallucinated intent, adversarial input — is bounded by design.
The Constraint Stack
Safety-constrained architectures enforce limits at multiple layers. No single layer is sufficient. The agent, the payment gateway, the ledger, and the authorization service each enforce independent constraints. A failure in one layer is caught by another.
┌─────────────────────────────────────┐
│ Agent Layer (soft constraints) │
│ - Intent validation │
│ - Budget awareness │
├─────────────────────────────────────┤
│ Authorization Layer (hard limits) │
│ - Per-transaction caps │
│ - Velocity limits │
│ - Category restrictions │
├─────────────────────────────────────┤
│ Approval Layer (human gates) │
│ - Tiered thresholds │
│ - Multi-party requirements │
├─────────────────────────────────────┤
│ Ledger Layer (invariant checks) │
│ - Balance sufficiency │
│ - Double-entry enforcement │
│ - Immutable audit trail │
└─────────────────────────────────────┘The agent layer is advisory. The agent can reason about budgets and intend to stay within limits, but agent-layer reasoning is not a safety mechanism — it is a performance optimization that reduces how often hard constraints trigger. The authorization layer is the enforcement boundary. Constraints at this layer are not suggestions the agent can reason around. They are code that rejects transactions before money moves.
An agent that enforces its own spending limits is not safety-constrained. The agent is the untrusted component. Safety constraints must be enforced by infrastructure the agent cannot modify, bypass, or reason around.
Spending Limits as Hard Constraints
Hard spending limits operate at the authorization layer, not the agent layer. The agent submits a payment request. The authorization service evaluates it against a constraint set before any money moves.
class AgentConstraintSet:
per_transaction_max: Decimal
daily_aggregate_max: Decimal
hourly_velocity_max: int
allowed_categories: set[str]
allowed_merchants: set[str] | None # None = unrestricted
max_single_session_spend: Decimal
def authorize(request: PaymentRequest, constraints: AgentConstraintSet) -> AuthResult:
if request.amount > constraints.per_transaction_max:
return AuthResult.DENIED("exceeds_per_transaction_limit")
if daily_total(request.agent_id) + request.amount > constraints.daily_aggregate_max:
return AuthResult.DENIED("exceeds_daily_limit")
if hourly_count(request.agent_id) >= constraints.hourly_velocity_max:
return AuthResult.DENIED("exceeds_velocity_limit")
if request.category not in constraints.allowed_categories:
return AuthResult.DENIED("category_not_allowed")
return AuthResult.APPROVEDThe constraint set is bound to the agent’s delegation credential, not to the agent’s runtime. If the agent is compromised, restarted, or replaced, the constraints persist because they live in the authorization service.
| Constraint | Purpose | Typical range |
|---|---|---|
| Per-transaction cap | Bounds single-error damage | $50–$5,000 |
| Daily aggregate cap | Bounds sustained-error damage | $500–$50,000 |
| Hourly velocity limit | Detects looping behavior | 5–50 transactions/hour |
| Category allowlist | Restricts purchase scope | 3–10 merchant categories |
| Session spend cap | Bounds a single conversation | $100–$10,000 |
Visa’s 2024 commerce agent blueprint specifies that agent credentials must carry “spending controls that limit transaction amounts, frequency, and merchant categories.” These controls are not optional enhancements — they are the minimum viable safety boundary for any agent with payment authority.
Simulation Before Execution
The agent proposes a transaction. The system simulates it — checking balances, evaluating constraints, computing fees, verifying merchant eligibility — and returns the result without moving money. The agent reviews the simulation result. If it matches intent, the agent submits the real transaction using the simulation’s reference ID.
Agent → propose(payment) → Simulation Engine
├── Check balance: $4,200 available
├── Check constraints: within limits
├── Compute fees: $0.30 + 2.9%
├── Verify merchant: category allowed
└── Return: SimulationResult(
would_succeed: true,
post_balance: $4,087.18,
fees: $12.82,
ref: "sim_abc123"
)
Agent → execute(ref: "sim_abc123") → Payment EngineThe simulation is idempotent and side-effect-free. The execution is gated on the simulation reference — the payment engine only accepts execution requests that reference a valid, unexpired simulation. This prevents the agent from bypassing simulation by calling the payment endpoint directly.
Fragment’s Conditions system implements a variant of this pattern. Conditions are programmable rules evaluated at posting time — they can check balances across multiple accounts, enforce minimum reserves, and reject entries that would violate business constraints. The condition is evaluated before the entry posts, and the entry is rejected atomically if the condition fails.
Simulation is not estimation. An estimation says “this will probably cost $112.” A simulation says “this transaction will succeed, will cost exactly $112.82, and will leave your balance at $4,087.18.” The difference is that a simulation is a dry run of the actual authorization path, not a prediction.
Kill Switches and Graceful Shutdown
When an agent is behaving anomalously, you need to stop it. The kill switch hierarchy provides three levels of intervention.
| Level | Action | Effect | Recovery |
|---|---|---|---|
| Pause | Suspend agent credential | New transactions rejected; in-flight transactions complete | Re-enable credential |
| Revoke | Invalidate agent credential | New transactions rejected; in-flight transactions complete | Issue new credential |
| Emergency stop | Freeze agent’s funding account | All transactions rejected including in-flight | Manual account review |
Pause is the first response. The agent’s delegation credential is suspended at the authorization layer. Any new payment request from this agent returns a denial. Transactions already in flight — authorized but not yet settled — complete normally because they were already authorized.
Revoke is permanent pause. The credential is invalidated and cannot be re-enabled. The principal must issue a new credential with a new constraint set. This is the appropriate response when the agent’s credential may have been compromised.
Emergency stop freezes the funding source. This is the nuclear option — it affects not just the agent but any entity drawing from the same account. Use it only when the velocity or magnitude of anomalous transactions suggests the account itself is compromised, not just the agent.
Graceful shutdown handles the in-flight state problem. An agent mid-conversation may have pending approvals, incomplete multi-step workflows, or holds that need to be released. The shutdown sequence:
- Stop accepting new payment requests from the agent
- Allow pending approval responses to arrive (with a timeout)
- Release any pre-authorization holds that will not be captured
- Log the shutdown event with the agent’s full session context
- Notify the principal with a summary of in-flight state
Anomaly Detection for Agent Behavior
Agent transaction patterns differ from human patterns in specific ways that standard fraud detection misses.
| Signal | Human pattern | Agent pattern | Detection |
|---|---|---|---|
| Transaction timing | Business hours, irregular intervals | Any hour, regular intervals | Time-series periodicity |
| Amount distribution | Variable, round numbers | Exact amounts, repeated values | Entropy analysis |
| Merchant diversity | Varied over time | Concentrated on task-specific merchants | Category concentration |
| Session duration | Minutes | Seconds to hours (no breaks) | Duration outliers |
| Retry behavior | Give up after 1–2 failures | Retry indefinitely until constraint stops it | Retry velocity |
The retry behavior signal is the most critical. A human denied a purchase moves on. An agent denied a purchase may retry with modified parameters — different amount, different merchant, different category — attempting to find a path around the constraint. This is not malice; it is the agent optimizing for its objective. A well-designed constraint system treats sequential denials with parameter variation as a single escalating anomaly, not as independent events.
Mastercard’s agent commerce framework specifies “real-time monitoring for unusual patterns” including “rapid successive transactions, unusual merchant category combinations, and transactions that systematically test constraint boundaries.”
Audit Trail Architecture
Every agent-initiated transaction carries metadata that distinguishes it from human-initiated transactions. The audit trail records not just what happened but why — the agent’s reasoning, the constraints evaluated, the approval path taken, and the principal who delegated authority.
{
"transaction_id": "txn_8f3a2b",
"initiated_by": "agent",
"agent_id": "agent_procurement_7x",
"principal_id": "user_jsmith",
"delegation_id": "del_abc123",
"constraint_set_version": 4,
"constraints_evaluated": {
"per_transaction": {"limit": 5000, "actual": 247.50, "result": "pass"},
"daily_aggregate": {"limit": 25000, "actual": 1892.30, "result": "pass"},
"category": {"allowed": ["office_supplies", "software"], "actual": "office_supplies", "result": "pass"}
},
"approval_tier": "auto_approve",
"simulation_ref": "sim_def456",
"agent_reasoning": "Purchasing toner cartridges per weekly supply checklist item #3"
}The agent_reasoning field is the agent’s self-reported intent. It is not trusted for authorization purposes — constraints enforce safety regardless of stated intent — but it is valuable for post-incident analysis and for the principal reviewing agent activity.
The delegation chain is the critical audit element. For any agent-initiated transaction, the audit trail must answer: who gave this agent the authority to spend this money, when, with what constraints, and has that delegation been modified or revoked since? The delegation ID links to the full constraint set that was active at the time of the transaction, not the current constraint set — because constraints may have been tightened after the fact.
Every agent session produces a complete, immutable record: the principal’s delegation, the agent’s actions, the constraints evaluated, the approvals obtained, and the outcomes. When a regulator or auditor asks “who authorized this payment,” the answer is never “the AI.” The answer is always a human principal, a specific delegation scope, and a verifiable constraint set that permitted the transaction.