HSMs and Key Management at Scale

Prerequisite: Security Model

Your stablecoin mint authority is a single ECDSA private key. That key can create money — anyone who obtains it can mint tokens without limit, drain reserves by flooding the market with unbacked supply, and do it in a single transaction that takes 12 seconds to finalize on Ethereum. The key is not a credential you can rotate after a breach. Once the attacker has it, every token minted before you redeploy the contract is at risk. The entire security model of a stablecoin issuer, a custodian, or a crypto payment processor reduces to one question: where does the private key live, and who can make it sign?

Why Software Keys Fail

A private key stored in a file, an environment variable, or an encrypted blob on disk is extractable. The key material exists in process memory during signing. A memory dump, a side-channel attack, a compromised dependency, or a misconfigured container mount can expose it. Once copied, the attacker has the same signing capability as the legitimate system — and you cannot tell the difference between their signatures and yours.

Storage method Extraction vector Detection difficulty
File on disk File read, backup exfiltration Low — access logs may exist
Environment variable Process inspection, container metadata API Low — often logged
Encrypted blob (KMS envelope) KMS API call from compromised workload Medium — KMS logs the decrypt
In-memory only Memory dump, cold boot, side-channel High — no persistent artifact

Even KMS-managed keys that never touch disk in plaintext can be extracted through the application layer. If your signing service has a KMS decrypt permission and an attacker compromises that service, they can call KMS directly. The envelope encryption protects the key at rest — not the key in use.

HSM Architecture

A Hardware Security Module is a tamper-resistant device that generates, stores, and uses cryptographic keys without ever exposing the key material outside its physical boundary. The private key is generated inside the HSM and never leaves. Signing operations are performed inside the HSM — your application sends the data to be signed, the HSM returns the signature, and the key material stays on the device.

Application → "sign this hash" → HSM → signature
                                  ↑
                          key never exits

HSMs are validated against FIPS 140-2 (or its successor FIPS 140-3) at defined security levels. Level 3 — the standard for financial key management — requires physical tamper-resistance, identity-based authentication, and physical or logical separation between interfaces.

FIPS 140 level Tamper protection Authentication Use case
Level 1 None Role-based Development, testing
Level 2 Tamper-evident seals Role-based Low-value signing
Level 3 Tamper-resistant enclosure, zeroization Identity-based Production financial keys
Level 4 Active tamper response Multi-factor Military, government

Cloud HSM services — AWS CloudHSM, Azure Dedicated HSM, Google Cloud HSM — provide FIPS 140-2 Level 3 validated hardware in your cloud VPC. You manage the keys; the cloud provider manages the hardware. AWS CloudHSM runs on Cavium/Marvell LiquidSecurity hardware at approximately $1.50/hour per HSM instance ($1,095/month). A production cluster requires at least two HSMs across availability zones for redundancy, bringing the baseline to $2,190/month before considering throughput scaling.

See Security Model. The security model chapter covers defense in depth. HSMs are one layer — they protect the key at rest and during signing, but they do not protect the authorization logic that decides what gets signed.

Key Hierarchy

A flat key structure — one key that signs everything — creates a single point of compromise. A key hierarchy separates authority into layers, where higher-level keys are used rarely and protected more heavily, while lower-level keys handle daily operations and can be rotated without ceremony.

Root Key (offline, ceremony-only)
├── Intermediate Key (HSM-resident, rare use)
│   ├── Operational Signing Key (HSM-resident, daily use)
│   │   ├── Hot Wallet Signer (automated, per-chain)
│   │   └── Hot Wallet Signer (automated, per-chain)
│   └── Mint Authority Key (HSM-resident, gated)
└── Backup Root (cold storage, disaster recovery only)

The root key signs intermediate certificates and is used only during key ceremonies — typically once at initial setup and then only for intermediate key rotation (annually or less). It lives offline: on an air-gapped machine, on hardware tokens stored in a vault, or split across multiple geographic locations using Shamir’s Secret Sharing.

Intermediate keys sign operational keys and are HSM-resident but invoked infrequently. Rotating an operational signing key requires the intermediate key. Compromising an operational key does not compromise the root — you revoke the operational key, generate a new one signed by the intermediate, and resume operations without a full ceremony.

Operational signing keys handle day-to-day transactions: signing blockchain transactions, authorizing API calls, encrypting sensitive payloads. These keys live in HSMs or managed KMS and are scoped to specific operations. A hot wallet signer for Ethereum cannot sign Solana transactions. A mint authority key cannot sign withdrawal transactions.

Key Ceremonies

A key ceremony is a procedural event where cryptographic keys are generated, rotated, or recovered under controlled conditions with witnesses, audit documentation, and physical security controls. The ceremony exists because key generation is the highest-risk operation in the system — a compromised key generation process compromises every future operation that key touches.

A production key ceremony for a root or intermediate key involves:

  1. Preparation. Air-gapped machine, verified firmware, tamper-evident bags for hardware tokens, printed scripts (not from network-connected devices). Two independent witnesses minimum.
  2. Key generation. Generate the key pair on the air-gapped HSM or hardware token. Verify the public key fingerprint independently on two separate devices.
  3. Backup creation. Split the key into M-of-N shares using Shamir’s Secret Sharing. Common configuration: 3-of-5 shares, each stored in a separate geographic location in tamper-evident containers.
  4. Verification. Sign a test message with the new key. Verify the signature using only the public key on a separate machine. Confirm the key ID matches across all records.
  5. Documentation. Record the public key, key ID, share custodian assignments (by role, not name), tamper-evident bag serial numbers, and witness signatures. This document is the audit trail.

The ceremony typically takes 2–4 hours for a root key and 1–2 hours for an intermediate key rotation. The cost is primarily personnel time — cryptographic engineers, security officers, and witnesses. For a stablecoin issuer, the ceremony may also require the presence of a compliance officer and the generation of attestation records for regulators.

The ceremony script must be tested on non-production hardware before execution. A ceremony that fails midway — hardware incompatibility, firmware mismatch, corrupted media — leaves you with partially generated keys and no clean recovery path. Run the full script end-to-end on identical hardware at least once before the production ceremony.

Rotation Policies

Key rotation limits the blast radius of a compromise. A key that has been in use for three years has three years of signed transactions that an attacker with the compromised key could retroactively claim to have authorized. A key rotated monthly limits that window to 30 days.

Key type Rotation frequency Rotation method
Root key 3–5 years or on compromise Full ceremony, share redistribution
Intermediate key 12–18 months Ceremony with intermediate shares
Operational signing key 30–90 days Automated, HSM-to-HSM
Hot wallet signer 30 days or per-incident Automated with address migration
API signing key 90 days Automated, dual-key overlap period

Automated rotation for operational keys follows a dual-key overlap pattern. Generate the new key, begin signing with both old and new keys (verifiers accept either), migrate all verifiers to the new key, then decommission the old key after a grace period. The grace period must exceed the longest in-flight transaction lifetime — if cross-chain messages take up to 30 minutes to finalize, the old key must remain valid for at least 30 minutes after the last transaction signed with it.

Hot wallet signer rotation on blockchains requires address migration. The new key produces a new address. Funds must be transferred from the old address to the new one. During the migration window, both addresses are active, and monitoring must cover both. A rotation that moves $50M in assets across 10 chains at current gas costs runs $500–$5,000 in transaction fees depending on chain congestion.

Authorization Controls

The HSM protects the key from extraction. Authorization controls protect the key from misuse. A compromised application with HSM access can sign anything the HSM’s access policy allows — the HSM does not understand business logic. It signs what it is asked to sign.

Authorization layers between the application and the HSM:

  1. Authentication. The application must authenticate to the HSM with credentials that are themselves protected — typically mutual TLS certificates or HSM partition passwords. AWS CloudHSM uses a crypto user (CU) account with a password stored in Secrets Manager.
  2. Rate limiting. Cap signing operations per minute, per hour, per day. A mint authority key that normally signs 10 transactions per day should alert at 50 and hard-block at 100.
  3. Policy engine. Before forwarding a signing request to the HSM, a policy engine evaluates the request against rules: transaction amount limits, destination whitelist, time-of-day restrictions, velocity checks. The policy engine runs outside the HSM but before it.
  4. Multi-party authorization. High-value operations require approval from multiple parties before the HSM signs. A mint of $10M requires two of three authorized signers to approve through an out-of-band channel before the signing service forwards the request.
Request → Auth → Rate Limiter → Policy Engine → [MPA if threshold] → HSM → Signature

The multi-party authorization (MPA) threshold is the most consequential configuration decision. Set it too low and routine operations require manual approval, creating bottlenecks and on-call fatigue. Set it too high and large unauthorized transactions execute without human review.

Operation Typical MPA threshold Approval requirement
Hot wallet transfer (routine) < $50K Automated
Hot wallet transfer (large) $50K–$500K 1-of-2 approvers
Mint/burn Any amount 2-of-3 approvers
Key rotation N/A Ceremony required
Emergency pause Any 1-of-N (speed over consensus)

Disaster Recovery

Key loss is not a standard recovery scenario. If the root key is lost and no backup shares exist, every key signed by that root is orphaned. For a stablecoin issuer, this means the mint authority cannot be rotated, and a contract migration is required — a months-long process involving audits, regulatory notification, and customer communication.

The disaster recovery plan for cryptographic keys includes:

Share recovery. Assemble M-of-N Shamir shares to reconstruct the root or intermediate key. This requires physically transporting shares from multiple locations, verifying tamper-evident seals, and reconstructing in a ceremony environment. Recovery time: 24–72 hours depending on share custodian availability and geography.

HSM cluster recovery. Cloud HSM services replicate key material across availability zones within a cluster. If one HSM instance fails, the cluster continues operating. If the entire cluster fails (region outage), you restore from a backup of the HSM cluster to a new region. AWS CloudHSM backups are encrypted and can only be restored to CloudHSM instances in the same AWS account.

Cold storage activation. The backup root key, stored offline in a separate jurisdiction, can sign new intermediate keys if the primary root is compromised or lost. Activating cold storage requires a full ceremony with the backup share custodians.

The recovery time objective (RTO) for signing operations depends on the key tier. Operational key recovery from HSM cluster redundancy: minutes. Intermediate key recovery from share reconstruction: 24–72 hours. Root key recovery from cold storage: 3–7 days. Your system must tolerate signing unavailability for the duration of the recovery — hot wallet operations halt, minting stops, and the operational impact cascades to every service that depends on signed transactions.