Operations

How to run, deploy, monitor, and shut down the Agent Passport service. Source-of-truth for: environment variables, deployment, observability (metrics, SLOs, alerts, runbooks), rate limiting, idempotency, system-exposure cap, graceful shutdown, and load testing.

1. Environment variables

The canonical env-var table. Every variable the service reads is listed here, sourced from .env.example and src/config.ts. If you add a new env var, update this page, .env.example, and src/config.ts together.

Copy .env.example to .env and edit. The service calls dotenv.config() on startup so .env is loaded automatically.

Service

Algorand

For mainnet, point at AlgoNode mainnet, a hosted provider (Nodely, BCC), or your own node. See § 4 for latency trade-offs.

Smart contracts

See security.md for the operator mnemonic handling and KMS guidance.

x402

Rate limiting

Persistence

Idempotency

The idempotency store is configured entirely in code (no env vars):

Auth (HMAC)

Sanctions

System exposure cap (hard-coded)

Constants (hard-coded, no env var)

x402 pricing (X402_PRICING in src/lib/constants.ts)

2. Rate limiting

Per-IP fixed-window rate limiter at src/lib/security.ts. Default 600 req/min/IP.

Configuration

Bypass lists

The middleware short-circuits to next() in three cases:

/metrics, /registry/status are never rate-limited.

use; never in production).

limit. Use for internal services, the operator host, or your monitoring agent.

1. Operational endpoints. /health, /ready, /health/deep,
2. LOAD_TEST_MODE=1. All rate limiting is disabled (k6 suite
3. Trusted IPs. Any IP in RATE_LIMIT_TRUSTED_IPS bypasses the

Response headers

When the limit is exceeded, the response is 429 with body:

{ "error": "Too many requests. Try again later." }

State is persisted to data/rate-limit.json via the shared json-store.ts write-queue mutex (race-free on concurrent saves).

3. Idempotency

The Idempotency-Key middleware (src/lib/idempotency.ts) makes mutating calls safe to retry. It applies only to non-GET / non- HEAD / non-OPTIONS requests; safe methods pass through with no key required.

Flow

For each request with a valid Idempotency-Key header:

idempotent-replay: true.

reused with different request body` and record a metric.

middleware no longer auto-generates one). The request is processed and the response cached.

1. Look up the key in the in-memory store.
2. On hit + same body hash → return the cached response with
3. On hit + different body hash → return `409 Idempotency-Key
4. On miss → require the client to send an Idempotency-Key (the

Header format

Idempotency-Key: <8-255 chars, [A-Za-z0-9_\-:]+>

POST /reputation/record — missing key returns 400

• 8–255 characters
• Allowed: ASCII letters, digits, underscore, hyphen, colon
• Anything else returns 400 Invalid Idempotency-Key format
• Required on POST /delegate, POST /revoke, and

Body hashing

Body hash is sha256(canonicalJson(body)) where canonicalJson sorts keys recursively. So {"a":1,"b":2} and {"b":2,"a":1} produce the same digest. Same key + same body → cached response. Same key + different body → 409.

In-memory store

Map<string, IdempotencyRecord> at src/lib/idempotency.ts:21. Each record:

interface IdempotencyRecord {
  key: string;
  bodyHash: string;     // sha256 hex
  status: number;       // HTTP status code (200-299)
  body: unknown;        // Response body
  createdAt: number;    // Unix ms
  expiresAt: number;    // Unix ms (createdAt + 24h)
}

The sweeper runs every 5 minutes:

oldest keys in insertion order (FIFO overflow)

• Removes any record where expiresAt <= now
• If the store is over MAX_STORE_SIZE = 10 000, evicts the

The store is not persisted to disk. A process restart loses all in-flight idempotency state. Clients that retry with the same key after a server restart will re-execute the request.

Metrics

Endpoints requiring Idempotency-Key

Multi-replica

For deployments with > 1 replica, back the idempotency store with Redis. The Idempotency-Key contract guarantees at-most-once execution; without a shared store, two replicas can both serve the same key and both execute the underlying operation.

4. Deployment

Quick start

npm install
cp .env.example .env
npm start

By default this points at the public Algorand testnet — no setup beyond the env file is needed.

Going to production — checklist

1. Choose your Algorand network

The measured k6 testnet baseline (P95 < 1.5s, 99% availability) is what you should expect with any of the first three options. The local-node option is an upgrade path if you need the stricter prod-strict SLOs (P95 < 500ms, 99.9% availability). See § 7.

2. Set ALGOD_URL and INDEXER_URL

# Testnet (default — no change needed)
ALGOD_URL=https://testnet-api.algonode.cloud:443
INDEXER_URL=https://testnet-idx.algonode.cloud:443

# Mainnet via AlgoNode
ALGOD_URL=https://mainnet-api.algonode.cloud:443
INDEXER_URL=https://mainnet-idx.algonode.cloud:443

3. Set the operator mnemonic

OPERATOR_MNEMONIC="word1 word2 ... word25"

Or load from a secret manager at startup. See security.md for KMS guidance.

4. Set HMAC_SECRET (recommended for production)

HMAC_SECRET="$(openssl rand -hex 32)"   # 64 hex chars = 256 bits

Any state-changing endpoint will then require HMAC-SHA256 authentication. Public reads and the operational endpoints remain unauthenticated in development. Production requires HMAC. See security.md.

5. Build the Docker image

docker build -t agent-passport:0.1.0 .
docker run --rm -p 3000:3000 --env-file .env agent-passport:0.1.0

The Dockerfile is multi-stage, runs as non-root, includes a healthcheck, and uses tini as PID 1 for proper signal forwarding.

6. Kubernetes probe example

livenessProbe:
  httpGet: { path: /health, port: 3000 }
  initialDelaySeconds: 10
  periodSeconds: 30
readinessProbe:
  httpGet: { path: /ready, port: 3000 }
  initialDelaySeconds: 5
  periodSeconds: 10

5. Observability

The service exposes Prometheus-format metrics at GET /metrics. This endpoint is:

• Exempt from rate limiting (operational)
• Always returns 200 unless the process is severely broken
• Refreshes process gauges on every scrape (memory, CPU, uptime)

Metric inventory

API metrics

status_class is 2xx/3xx/4xx/5xx (not the raw status code) to bound label cardinality. Buckets for http_request_duration_seconds: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10.

Trust engine metrics

x402 payment metrics

Contract metrics

Cache metrics

Business metrics

Infrastructure metrics

Label cardinality rules

To keep Prometheus healthy:

req.path which can include wallet addresses

• path is normalized to the route template — never use the raw
• method is GET/POST/etc. (bounded)
• status_class is 2xx/3xx/4xx/5xx (bounded)
• wallet is never a label — use unique_wallets gauge
• error_type is client_error/server_error (bounded)
• outcome is approved/denied or allow/deny (bounded)
• risk_level is low/medium/high/critical (bounded)
• network is testnet/mainnet (bounded)
• flag is funded/active/empty/lookup_failed (bounded)

Scrape configuration

scrape_configs:
  - job_name: agent-passport
    metrics_path: /metrics
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets: ['agent-passport:3000']

6. SLOs

The SLO files are split by deployment target:

Prod-relaxed SLOs (default, measured)

Based on the k6 load test run against the public Algorand testnet (AlgoNode free tier):

The prod-relaxed SLOs are realistic for any deployment using a public Algorand endpoint because:

produces natural 429s

call

bounded by 5 × 800ms = 4s in the worst case

• AlgoNode's free tier rate-limits at ~1,000 req/s per IP, which
• Round-trips to a remote indexer/algod add 200-800ms latency per
• /underwrite makes 4-5 Algorand round-trips, so its P95 is

Prod-strict SLOs (aspirational)

How to hit the prod-strict targets: any combination of:

— the single biggest lever)

• Local Algorand node (drops per-round-trip from 200-800ms to 5-20ms
• Premium hosted mainnet provider (Nodely, BCC, AlgoNode paid tier)
• Geographic co-location with an Algorand relay

The prod-relaxed targets are real, measured, and production-grade. Switch to prod-strict only if you need 500ms P95 and are willing to operate the infrastructure for it.

Per-endpoint latency projections

7. Alert-to-runbook map

Runbooks live in alerts/runbooks/<name>.md.

8. Dashboard

The Grafana dashboard JSON is at alerts/grafana-dashboard.json. It includes:

• API Request Rate (overall and per-endpoint)
• API Latency (P50/P95/P99)
• Error Rate
• Trust Score Latency
• Graph Traversal Latency
• x402 Payments (verified/failures/replay)
• Contract Events (endorsements/revocations/disputes/success)
• Process Memory (heap/RSS)
• Process Uptime
• Passports Generated
• Unique Wallets
• Cache Performance (hits/misses/evictions)

9. System exposure cap

write-queue mutex

globalRemaining, walletRemaining)` — the final amount reserved

• MAX_SYSTEM_EXPOSURE = 100_000 USDC (hard-coded)
• Per-wallet cap: MAX_SYSTEM_EXPOSURE / 10 = 10_000 USDC
• Persisted to data/system-exposure.json via json-store.ts
• capToSystemCapacity(wallet, limit) returns `min(limit,

Multi-replica needs Redis (or accept the overshoot) — the JSON file is per-process.

10. Load testing

k6 suite under load-tests/. Run via:

brew install k6
cd load-tests
LOAD_TEST_MODE=1 ./run-all.sh

Four scenarios: 100 VU, 500 VU, 1000 VU, and sustained 12 rps. The sustained scenario is a smoke test that always passes; the VU scenarios require a local Algorand node to hit P95 < 1.5s.

Thresholds:

• Error rate: < 1%
• P95 latency: < 1.5s (prod-relaxed), < 500ms (prod-strict)
• Throughput: documented per scenario

11. Graceful shutdown

The service registers signal handlers in src/index.ts:38-46 and a matching cleanup hook in src/app.ts.

Signal handlers

Shutdown flow for SIGTERM / SIGINT

and drains in-flight requests.

complete, log "Forced shutdown after timeout" and call process.exit(1).

process.exit(0).

1. Log "Received SIGTERM/SIGINT, shutting down gracefully".
2. Call server.close() — Express stops accepting new connections
3. Set a 10-second setTimeout — if server.close() does not
4. On server.close() callback, log "HTTP server closed" and call

Resources stopped on shutdown

stopMetricsCollectors — clears the 15s setInterval for process gauges. stopRateLimiter — clears the 5-min cleanup interval. stopIdempotencySweeper — clears the 5-min idempotency sweeper. stopDedupCleanup — clears the reputation dedup cleanup. closeLoggerStreams — closes the log file streams.

All setInterval timers are .unref()-ed so they do not block process exit on their own.

Kubernetes shutdown order

Kubernetes sends SIGTERM first, then SIGKILL after terminationGracePeriodSeconds (default 30s). Configure the pod:

terminationGracePeriodSeconds: 30

to give the service time to drain. The 10s setTimeout in gracefulShutdown is the inner bound; the 30s K8s limit is the outer bound.