Operations

How to run, deploy, monitor, and shut down the Agent Passport service. Source-of-truth for: environment variables, deployment, observability (metrics, SLOs, alerts, runbooks), rate limiting, idempotency, system-exposure cap, graceful shutdown, and load testing.

1. Environment variables

The canonical env-var table. Every variable the service reads is listed here, sourced from .env.example and src/config.ts. If you add a new env var, update this page, .env.example, and src/config.ts together.

Copy .env.example to .env and edit. The service calls dotenv.config() on startup so .env is loaded automatically.

Service

Algorand

For mainnet, point at AlgoNode mainnet, a hosted provider (Nodely, BCC), or your own node. See § 4 for latency trade-offs.

Smart contracts

See the operator-wallet security guide for the operator mnemonic handling and KMS guidance.

x402

Rate limiting

Persistence

Idempotency

The idempotency store is configured entirely in code (no env vars):

Auth (HMAC)

Sanctions

System exposure cap (hard-coded)

Constants (hard-coded, no env var)

x402 pricing (X402_PRICING in src/lib/constants.ts)

2. Rate limiting

Per-IP fixed-window rate limiter at src/lib/security.ts. Default 600 req/min/IP.

Configuration

Bypass lists

The middleware short-circuits to next() in three cases:

1. Operational endpoints. /health, /ready, /health/deep, /metrics, /registry/status are never rate-limited.
2. LOAD_TEST_MODE=1. All rate limiting is disabled (k6 suite use; never in production).
3. Trusted IPs. Any IP in RATE_LIMIT_TRUSTED_IPS bypasses the limit. Use for internal services, the operator host, or your monitoring agent.

Response headers

When the limit is exceeded, the response is 429 with body:

{ "error": "Too many requests. Try again later." }

State is persisted to data/rate-limit.json via the shared json-store.ts write-queue mutex (race-free on concurrent saves).

3. Idempotency

The Idempotency-Key middleware (src/lib/idempotency.ts) makes mutating calls safe to retry. It applies only to non-GET / non- HEAD / non-OPTIONS requests; safe methods pass through with no key required.

Flow

For each request with a valid Idempotency-Key header:

1. Look up the key in the in-memory store.
2. On hit + same body hash → return the cached response with idempotent-replay: true.
3. On hit + different body hash → return 409 Idempotency-Key reused with different request body and record a metric.
4. On miss → require the client to send an Idempotency-Key (the middleware no longer auto-generates one). The request is processed and the response cached.

Header format

Idempotency-Key: <8-255 chars, [A-Za-z0-9_\-:]+>

• 8–255 characters
• Allowed: ASCII letters, digits, underscore, hyphen, colon
• Anything else returns 400 Invalid Idempotency-Key format
• Required on POST /delegate, POST /revoke, and POST /reputation/record — missing key returns 400

Body hashing

Body hash is sha256(canonicalJson(body)) where canonicalJson sorts keys recursively. So {"a":1,"b":2} and {"b":2,"a":1} produce the same digest. Same key + same body → cached response. Same key + different body → 409.

In-memory store

Map<string, IdempotencyRecord> at src/lib/idempotency.ts:21. Each record:

interface IdempotencyRecord {
  key: string;
  bodyHash: string;     // sha256 hex
  status: number;       // HTTP status code (200-299)
  body: unknown;        // Response body
  createdAt: number;    // Unix ms
  expiresAt: number;    // Unix ms (createdAt + 24h)
}

The sweeper runs every 5 minutes:

• Removes any record where expiresAt <= now
• If the store is over MAX_STORE_SIZE = 10 000, evicts the oldest keys in insertion order (FIFO overflow)

The store is not persisted to disk. A process restart loses all in-flight idempotency state. Clients that retry with the same key after a server restart will re-execute the request.

Metrics

Endpoints requiring Idempotency-Key

Multi-replica

For deployments with > 1 replica, back the idempotency store with shared storage such as Redis. Without shared storage, two replicas can both receive the same key and execute the underlying operation; the middleware is not a globally coordinated exactly-once ledger.

4. Deployment

Quick start

npm ci
cp .env.example .env
npm start

By default this points at the public Algorand testnet — no setup beyond the env file is needed.

Going to production — checklist

1. Choose your Algorand network

The measured k6 testnet baseline (P95 < 1.5s, 99% availability) is what you should expect with any of the first three options. The local-node option is an upgrade path if you need the stricter prod-strict SLOs (P95 < 500ms, 99.9% availability). See § 7.

2. Set ALGOD_URL and INDEXER_URL

# Testnet (default — no change needed)
ALGOD_URL=https://testnet-api.algonode.cloud:443
INDEXER_URL=https://testnet-idx.algonode.cloud:443

# Mainnet via AlgoNode
ALGOD_URL=https://mainnet-api.algonode.cloud:443
INDEXER_URL=https://mainnet-idx.algonode.cloud:443

3. Set the operator mnemonic

OPERATOR_MNEMONIC="word1 word2 ... word25"

Or load from a secret manager at startup. See the operator-wallet security guide for KMS guidance.

4. Set HMAC_SECRET (recommended for production)

HMAC_SECRET="$(openssl rand -hex 32)"   # 64 hex chars = 256 bits

Any state-changing endpoint will then require HMAC-SHA256 authentication. Public reads and the operational endpoints remain unauthenticated in development. Production requires HMAC. See the HMAC authentication guide.

5. Build the Docker image

docker build -t agent-passport:1.0.0 .
docker run --rm -p 3000:3000 --env-file .env agent-passport:1.0.0

The Dockerfile is multi-stage, runs as non-root, includes a liveness healthcheck at /health, and uses tini as PID 1 for proper signal forwarding. /ready remains the dependency-aware readiness probe and should be used for load balancer or orchestrator traffic gating. CI builds and starts this image with production settings as a release gate.

6. Kubernetes probe example

livenessProbe:
  httpGet: { path: /health, port: 3000 }
  initialDelaySeconds: 10
  periodSeconds: 30
readinessProbe:
  httpGet: { path: /ready, port: 3000 }
  initialDelaySeconds: 5
  periodSeconds: 10

5. Observability

The service exposes Prometheus-format metrics at GET /metrics. This endpoint is:

• Exempt from rate limiting (operational)
• Always returns 200 unless the process is severely broken
• Refreshes process gauges on every scrape (memory, CPU, uptime)

Metric inventory

API metrics

status_class is 2xx/3xx/4xx/5xx (not the raw status code) to bound label cardinality. Buckets for http_request_duration_seconds: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10.

Trust engine metrics

x402 payment metrics

Contract metrics

Cache metrics

Business metrics

Infrastructure metrics

Label cardinality rules

To keep Prometheus healthy:

• path is normalized to the route template — never use the raw req.path which can include wallet addresses
• method is GET/POST/etc. (bounded)
• status_class is 2xx/3xx/4xx/5xx (bounded)
• wallet is never a label — use unique_wallets gauge
• error_type is client_error/server_error (bounded)
• outcome is approved/denied or allow/deny (bounded)
• risk_level is low/medium/high/critical (bounded)
• network is testnet/mainnet (bounded)
• flag is funded/active/empty/lookup_failed (bounded)

Scrape configuration

scrape_configs:
  - job_name: agent-passport
    metrics_path: /metrics
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets: ['agent-passport:3000']

6. SLOs

The SLO files are split by deployment target:

Prod-relaxed SLOs (default, measured)

Based on the k6 load test run against the public Algorand testnet (AlgoNode free tier):

The prod-relaxed SLOs are realistic for any deployment using a public Algorand endpoint because:

• AlgoNode's free tier rate-limits at ~1,000 req/s per IP, which produces natural 429s
• Round-trips to a remote indexer/algod add 200-800ms latency per call
• /underwrite makes 4-5 Algorand round-trips, so its P95 is bounded by 5 × 800ms = 4s in the worst case

Prod-strict SLOs (aspirational)

How to hit the prod-strict targets: any combination of:

• Local Algorand node (drops per-round-trip from 200-800ms to 5-20ms — the single biggest lever)
• Premium hosted mainnet provider (Nodely, BCC, AlgoNode paid tier)
• Geographic co-location with an Algorand relay

The prod-relaxed figures are measured load-test observations, not an SLA or a guarantee for a particular deployment. Switch to prod-strict only if you need 500ms P95 and are willing to operate and measure the required infrastructure.

Per-endpoint latency projections

7. Alert-to-runbook map

Runbooks live in alerts/runbooks/<name>.md.

8. Dashboard

The Grafana dashboard JSON is at alerts/grafana-dashboard.json. It includes:

• API Request Rate (overall and per-endpoint)
• API Latency (P50/P95/P99)
• Error Rate
• Trust Score Latency
• Graph Traversal Latency
• x402 Payments (verified/failures/replay)
• Contract Events (endorsements/revocations/disputes/success)
• Process Memory (heap/RSS)
• Process Uptime
• Passports Generated
• Unique Wallets
• Cache Performance (hits/misses/evictions)

9. System exposure cap

• MAX_SYSTEM_EXPOSURE = 100_000 USDC (hard-coded)
• Per-wallet cap: MAX_SYSTEM_EXPOSURE / 10 = 10_000 USDC
• Persisted to data/system-exposure.json via json-store.ts write-queue mutex
• capToSystemCapacity(wallet, limit) returns min(limit, globalRemaining, walletRemaining) — the final amount reserved

Multi-replica needs Redis (or accept the overshoot) — the JSON file is per-process.

10. Load testing

k6 suite under load-tests/. Run via:

brew install k6
cd load-tests
LOAD_TEST_MODE=1 ./run-all.sh

Four scenarios: 100 VU, 500 VU, 1000 VU, and sustained 12 rps. The sustained scenario is a smoke test that always passes; the VU scenarios require a local Algorand node to hit P95 < 1.5s.

Thresholds:

• Error rate: < 1%
• P95 latency: < 1.5s (prod-relaxed), < 500ms (prod-strict)
• Throughput: documented per scenario

11. Graceful shutdown

The service registers signal handlers in src/index.ts:38-46 and a matching cleanup hook in src/app.ts.

Signal handlers

Shutdown flow for SIGTERM / SIGINT

1. Log "Received SIGTERM/SIGINT, shutting down gracefully".
2. Call server.close() — Express stops accepting new connections and drains in-flight requests.
3. Set a 10-second setTimeout — if server.close() does not complete, log "Forced shutdown after timeout" and call process.exit(1).
4. On server.close() callback, log "HTTP server closed" and call process.exit(0).

Resources stopped on shutdown

stopMetricsCollectors — clears the 15s setInterval for process gauges. stopRateLimiter — clears the 5-min cleanup interval. stopIdempotencySweeper — clears the 5-min idempotency sweeper. stopDedupCleanup — clears the reputation dedup cleanup. closeLoggerStreams — closes the log file streams.

All setInterval timers are .unref()-ed so they do not block process exit on their own.

Kubernetes shutdown order

Kubernetes sends SIGTERM first, then SIGKILL after terminationGracePeriodSeconds (default 30s). Configure the pod:

terminationGracePeriodSeconds: 30

to give the service time to drain. The 10s setTimeout in gracefulShutdown is the inner bound; the 30s K8s limit is the outer bound.