Operations Guide

Startup

CLI

# Start one or more services interactively
underwrite run mechanism audit risk

# Start as HTTP daemon (default: mechanism,audit)
underwrite serve
underwrite serve --host 0.0.0.0 --port 8080
underwrite serve --services "mechanism,audit,fraud" --rate-limit 200

# With auth
UNDERWRITE_API_TOKEN=prod-token underwrite serve --require-auth

# Init default config
underwrite init
underwrite init config.production.json

The run command starts services in the foreground with a synchronous event loop. The serve command wraps them in a FastAPI/uvicorn HTTP server.

Configuration Loading

On startup, the Runtime loads configuration in this order:

1. underwrite.json in working directory (if it exists)
2. config.<UNDERWRITE_ENV>.json (if UNDERWRITE_ENV is set)
3. Environment variable overrides (UNDERWRITE_*)

Create a default config with underwrite init — it enables mechanism and audit by default.

---

Shutdown

Graceful Shutdown

Send SIGTERM or SIGINT (Ctrl+C):

kill <pid>
docker stop underwrite

The Runtime performs an orderly shutdown:

1. Stops the metrics export loop
2. Stops all registered services (unsubscribes from bus)
3. Stops the event bus (flushes remaining events, waits for pending futures up to 5s)
4. Shuts down store backends (closes connection pools, shuts down thread pools)

The serve command supports --shutdown-timeout 30 (default 30s) for the HTTP server's graceful drain period.

Forcing Shutdown

SIGKILL (kill -9) is safe—saga idempotency keys in the store prevent duplicate event processing on restart.

---

Supervisor Auto-Restart

The ServiceSupervisor monitors service handler failures and auto-restarts crashed services.

Enable/disable with:

UNDERWRITE_RECOVERY_AUTO_RESTART=true    # enable (default)
UNDERWRITE_RECOVERY_AUTO_RESTART=false   # disable

Configured via:

{
  "recovery": {
    "auto_restart": true,
    "max_restarts": 5,
    "backoff_seconds": 2.0
  }
}

Behaviour

• After a handler exception, the supervisor records a failure
• Exponential backoff before restart: backoff_seconds * 2^(failure-1), capped at 60s
• After max_restarts consecutive failures, the service is marked permanently unhealthy
• On successful handler execution, the failure count resets
• The supervisor health check reports restarting services and total failures

Runtime Restart

The Runtime's restart_failing_services() method re-registers, re-wires, and re-starts failing services:

restarted = rt.restart_failing_services()
print(f"Restarted: {restarted}")  # ['fee', 'risk']

---

Circuit Breaker

The platform uses two layers of circuit breakers:

Bus Circuit Breaker (per-subscriber)

Tracks failures per subscriber ID (not per service). After 5 consecutive failures, the circuit opens for 60 seconds. While open, events are sent directly to the DLQ without invoking the handler. A successful request on the half-open state resets the circuit.

Store Resilience

The Sqlite store does not embed a per-store circuit breaker. The SQLite busy_timeout PRAGMA (default 30 s, configurable via UNDERWRITE_STORE_BUSY_TIMEOUT) covers transient contention by making contending callers wait rather than fail. Operators who want circuit-breaker semantics around the store can wrap calls in their own CircuitBreaker (underwrite.circuit).

Health:

underwrite health
# [OK] store — {"ok": true, "path": "./store.db"}

Or programmatically:

from underwrite.store import Sqlite

s = Sqlite(path="./store.db")
h = s.health()

Recovery from a corrupted file is manual: replace the SQLite file from a backup and re-run underwrite migrate so the migration runner no-ops.

---

Dead Letter Queue

The DLQ captures events that failed processing (handler exceptions, rate limiting, open circuits).

Inspect

underwrite dlq

Output:

Dead-letter queue: 3 entries
  [1717785600.0] subscriber-id: fee.assess — ProtocolError: must be finite
  [1717785601.5] subscriber-id: risk.scored — RateLimitError: rate limit exceeded
  [1717785602.0] subscriber-id: fraud.alert — CircuitBreakerOpenError: circuit is open

Replay

underwrite dlq --replay            # replay all
underwrite dlq --replay --max 10   # replay at most 10

Replayed events are re-published to the bus. Services with idempotency guards skip duplicate events. The DLQ is bounded at 10,000 records (oldest evicted first).

Persistence

When the Sqlite store is configured with a file path, the DLQ persists across restarts in the store table under the key bus:dlq. The dead-letter table created by migration v2 is also available for downstream tooling that prefers SQL access.

Programmatic

rt = Runtime()
dlq = rt.bus.dlq
print(dlq.count)
for record in dlq.records:
    print(record.event.event_type, record.error)
dlq.clear()

---

Migrations

The migration engine applies pending schema changes on startup (auto-migrate enabled by default).

# Run pending migrations manually
underwrite migrate

# Check applied versions
sqlite3 ./store.db "SELECT * FROM migrations ORDER BY version;"

Migration Plan

Current migrations (defined in migrate.py):

Version	Description
1	Initial store schema — key-value table, migrations table
2	Dead-letter queue table
3	Metrics snapshot table

Manual Rollback

-- Rollback version 3
DROP TABLE IF EXISTS metrics_snapshots;
DELETE FROM migrations WHERE version = 3;

-- Rollback version 2
DROP TABLE IF EXISTS dead_letters;
DELETE FROM migrations WHERE version = 2;

After rollback, underwrite migrate re-applies the migration.

---

Indian Regulatory Operations

NPA / SMA Monitoring

The platform tracks asset quality per RBI Master Circular on Income Recognition and Asset Classification (IRAC):

Classification	Trigger	Event	Action
SMA-0	30 days past due	sma.classified	Alert relationship manager
SMA-1	60 days past due	sma.classified	Initiate collection
SMA-2	90 days past due	sma.classified	Prepare NPA report
NPA (Substandard)	91-180 days	npa.bucket.changed	Provision at 15%, suspend income recognition
NPA (Doubtful)	181-360 days	npa.bucket.changed	Provision at 25% (secured)
NPA (Loss)	>360 days	npa.bucket.changed	Provision at 100%
DLG Trigger	120+ days	npa.dlg.triggered	Invoke default loss guarantee

Monitor NPA ratios:

# Check current NPA classification counts
underwrite health | grep npa

# Expected output:
# service:npa — events_handled=42 sma0=5 sma1=2 sma2=1 npa_substandard=1

Pricing Compliance Monitoring

Monitor that all loans fall within RBI-mandated rate caps:

underwrite metrics | grep -E "pricing|caps|penal"

Key metrics: - pricing.rate_caps — count of rate cap applications - pricing.penal_interest — penal interest assessments (should be ≤24% p.a.) - pricing.foreclosure — foreclosure charge computations (0% for personal/home loans)

Consent Audit Trail

All consent lifecycle events are recorded in the audit ledger:

# Query consent events (requires store inspection)
underwrite dlq | grep consent

Monitor consent expiry and withdrawal rates to ensure DPDPA compliance.

DSR Fulfillment SLA

The platform tracks DSR response times. If a DSR exceeds 30 days:

1. dsr.fulfilled is not emitted — check underwrite dlq for pending requests
2. Escalate via grievance.logged event
3. Manually verify DPO notification (configured via dpdpa.dsr.dpo_email)

Breach Detection

When breach.detected fires:

1. Identify scope via audit log: underwrite health and check store
2. Notify Data Protection Board within 72 hours (configurable)
3. Record breach closure via breach.closed event
4. Document in breach register

RBI Reporting Schedule

Report	Frequency	Data Source	Notes
NPA classification	Monthly	npa.bucket.changed events	RBI return on asset quality
Capital adequacy	Quarterly	Store aggregation	Leverage ratio monitoring
Interest rate disclosure	Monthly	Pricing service	Rate cap compliance report
KFS issuance log	Daily	KFS service	Cooling-off period tracking
Consent register	Monthly	Consent service	DPDPA compliance audit
Grievance register	Monthly	DSR service	DPDPA Section 13 compliance
Credit bureau data submission	Weekly	Credit bureau service	CIBIL/Experian/Equifax data refresh

---

Monitoring

Health

underwrite health

HTTP health endpoints (requires underwrite serve):

Endpoint	Path
Liveness probe	GET /healthz
Readiness probe	GET /readyz
Full status	GET /v1/health
Legacy	GET /health

Metrics

underwrite metrics

Counters: events.emitted, events.handled, events.failed, store.corruption, store.io_error, authz.failures

Timers: handle.duration (per-service, per-event-type with count/avg/min/max)

HTTP: GET /v1/metrics returns Prometheus text format (requires underwrite[serve]).

Logging

Configure via environment:

export UNDERWRITE_LOG_LEVEL=DEBUG
export UNDERWRITE_LOG_FORMAT=json

JSON format includes timestamp, level, logger, message, module, line, and optional correlation_id, trace_id, and exception. Sensitive fields (SSN, PAN, tokens, passwords) are automatically redacted.

Tracing

OpenTelemetry distributed tracing:

{
  "tracing": {
    "enabled": true,
    "exporter": "otlp"
  }
}

Requires underwrite[otlp] extra. Console exporter is also available for development.

---

Backup

Sqlite Backup

The store is a single file (default ./store.db). The simplest production backup is to copy the file while the runtime is paused or to use SQLite's .backup command:

# Hot backup via the sqlite3 CLI
sqlite3 ./store.db ".backup '/var/backups/underwrite-$(date +%Y%m%d).db'"

# Restore
cp /var/backups/underwrite-YYYYMMDD.db ./store.db

If WAL is enabled (the default), include -wal and -shm siblings in the copy or use the .backup command which freezes the write-ahead log atomically.

---

Recovery

Saga Replay

Sagas that were interrupted by a crash can be replayed:

from underwrite.runtime import Runtime

rt = Runtime()
success = rt.replay_saga("saga-id-here")

replay_saga() finds the next unexecuted step after the last completed one and executes all remaining steps. Idempotency keys ensure no step is executed twice.

Saga status values: started → completed (success), or compensating → rolled_back (failure).

DLQ Replay

After fixing the root cause (e.g., misconfiguration, missing env var), replay failed events:

underwrite dlq --replay

Service Restart

Manually restart a failing service via the Runtime:

rt.restart_failing_services()

---

Incident Response

1. Check System Health

underwrite health

If degraded, check individual checks: bus, store, service:<name>, supervisor.

2. Check Dead Letter Queue

underwrite dlq

Look for patterns: all errors from one service, rate limiting, circuit open.

3. Check Logs

UNDERWRITE_LOG_LEVEL=DEBUG underwrite run <service>

With JSON logging:

underwrite serve --port 8080 | jq 'select(.level == "ERROR")'

4. Check Circuit Breakers

underwrite health | grep circuit

If circuits are open, wait for automatic recovery (15–60s depending on component).

5. Common Recovery Actions

Issue	Action
Circuit breaker open	Wait for cooldown, or check store connectivity
DLQ accumulating	Fix handler error, then underwrite dlq --replay
Service crash-looping	Check logs, increase max_restarts or disable auto_restart
Saga stuck in started	rt.replay_saga(id) to retry
Migration failed	SELECT * FROM migrations, rollback failed version, fix and re-migrate
Store connection lost	Check DB endpoint, credentials, network policy
Signature verification failures	Check authz policy file and service identities

---

CLI Command Reference

Command	Description
underwrite init [path]	Create default config file
underwrite run <service>...	Start services in foreground
underwrite serve	Start HTTP daemon with health/metrics endpoints
underwrite list	List all available services
underwrite health	Show health status
underwrite metrics	Show metrics snapshot
underwrite dlq	Show dead-letter queue
underwrite dlq --replay	Replay dead-letter events
underwrite migrate	Run pending migrations
underwrite identity <service>	Generate Ed25519 identity for a service

---

Supported Plugins and Extras

Install extras with pip install underwrite[<extra>]:

Extra	Provides
serve	FastAPI + uvicorn HTTP server
otlp	OpenTelemetry distributed tracing
risk	NumPy + scikit-learn for risk scoring
vault	HashiCorp Vault secrets backend
aws	AWS Secrets Manager / S3 / SQS backends
gcs	Google Cloud Storage backend
dev	Pytest, ruff, mypy, bandit
mutation	Mutation testing (mutmut)
security	Bandit + pip-audit
all	All extras combined

Prometheus

When serve extra is installed, GET /v1/metrics exposes runtime and service metrics in Prometheus text format at text/plain; version=0.0.4. FastAPI can also be instrumented with OpenTelemetry via opentelemetry-instrumentation-fastapi.