Memory API and routing

Ask a node what an inference engine needs to know before it runs a prompt: what is already cached, where it lives, and whether to compute the rest locally or offload it. These routes sit alongside store and retrieve, and every one authenticates and is tenant-scoped like the rest of the API.

Routes

The Python clients (MembraneClient, AsyncMembraneClient) have a method for each: reconstruct, prefix_lookup, session, put_object, get_object, and route. Fragments of other tenants are never visible; see Security.

Reconstruct a prompt

curl -s localhost:8080/reconstruct -H "Authorization: Bearer $KEY" \
  -d '{"tokens": [1, 2, 3, ...], "model_id": "llama-3-8b"}'

{
  "fragments": [{"identity": {"payload_hash": "…", "token_span": [0, 127], …}, …}],
  "coverage": 0.85,
  "missing": [[256, 299]],
  "prefilled": false,
  "prefetch": ["…"]
}

The node walks its indexes: the longest cached prefix first, then fragments adjacent to it, then any fragment that fills a gap.

• A fragment is returned only when its content hash matches the prompt tokens it covers. KV computed for a different prompt is never returned just because it sits at the same position or has a similar embedding.
• missing lists the token spans that are not cached, and coverage is the fraction that is.
• prefetch lists fragments that past requests read right after these.

With "prefill": true, the node computes the missing spans with its compute backend, stores them under the caller’s tenant, and returns the complete set.

Look up a cached prefix

curl -s "localhost:8080/prefix/lookup?model_id=llama-3-8b&tokens=1,2,3,4"

The response has matched_tokens, total_tokens, full, and the fragments covering the match. Answers are memoized, and an answer is forgotten as soon as one of its fragments leaves the node. For prompts longer than 4,096 tokens, POST the same fields as a JSON body.

Sessions

A read that carries X-Membrane-Session: <id> (on /retrieve or /reconstruct) is recorded under that session, and GET /sessions/<id> returns the history, oldest first. Sessions are per tenant and bounded: 10,000 sessions of 1,000 reads each, evicting the least recently active first.

Typed memory objects

POST /objects stores an object and the bytes behind it in one request. The node derives the content hash from the bytes, so a caller cannot bind a hash to content it does not hold.

Every kind may also carry reuse_score and ttl. GET /objects/<hash> returns kind, the object’s fields, and data.

Route a request

curl -s localhost:8080/route -d '{"tokens": [...], "model_id": "llama-3-8b", "local_cached_tokens": 0}'

{
  "policy": "latency",
  "placement": {"fetch_from": "node-2", "fetch_from_url": "https://node-2:8080",
                "store_on": "node-1", "prefill_on": "node-1", "reason": "nearest holder", …},
  "fragments": [{"content_hash": "…", "node_id": "node-1", "url": null},
                {"content_hash": "…", "node_id": "node-2", "url": "https://node-2:8080"}],
  "matched_tokens": 256,
  "reuse": true,
  "offload": {"target": "membrane", "incremental_length": 44, "cached_prefix_length": 256,
              "cross_cluster_cache_transfer": false, "threshold": 100}
}

A request may send content_hash instead of tokens to place a single fragment.

Placement policies

--placement chooses the policy. Each is a membrane.placement plugin, so you can add your own; see Plugins.

Peers report their load in every heartbeat: round-trip latency (a moving average), memory pressure, GPU load, region, and role. A peer in another --region costs bandwidth.

Background policies