The shared KV‑cache for LLM serving.

Membrane stores prefill results once and serves them to every node in your fleet. Repeated prompts skip prefill, time-to-first-token drops, and GPUs stop recomputing the same context.

Get started View on GitHub

Adapters for vLLM, SGLang, TensorRT-LLM
Deduplicated, content-addressed storage
mTLS or API keys on every route
Tenant isolation, encryption at rest

node-a · :8080

# 1. A prompt is prefilled once, anywhere in the fleet
$ curl -s :8080/prefill -d '{"prompt_tokens": [0, …, 299],
         "model_id": "llama-3-8b"}'
→ 3 fragments stored   tokens 0–127 · 128–255 · 256–299

# 2. The same context, on any node, comes back from cache
$ curl -s :8080/reconstruct -d '{"tokens": [0, …, 299],
         "model_id": "llama-3-8b"}'
{ "coverage": 1.0, "prefilled": false,
  "fragments": [ "1fbfae53…", "ad9a27a8…", "8668eeed…" ] }

Prefill skipped

300 / 300 tokens served from cache

Reads on 4 cores vs 1: 3.2×; Free-threaded Python 3.14, one node
Read capacity, 3 → 5 nodes: 1.6×; Linear is 1.67×; fixed CPU per pod
Test coverage: 92.8%; 1,700+ tests; no module under 70%
Authenticated by default: mTLS; Or API keys; required off localhost

How it works

Prefill is the expensive part. Do it once.

Most serving fleets recompute the same system prompts, documents, and chat histories on every node. Membrane turns that work into a shared, addressable cache.

0–127128–255…
1fbfae53…ad9a27a8…8668eeed…
01
Prefill once
A node runs prefill and cuts the KV-cache into token windows. Each window becomes a fragment addressed by its hash: model, layers, heads, tokens, dtype, and bytes.
02
Share across the fleet
Fragments are placed on a consistent-hash ring and replicated with their bytes. Gossip tells every node where they live; identical content is stored once.
Cached promptcoverage 1.0
New suffixprefill 44 tokens
03
Reconstruct, don’t recompute
When a prompt arrives, Membrane rebuilds its context from cached fragments and prefills only the gaps. Full coverage means no prefill at all.

Features

Everything a shared KV‑cache needs in production.

Storage, lookup, routing, and cluster primitives, with the security and operations work already done.

Content-addressed fragments

KV segments keyed by a ten-field identity hash. Identical content is stored once; each tenant keeps its own copy of the metadata.

Four indices, one lookup

Exact, semantic, positional, and co-access indices over the same fragments, behind a single Index API.

Reconstruction engine

Rebuilds a context from cached fragments and prefills only the uncovered spans, never the whole prompt.

Cache-aware routing

Economic, latency, and joint routers send each request to the node that already holds its context.

A cluster that heals

Consistent-hash ring, gossip membership, failure detection, byte-verified replication, and quorum writes.

Secure by default

API keys or mTLS on every route, per-tenant reads, AES-256-GCM at rest, and a tamper-evident audit log.

Bring your engine

Adapters for vLLM, SGLang, and TensorRT-LLM; backends for Transformers, OpenAI, Anthropic, and Ollama.

Operable

Prometheus metrics, OpenTelemetry traces, a live dashboard, and admin commands to inspect and repair.

Architecture

Five planes, one runtime.

The same Python runtime runs as a single node or a gossip-meshed fleet: same code, same semantics, same wire format. Compute, storage, persistence, eviction, placement, auth, and secrets are plugins you can replace without forking.

Read the architecture

01
Client surface
How engines and operators talk to a node.
- Python SDK (sync + async)
- HTTP + mTLS
- OpenAPI 3
- CLI and dashboard
02
Routing
Which node serves a request.
- Economic router
- Latency router
- Joint optimizer
- Workload analyzer
03
Index
Finding the fragments that cover a prompt.
- Exact
- Semantic
- Positional
- Co-access
04
Cluster fabric
Where fragments live, and keeping them there.
- Consistent-hash ring
- Gossip membership
- Byte-verified replication
- Failure detection
05
Storage
The bytes, in memory and on disk.
- In-memory store
- Warm tier on disk
- Redis (Sentinel, Cluster)
- AES-256-GCM at rest

Performance

Scales with cores. Scales with nodes.

Measured numbers come from this repository's benchmarks and CI jobs. The modeled numbers reproduce the Prefill-as-a-Service paper's case study, and are labelled as such.

Measured

Benchmarks and CI

8.1Mreads/s
One node, 4 threads: 3.2× one thread on free-threaded Python 3.14. CI requires ≥ 3.0× on 8+ CPU hosts.; 1.6×
Read capacity, 3 → 5 nodes: 1,848 → 2,955 reads/s on kind with 250m CPU per pod. CI fails below 1.5×.; 15.4kreq/s
HTTP /retrieve, one node: Two event loops on free-threaded Python, keep-alive; the load generator saturates beyond.

Modeled: the paper’s case study

scripts/demo.py

1.39×
Max sustainable request rate: 3.12 vs 2.25 req/s; 42%
Lower mean TTFT: 2.31 s vs 4.00 s; 10%
Lower P90 TTFT: 3.62 s vs 4.00 s

Max request rate (req/s)vs homogeneous PD

Homogeneous PD2.25

Naive heterogeneous PD2.50

Membrane-PD3.12

Eqs. (1)–(6) of the paper with a piecewise-linear Table-5 fit, run on its Section 4 workload. A model, not a benchmark of a live cluster. Source

Developer experience

From git clone to a running node in minutes.

Every snippet here is copied from the quickstart and runs in CI.

One CLI
Serve, query, inspect, evict, and repair from the same membrane command.
Typed Python client
Sync and async clients, plus an OpenAPI 3 schema for every route.
Secure container
Python 3.14, non-root, read-only root filesystem, drains on SIGTERM.

git clone https://github.com/sachncs/membrane.git
cd membrane
uv sync --frozen --extra server   # Python 3.14 + locked deps
source .venv/bin/activate

$ membrane serve --daemon &
$ membrane client prefill --prompt-tokens "1 2 3 4 5 6 7 8"
$ membrane client inventory
$ membrane client retrieve --hash <payload_hash from inventory>

import logging

from membrane.client import MembraneClient

logging.basicConfig(level=logging.INFO)
log = logging.getLogger("quickstart")

client = MembraneClient("http://localhost:8080")
result = client.prefill(list(range(256)), model_id="llama-3")
for frag in result["fragments"]:
    content_hash = frag["identity"]["payload_hash"]
    log.info("%s found=%s", content_hash, client.retrieve(content_hash)["found"])
client.close()

# Any non-loopback bind requires authentication.
# Keys are stored hashed; line 1 is the key, line 2 the keyfile entry.
$ membrane keys generate --subject ops --scope admin > ops.txt
$ sed -n 2p ops.txt > api-keys && chmod 600 api-keys
$ membrane serve --host 0.0.0.0 --api-key-file api-keys --daemon

# Or as a container (Python 3.14, read-only root, non-root user):
$ docker build -t membrane .
$ docker run --read-only --tmpfs /tmp -p 8080:8080 \
    -v "$PWD/api-keys:/run/secrets/api-keys:ro" \
    -e MEMBRANE_API_KEY_FILE=/run/secrets/api-keys membrane

# Three local nodes; each seeds from the other two.
$ membrane serve -n n1 -p 8080 --daemon --peer localhost:8081 --peer localhost:8082 &
$ membrane serve -n n2 -p 8081 --daemon --peer localhost:8080 --peer localhost:8082 &
$ membrane serve -n n3 -p 8082 --daemon --peer localhost:8080 --peer localhost:8081 &
$ membrane cluster-status --port 8080

Built for

Wherever prefill is the bottleneck.

LLM serving teams

Cut time-to-first-token without re-architecting inference.

Run Membrane next to vLLM, SGLang, or TensorRT-LLM through the bundled adapters. Shared system prompts and documents come back from cache instead of being prefilled again.

Multi-tenant platforms

Share the cache, not the data.

Every tenant keeps its own copy of a fragment and can only read its own. A cross-tenant read is indistinguishable from a miss, and identical bytes are still stored once.

Research and infrastructure

A faithful reference of the paper.

The analytical throughput model, a Table-5 fit, and an end-to-end simulator. Reproduce, stress, and extend Prefill-as-a-Service on real code.

Run a node in five minutes.

Python 3.14, locked dependencies, no account. The quickstart takes you from clone to a cache hit.

Read the quickstart Star on GitHub

install from source

git clone https://github.com/sachncs/membrane.git
cd membrane
uv sync --frozen --extra server   # Python 3.14 + locked deps
source .venv/bin/activate