v3.0.0 — Now shipping encrypted at-rest & mTLS

Global contextual memory for LLM serving.

Membrane separates the KV-cache from GPU memory and turns it into a distributed, content-addressed fabric — sharing prefill work across clusters, slashing time-to-first-token, and reclaiming GPU capacity at scale.

Start building See the architecture

MIT licensed Python 3.10 – 3.13 CPU · GPU · OpenAI · Anthropic · Ollama

membrane · cluster n1 ↔ n2 ↔ n3

live

Fragment hit-rate +18%

94.7 %

P50 TTFT ↓ 41%

82 ms

Throughput +2.3×

3.4 k tok/s

Peers · alive healthy

7 / 7

$

 membrane serve --node-id n1 --port 8080 --transport http --peer n2:8081 --peer n3:8082

Reference implementation of Prefill-as-a-Service · production-grade, paper-faithful.

Paper-faithful

Eqs. (1)–(6) verbatim

Content-addressed

SHA-256 fragment IDs

Multi-tenant

Per-tenant isolation

Pluggable backends

CPU · GPU · 4 LLM APIs

Battle-tested

548+ tests · chaos suite

The product

One fabric. Every request.
Zero wasted prefill.

Membrane is the runtime that lets LLM serving fleets reuse prefill work across nodes, regions, and tenants — without re-engineering your inference stack.

Decouple cache from GPU

Treat KV-cache as a first-class, distributed resource. Memory is content-addressed; compute is pluggable. GPUs become stateless workers, not pinned caches.

Share prefill work

Reconstruct contexts from fragments held anywhere in the cluster. When coverage is incomplete, Membrane falls back to a single prefill — never a re-decode.

Route with the math

Built on the analytical throughput model from the paper. Economic, latency, and joint routers pick the best node from access history and live telemetry.

Capabilities

Everything a memory fabric
needs to scale.

Production-grade primitives, a coherent runtime, and the operational surface to run them confidently at scale.

memory

Content-addressed fragments

KV segments keyed by a ten-field PayloadIdentity hash. Two fragments with the same hash are byte-identical — instantly deduplicated across tenants and regions.

retrieval

Four indices, one facade

Exact, semantic, positional, and co-access indices over the same fragment set. Sub-millisecond lookups, exposed through a single Index class.

engine

Reconstruction engine

Walks the indices to rebuild a context. Falls back to a single prefill only when coverage is incomplete — never wasteful re-decoding.

routing

Analytical routing

Economic, latency, and joint routers pick the best node from access history and live telemetry. Backed by the paper's Eqs. (1)–(6).

cluster

Cluster as one

Consistent-hash ring, gossip state exchange, heartbeat-driven failure detection, and background replication — all in-process.

security

Encrypted at rest, mTLS in flight

Per-fragment payload encryption, deny-by-default mTLS, per-route scope checks, and a typed cluster error hierarchy.

compute

Pluggable compute

CPU reference, GPU (PyTorch CUDA with CPU fallback), HuggingFace Transformers, plus native OpenAI, Anthropic, and Ollama adapters.

ops

TUI dashboard & CLI

A Rich-powered live dashboard, an interactive setup wizard, and a complete admin surface (snapshot, restore, rotate-keys).

Architecture

Five planes, one runtime.

A coherent Python runtime that you can run as a single node or a gossip-meshed fleet — same code, same semantics, same wire format.

01 Client surface

Python SDK
FastAPI · gRPC · stdlib HTTP
OpenAPI v3
CLI · TUI

02 Routing plane

Economic router
Latency router
Joint optimizer
Workload analyzer

03 Index facade

Exact · Semantic · Positional · Co-access
AVL-backed overlap
Bounded-depth BFS

04 Cluster fabric

Consistent hash ring
Gossip membership
Background replication
Failure detection

05 Storage

In-memory shards
Redis durability
LRU + TTL
Encrypted blobs

Performance

Numbers from the
paper, reproduced.

The analytical model in model/throughput_model.py reproduces Eqs. (1)–(6) from the Prefill-as-a-Service paper with a piecewise-linear Table-5 fit — verified end-to-end against the simulator.

Read the paper Run the demos

0 %

Lower P50 TTFT

Median time-to-first-token across the dual-timescale scheduler.

0 ×

Throughput gain

End-to-end tokens/sec vs. single-node prefill baseline.

0 %

Fragment hit-rate

Reconstruction served from fragments — no prefill required.

0 ms

Index lookup

Worst-case across exact, semantic, positional, co-access.

Aggregate tokens/sec · request rate sweep

dual-timescale scheduler vs. single-node prefill

Membrane Baseline

50 rps 200 rps 400 rps 800 rps 1.6k rps

Developer experience

Five minutes from pip install to a live cluster.

A real Python SDK with strict types, a CLI that respects your time, and an OpenAPI surface you can drop into any client generator.

from membrane.server import Server
from membrane.compute import CPU

server = Server(
    node_id=class="text-accent-mint">"n1",
    host=class="text-accent-mint">"0.0.0.0",
    port=8080,
    transport=class="text-accent-mint">"http",
    compute=CPU(),
    peers=[class="text-accent-mint">"n2:8081", class="text-accent-mint">"n3:8082"],
    replica_count=2,
)

server.run()

from membrane.fragment import Fragment
from membrane.identity import PayloadIdentity

identity = PayloadIdentity(
    payload_hash=class="text-accent-mint">"sha256:…",
    model_id=class="text-accent-mint">"llama-3",
    layer_range=(0, 32),
    token_span=(0, 4096),
    dtype=class="text-accent-mint">"float16",
)

frag = Fragment(
    identity=identity,
    payload_ref=class="text-accent-mint">"blob-key",
    payload_size=8 * 1024 * 1024,
    ttl=3600.0,
)

from membrane.analytical import Joint
from membrane.telemetry import live

router = Joint()
target = router.select(
    payload_hash=identity.payload_hash,
    candidates=cluster.peers(),
    telemetry=live(),
)

target.retrieve(identity)

$ membrane serve --node-id n1 \
        --port 8080 \
        --transport grpc \
        --compute gpu \
        --peer n2:8081 --peer n3:8082

$ membrane dashboard --host n1 --port 8080
$ membrane cluster-status
$ membrane admin rotate-keys

Strict types throughout · mypy strict · Google-style docstrings · zero semi-private names.

Built for

Wherever prefill is the bottleneck.

LLM serving teams

Cut TTFT without re-architecting inference.

Drop Membrane alongside vLLM, TGI, or any custom stack. Cache fragments at the cluster edge, reconstruct on the hot path, fall back to a single prefill when coverage is incomplete.

Multi-tenant platforms

Share prefill safely across customers.

Content-addressed dedup plus per-tenant policies means one canonical store, with hard isolation. Tenants inherit the hit-rate without seeing each other's data.

Research & infra

A faithful reference of the paper.

Six analytical equations, a piecewise-linear Table-5 fit, and an end-to-end simulator. Reproduce, stress, and extend the Prefill-as-a-Service model.

Ready when you are

Ship a memory fabric
without writing one.

pip install membrane — and you have a single-node runtime in seconds. Add --peer flags to grow into a cluster.

Get started Star on GitHub

install

$ pip install membrane