Skip to content
v0.1 · Open source · MIT

Documentation,
structured.

knowledge is a Python SDK that splits any URL or file by heading, sends each section to a large language model, and writes an OKF v0.1 bundle of linked Markdown — versionable, diffable, and ready to serve.

Install knowledge pip Star on GitHub —
$ pip install  
OKF v0.1
Pipeline3 stages
1source.pyguide.html
2split by heading
3extract via LLM
$ terminalbash
$ knowledge create https://google.github.io/styleguide/pyguide.html ./pyguide
▸ fetching…
▸ splitting… 6 sections
▸ extracting concepts via gpt-4o…
✓ wrote 6 concept files
Output bundlepyguide/
pyguide
  • index.mdroot
  • naming-conventions.mdconcept
  • imports.mdconcept
  • comments.mdconcept
  • type-annotations.mdconcept
  • error-handling.mdconcept
6
concepts
0
broken links
100%
diffable

Powered by trusted, proven technology

Pydantic
litellm
OpenAI
Anthropic
Gemini
Ollama
What's inside

Engineered for the long tail of documentation work.

Six composable capabilities. No vendor lock-in. No magic. Just a small, sharp tool that does one thing well.

LLM-powered extraction

Route any provider litellm supports — OpenAI, Anthropic, Gemini, Bedrock, Azure, vLLM, Ollama — through a single interface. Swap models with one flag.

OKF v0.1 bundles

Output is a real directory: index.md, per-concept Markdown, YAML frontmatter, tag subfolders. Diff, link, search — like documentation should.

Section-aware splitting

Native handling of HTML (<h2>–<h4>) and Markdown (##) headings. One concept per section, ordered and resolvable.

Resilient by default

Exponential backoff retries, 50 MiB body cap, charset detection, HTTP error classification — production hygiene built in.

CLI + Python API

Use the `knowledge` CLI for ad-hoc work, or import the SDK directly. Immutable Pydantic models, predictable behaviour.

Validated output

Structural checks for link resolution and orphan detection. Curate bundles by removing concepts without rewriting the whole graph.

How it works

From a URL to a structured knowledge graph in four steps.

No schema to maintain. No infra to deploy. The output is plain files you can read, version, and serve — exactly what good docs should be.

  1. 01

    Point at any source

    Pass a URL or a local HTML/Markdown file. Resilient HTTP client handles redirects, retries, and size limits.

    URL HTML Markdown 50 MiB cap
  2. 02

    Split by heading

    Each <h2>–<h4> (or ##) section becomes a unit. One concept per section — never more, never less.

    HTML headings MD headings Ordered
  3. 03

    Extract via LLM

    litellm routes the call to any provider. Each section becomes a structured Pydantic Concept.

    OpenAI Anthropic Ollama Bedrock
  4. 04

    Write OKF bundle

    A real directory on disk: index.md, per-concept files, YAML frontmatter, tag subfolders. Validated, diffable, ready.

    index.md Concept.md Tags/ Validated
knowledge graph · live
OKF v0.1
index.md
naming
imports
errors
style/
The interface

Three ways in. One sharp tool.

Use the CLI for ad-hoc work, the Python API for production pipelines, or read the output bundle directly — it's plain Markdown. 

# Create a bundle from any documentation URL$ knowledge create \    https://google.github.io/styleguide/pyguide.html \    ./pyguide▸ fetching…                     1.2 MB, 200 OK▸ splitting by <h2>–<h4>       6 sections▸ extracting via gpt-4o…       6 concepts✓ wrote 6 concept files✓ index.md linked to all concepts# Re-extract after the source changes$ knowledge update \    https://google.github.io/styleguide/pyguide.html \    ./pyguide# Trim a bundle by removing specific concepts$ knowledge remove deprecated-section \    outdated-topic ./pyguide/

Ready in 30 seconds.

Requires Python 3.12+. Works on macOS, Linux, and Windows.

Read the install guide
100%
Test coverage

65+ tests across SDK, CLI, and serialization layers.

Any
Provider support

Routes through litellm — OpenAI, Anthropic, Gemini, Bedrock, Ollama, vLLM, and more.

OKF v0.1
Bundle format

Standardized, directory-based, and diffable in Git.

2
Runtime deps

Just pydantic and litellm. No transitive bloat.

Use cases

One tool. Many surfaces.

Anywhere documentation lives, knowledge makes it structured.

RAG

Documentation → RAG corpora

Convert any docs site into a clean, linked corpus you can drop into a vector store — without scraping artefacts.

Docs

Style guides & references

Pull an HTML reference into per-section concept files. Diff them in pull requests. Review like real docs.

Content

Blog → knowledge base

Turn a long-form blog or knowledge dump into a navigable, searchable bundle. One concept per heading.

API

API doc synthesis

Ingest public API references and get back a normalized OKF bundle — ready for a static-site generator.

Enterprise

Internal handbook archival

Convert handbook pages into concept files so they're diffable, reviewable, and auditable over time.

Research

Research paper summarization

Extract structured concept summaries from long papers or reports. Each section becomes its own file.

FAQ

Questions, answered.

Which LLM providers are supported?
Any provider that litellm supports: OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, vLLM, Ollama, and more. Swap models with the --model flag or the model= constructor argument.
Is my data sent anywhere?
Only the document section text goes to the LLM provider you choose. If you point at a local model via Ollama or vLLM, nothing leaves your machine.
What is OKF v0.1?
Open Knowledge Format v0.1 is a directory-based format: an index.md, per-concept Markdown files, YAML frontmatter, and tag-based subdirectories. Designed to be readable by both humans and tools.
Can I update an existing bundle?
Yes. knowledge.update() re-extracts from the source and overwrites the bundle in place. It's destructive — manual edits will be lost — so use it like a build step.
Does it work offline?
The HTTP fetcher needs network access, but you can also point at a local HTML or Markdown file. Pair that with a local LLM via Ollama for a fully offline workflow.
How does it handle large documents?
Each request is capped at 50 MiB by default. Retries use exponential backoff. Sections are extracted independently, so a single failure won't lose the whole bundle.
Get started in seconds

Turn documentation into infrastructure.

One pip install. One CLI command. A directory of structured concepts — version-controlled, diffable, and ready to serve.

View on GitHub

MIT licensed · Python 3.12+ · Zero config