Insurgently

Technical details

Topic Map reads your books and notes into cards, finds the topics they share and draws them as a map. This is how it's built.

Its models run through Model Hub, and it reads your books through Library Hub, which asks it in turn which books belong together. Everything below is in the code, and anything still to come is marked Soon or Later.

All the pagesAgent docs →

The shape of it

A collection names a source and a lens. The lens cuts the source into passages, a model reads each passage into cards, and the cards become vectors. Topics, the map, search and write-ups all work from those cards.

  1. Sources

    Books through Library Hub, a shelf, a folder, a tag or a Base

  2. cut

    Lens → passages

    A lens says how to cut a source and what to read from it: nonfiction, fiction and notes, and journals, saved chats and web clips.

  3. read

    Cards

    What a passage says: a concept, a claim, a case or a practice, with a quote that anchors it.

  4. embed

    Vectors

    One float16 row per card, scored in a worker by a small Rust kernel compiled to WebAssembly

  5. use

    Topics · The map · Ask

    Themes and topics, named for you. Themes, Network and Words. Passages back, and write-ups with checked quotes.

Written down, in your vault

  • Topic Map index/: every card, in .jsonl files under 4 MB. This is the work you paid for.
  • Topic Map/Topics: a note per theme
  • Topic Map/Drafts: write-ups

Vectors, neighbors and the map's layout are a cache in the plugin's folder, rebuilt from the cards when needed.

Indexing, step by step

What happens between adding a collection and seeing its topics.

  1. Price it. A dry run counts the passages and prices the run through Model Hub, and writes nothing.
  2. Cut. A book is cut by its table of contents, by headings, or by a model where neither works. The nonfiction lens keeps passages under 3,000 words. Book text comes from Library Hub; Topic Map never reads Calibre itself.
  3. Read. The topic-map.extract role (quick) reads each passage into cards. Each card's anchor is a quote from its passage.
  4. Embed. topic-map.embed turns cards into vectors through Model Hub's embedding role, as passages, and questions later as queries. They're made on this computer only when you pick a local provider for that role. Vector files are named by the embedding's fingerprint, so a model change can't mix rows.
  5. Keep up. A run picks up where it stopped, and only changed passages are read again. Shelves are checked every 30 minutes; notes and Bases follow the vault's own events.

Memory is given back. The vector worker stops after 5 idle minutes, never mid-job, and sooner when macOS reports memory pressure.

Finding topics

Topics are found in two levels, broad themes and the topics inside them, by spherical k-means on the card vectors.

Clean-up
The 3 strongest shared directions are removed first. Without that, unrelated cards look alike: the closest pair of themes dropped from 0.72 to 0.54 similarity, and loose clusters from 15 to 6.
How many
A starting count from the number of cards, then a sweep that keeps the count whose topics stay the same from run to run. You can ask for fewer or more.
Outliers
Cards far from every topic are set aside, not forced in.
Same answer
Clustering runs in the Rust kernel, seeded, so the same cards give the same topics on any machine. Topic ids carry over from run to run.
Summaries
Each theme and topic gets a short summary, written in the background on a cheap tier.
Names
topic-map.name (balanced) names each topic, and names each theme again with its siblings in view, so names don't overlap. Names are kept and reused.

The map

The map is drawn with cosmos.gl, which puts points and links on the GPU through WebGL, with labels on a canvas above it. It's bundled with the plugin.

Themes
Themes packed as groups, their topics inside.
Network
Cards or topics linked to their nearest neighbors: the top 30 per card, found exactly and cached.
Words
Topic names placed by size. It needs no WebGL.
Layout
d3-force, seeded, in its own worker, so the map is the same each time it opens.
Properties
The map and its charts can also be laid out by a Base's columns or your notes' properties.
Charts
A Base per collection, with sorted tables of its topics.

Answers and write-ups

Search

A question is embedded and matched against the cards. Hybrid search adds BM25 over three fields, exact names and close spellings of titles, merged with the vector results by reciprocal rank fusion. It's off by default, because it didn't do better on both test sets.

Write-ups

topic-map.write (careful) drafts from the cards and passages it's given. Every quote in a draft must be found in a card's anchor or one of those passages, allowing for spacing, case, curly quotes and dashes. A quote that isn't found is flagged or dropped. Other plugins get the same check through checkQuotes.

Tools and the API

Topic Map registers its model jobs, its tools and its lenses with Model Hub, and serves Library Hub through its own API.

Roles
7: extract, split, name, survey, write, embed and judge, on quick to careful tiers you can change.
Tools
6, all reads: list collections, search, follow a card to its neighbors, get cards, read a passage, open the map. Coding agents reach them through Model Hub's MCP server; Topic Map runs no server of its own.
Skills
The three lenses are skills, so you can copy one into your vault and change it.
For Library Hub
similar, rankGroups, place, groupHealth, overlaps and findGroups: similar books, the shelves that suit a book, where new books go, a shelf's health and overlaps. topic-map:index-changed says when to ask again.
// wanted: the API version your plugin was built for
const found = getTopicMap(this.app, wanted);
if (!found.topicMap) return new Notice(found.message);
const hits = await found.topicMap.search('Walden', 'solitude and work', { k: 10 });

Shared libraries

The same packages the rest of the family uses, published under MIT.

obsidian-ui
The family design system: the panels, dialogs and settings are built from its named controls.
retrieval
BM25 and rank fusion, for hybrid search.
text
Finds a card's anchor in its passage after the text moves.
Library releases ↓

How it's tested

Unit and end-to-end suites, benchmarks that fail on a slowdown, and a set of cards that say what a person is trying to do.

Unit
About 3,000 Vitest tests across 269 files. Two boundaries are tested too: the map's code never imports Obsidian, and the vector code imports nothing else from the plugin.
End to end
WebdriverIO specs against a real, sandboxed Obsidian, with real builds of Model Hub and Library Hub beside it. They run in Docker, so no window opens on anyone's screen. The map's screenshots come from the same runs, with WebGL drawn in software, in light and dark.
Benchmarks
11, at real scale (12,862 cards). A run 30% slower than the baseline fails.
Evals
8, for the choices that rest on quality: cluster quality, search, tuning and shelf groups.
Journeys
9 jobs, 17 journeys and 159 scenario cards.

The pages

Each part above has its own section with the detail. Developer docs are for people building on Topic Map; Agent docs are the same knowledge for coding agents.

Technical roadmap

Topic Map came out with the other two plugins, since the three call each other. Soon is the next release; Later is after it. Nothing has a date.

Soon

  • Gaps and bridges. Where two close themes have few cards linking them, and the cards that join them.
  • Answers as notes. Write-ups with numbered sources, saved back into the vault.
  • Search inside a topic. Retrieval scoped to one topic's cards.
  • Incremental topics. New cards join the topics they fit, with a drift check that suggests finding topics again.
  • Evidence labels. Each claim marked established, emerging or hypothesis, with the passage behind it.

Later

  • Entities. People, places and terms recognized across a collection.
  • Link-aware topics. Links and tags in your notes weigh into the clustering.
  • A wiki from your cards. Pages compiled from topics and themes, keeping your edits when the cards change.
  • Map v2. Each book's footprint, paths between topics, run-to-run changes, and saved views.
  • A reader. A book with its cards and topics in the margin.

Library releases

The shared packages Topic Map is built on, newest first. Each release has its own page, and the same notes go on GitHub.

  • @insurgently/text 0.6.2anchorsIn: find many spans of one text in a single pass.
  • @insurgently/retrieval 0.1.0The first version: tokenizing, BM25 search and rank fusion.

Getting involved

The source is public, under MIT. CONTRIBUTING.md lists the checks a pull request passes: lint, typecheck, unit tests, and end to end in Docker for anything that touches Obsidian.

It stands on other people's open-source work: cosmos.gl for drawing the map, d3-force for its layout, and wdio-obsidian-service for testing in a real Obsidian. What I've built for the family goes back out as the shared libraries above.

Get involved, for the whole family →

Join

One list for everything here: an email when a plugin ships, plus the posts I write about these projects on brianspringer.com.

Want to help build this, or join the co‑op? Email me about joining →

Insurgent Methods is where we build at a different scale, to take on the biggest problems.

API package
@insurgently/topic-map-api, on npm
Needs
Obsidian 1.13.1, desktop
License
MIT