Index a collection · Topic Map

Indexing is the step that costs money: each passage is sent to a model once, and each card is embedded for search.

Before you start#

Model Hub is connected and has a provider.
The collection exists: see Make a collection.
For books, Calibre's Content Server is running. Settings → Topic Map → Setup shows whether it is.

Steps#

Run Topic Map: Collection: Index… and pick the collection. Or open the collection on the Collections tab and select Index….
Read the dry run: the documents and passages, any book without passages and why, what the run would read and embed, and the estimated cost. When the reading task's model runs on a subscription, the price says the subscription covers it and gives what the calls would cost at API prices; on a model on this computer, it says there's no charge. When reading and embedding are billed differently, each gets its own line, followed by what would actually be charged. No model has been called and nothing is written. How this run works, under the fold, says what is read again and why, where the index lives, and opens Preview passages….
Choose Cheaper, Balanced or Better for this run. Cheaper asks OpenRouter for its lowest-priced provider; Better reads passages one tier up and asks OpenRouter for its fastest. The provider sort acts only through OpenRouter: when the reading task's models don't go through it, Better's tier is what changes, and the dialog says so. Nothing in Models changes.
Optionally set a spend limit. The run stops when it reaches it. It counts only money that's charged: calls covered by a subscription, or run on this computer, don't count toward it. Model Hub's own limits and budgets apply as well.
Select Index. When the run ends, a short summary says what failed, what was read and embedded and what it cost; the full report is under Full report.

The Index dialog before a run, with How this run works open and Preview passages… under it

The Index dialog when the run has ended: the summary and one Done button

The job shows its progress and its cost so far. It warns when it has made no progress for five minutes. Cancel stops it; so does the panel’s Runs tab.

What a later run does#

A run picks up where the last one stopped.
A passage is read again only when what would be sent for it has changed: its text, or the lens.
A book is split into passages once and they are kept. Notes are split again on each run, so new and changed notes are picked up.
A card is embedded again only when its text changes, or when the embedding model or the lens's search instruction does. Vectors are kept in a file named for the model and the instruction, so a change starts a new file rather than mixing vectors that can't be compared. Search then says the old vectors were made otherwise until you index again. An index made before this naming keeps its files and works as before.

If a book needs converting#

A book that has only an AZW3 or MOBI file needs an EPUB. By default Library Hub asks before it converts, and the conversion adds an EPUB to the book in your Calibre library. Settings → Topic Map → Setup → Convert AZW3 and MOBI books set to Never skips those books instead; the dry run names them.

Keep a collection up to date#

In the collection's dialog, Keep up to date indexes what changes on its own: notes added, changed or removed in the folder or with the tag, a Base's rows, or a shelf whose members changed in Calibre (compared every half hour). It is off until you turn it on, since it spends money unasked. Settings → Topic Map → Setup → Keep up to date spends up to caps what all automatic runs may spend in a day; at the cap they pause until tomorrow. Automatic runs never ask to convert a book, and show in Jobs with their cost like any run.

Where the index is kept#

In Topic Map's plugin folder, inside your vault's configuration folder: .obsidian/plugins/topic-map/corpora/. It holds the passages, cards, vectors and run logs. The Collections tab says how much each collection's index takes, on its row and in its dialog.

Uninstalling Topic Map deletes every index, including the reading and vectors you paid for. Copy the corpora folder out first if you want to keep them. Your notes stay.