How big a collection can be

Topic Map keeps a collection's index in its plugin folder and works on it inside Obsidian. Up to a point that is quick; past it, some things get slow, and past a further point a collection is too large to open for search.

What Topic Map tells you#

• Before you index: the Index dialog works out how big the collection will be once indexed, from its dry run. If that is large, it says what will be slow and about how slow, for example "Find topics: about 6 min" or "Index: about 400 MB on disk, in Topic Map’s plugin folder, which vault sync uploads".
• On a collection: its dialog in Settings → Topic Map → Collections says the same from the index as it is.
• Find topics' memory: while it runs, Find topics holds the card vectors twice more (they and a decorrelated copy). When that is more than the computer can spare, the dialogs say so. Search is unaffected, and Find topics still runs.
• When the computer is short of memory: Topic Map lets go of what it has loaded, the same as after five idle minutes: it stops its vector worker and forgets the open indexes, which load again from disk the next time you search. It checks every 30 seconds, and only while something is loaded. It never does this while a job is running; the job finishes, and its entry in the jobs view says memory got tight.
• Too large to search: when opening the index would take more memory than the computer can spare for the card vectors (a share of its RAM, so it depends on the computer), or more than the fixed ceiling Obsidian has for the passages and cards, Search and Find topics won't open it, and say so. Indexing still works.

The numbers behind these come from measurements on a laptop, in docs/limits.md in the plugin's repository.

Suggest a split…#

Suggest a split… (in the Index dialog, a collection's dialog, or the command palette) looks for a seam the collection already has:

• a notes folder: its subfolders;
• a tag: its nested tags (#topic/a, #topic/b);
• a Base: a property with two to six values; each part becomes a new view of the same Base, filtered on its value;
• books: a book field, such as role.

It shows what it would write, then writes one collection note per part beside the original, with the same lens and context. Each part's index starts with everything already read for its documents, so indexing a part reads only what is new. Notes that no part can hold (notes at the top of the folder, notes with the bare tag) are named and stay in the original. The original collection is not changed; delete it yourself once the parts are indexed. Undo a run takes the whole split back.