Catalogs · Topic Map · insurgently

A catalog takes one domain of a collection, a facet of its lens such as magic, characters or objects, and sorts its cards into families and types. A type is a group of cards that describe the same kind of thing. A family is a group of types. For each type, the catalog lists every source's version of it: what that version does, and where it appears, with a link to the passage.

Catalogs are written from the command line for now, from a copy of the Topic Map repository; the plugin draws a tree catalog's cards and renames its nodes. The collection must be indexed first.

Steps#

Run a dry run for one domain:
```
npm run catalog -- --vault <vault folder> --collection "Collections/Tales.md" --domain magic --dry-run
```
It clusters the cards and prints the notes it would write. No model is called and nothing is written. It still needs the headless Model Hub build (--hub, or MODEL_HUB_HEADLESS), and it stops with the missing path if that isn't there.
Run it again without --dry-run. Families and types are named through Model Hub, stopped at --max-usd (one dollar unless you say otherwise).
Open the catalog note, Topic Map/Catalogs/Magic/Magic.md. It is the folder note of the domain's folder, as the Folder notes plugin names one, and its family notes sit beside it in Topic Map/Catalogs/Magic/.

Leave out --domain, or say --domain all, to catalog every facet that has cards.

What gets written#

The catalog note holds a table of families (with their types, cards, sources and rarity) and a count of types at each rarity. With a tree, the families are cards instead (see Cards).
A family note has a section per type: its rarity and a Versions table of Source, Variant, What this version does, and Where.

Cards that clustering leaves out are counted as unplaced. The flat layout doesn't write them anywhere; the tree layout puts them in an Unplaced note.

Rarity counts the sources a type appears in: unique (1), rare (2–3), common (4–9), pervasive (10 or more). Cards whose source isn't known are listed but not counted. A source name with no Latin letters or digits counts as unknown, so give it a source_aliases entry.

Where the files go#

Each domain gets a folder under --folder (Topic Map/Catalogs unless you say otherwise), named after the domain. The catalog note is that folder's folder note, so the folder holds:

Magic.md, the catalog note;
one note per family, and the Unplaced note of the tree layout;
Magic.packet.md, Magic.packet.json and Magic.tree.yml, once you make a packet and a tree.

The catalog note and the packet are never taken for family notes, and a family is never named over them: a family called Magic gets Magic 2.md.

Catalogs made before this layout kept the catalog note beside the folder, as Topic Map/Catalogs/Magic.md. A run that finds a note there, and none at Topic Map/Catalogs/Magic/Magic.md, stops and says so. Move the note into Magic/ as its folder note, move its Magic.tree.yml, Magic.packet.json and Magic.packet.md there too if you have them, and run again. Its family notes are already in the right folder. Nothing is moved for you.

The catalog note's settings#

The first run makes the catalog note from the flags. After that, its frontmatter is what counts:

Key	What it does
`domain`	The facet it catalogs.
`collections`	The collection notes it gathers from. Collections listed together must be embedded with the same model.
`source_field`	The book field that names a card's source; `series` unless you say otherwise.
`source_aliases`	A source name and the one it counts as, so a work split in two counts once. On the first run, `--alias "from=to"` fills it. Catalog notes used to call it `aliases`, which is Obsidian's own property for a note's other names: an old `aliases` map is still read, and the next run renames it `source_aliases`. A note with both is refused until you delete one. An `aliases` list is Obsidian's and is left alone.
`folder`	Where its family notes go: the catalog note's own folder unless you say otherwise.

Editing the notes#

You can write in the catalog note and the family notes. A run replaces only the parts between  and  markers and keeps everything else. If a note's markers are gone, that note isn't written, and the run says so.

An edited family note keeps its old rarity and sources frontmatter for now, even when a run changes the numbers. Notes you haven't edited are rewritten whole, so they stay right.

Each run is logged, and Undo a run puts the notes back as they were.

Organize with a tree#

The flat layout is two tiers deep, and its ids come from clustering, so they can change from run to run. A tree gives the domain four tiers and ids that stay put: family → sub-family → type → variant, each with its own id, and each source's version as a row under its variant (see Variants). A strong model drafts the tree from a packet; Topic Map checks it and writes it.

Make the packet:
```
npm run catalog -- packet --vault <vault folder> --collection "Collections/Tales.md" --domain magic
```
It writes Magic.packet.md and Magic.packet.json beside the catalog note, in Topic Map/Catalogs/Magic/: every clustered type with its description, size and sources, and every version as one line under a key. No model is called. The types keep the names the catalog notes already give them, or get placeholder names; add --name to name them through Model Hub (stopped at --max-usd). The packet files are remade each time, so they aren't part of the run log: Undo leaves them, and there's no dry run.
Give Magic.packet.md to a strong model and ask it for the tree file in the shape below, with every take entry on its own line and in double quotes (version keys can hold characters YAML would otherwise read differently, such as : or a leading [). Save it as Magic.tree.yml beside the catalog note, in the same folder. Each family, sub-family and type has an id, a name and an about; each type lists what it takes. The file starts with the domain and the packet's hash, which the packet subcommand prints:
```
domain: magic
packet: 4603d2399eb22922
families:
  - id: mg.f1
    name: Shape-changing
    about: Magic that changes a body's form.
    subfamilies:
      - id: mg.s1.1
        name: Into beasts
        about: A person becomes an animal, or the reverse.
        types:
          - id: mg.t1.1.1
            name: Swan maidens
            about: A bird's skin taken off and hidden.
            take:
              - "f012"
              - "f031/river-songs/the swan cloak"
```
Ids are a two-letter key for the whole tree, then f, s or t and the numbers above it: mg.f1, mg.s1.1, mg.t1.1.1. In take, a packet type id takes all of that type's versions, and a version key takes just that one, even when another type takes its type id. JSON works too.
Check it, and see what would be written:
```
npm run catalog -- tree --vault <vault folder> --domain magic --dry-run
```
Every problem is printed, and nothing is written while there is one: an id of the wrong form or used twice, a missing name or about, a key that isn't in the packet, a version taken twice, a tree drafted from another packet. Versions no type takes are listed; they go in the Unplaced note.
Run it again without --dry-run to write it. Point at another file with --tree <vault path>.

The tree layout writes:

The catalog note, with its families as cards (see Cards).
A note per family, named after it, with its id in the frontmatter, a section per sub-family and, under it, a section per type with its Versions table. Each heading sits between  and  markers of its own, so it always carries the tree's name.
An Unplaced note, with the versions no type takes and the cards clustering left out.

Each type's table is keyed by its tree id, so renaming a type in the tree keeps its place, and renaming a family keeps its note, found by its id. Headings take the tree's names at every run, in a note you've edited too, as long as their markers are there; a note written before headings had markers keeps the headings you gave it.

A card is never left in two places without a word. When the tree drops a family, or you switch from the flat layout, the notes it no longer writes go to the vault's .trash folder if Topic Map wrote them and nobody edited them since (Undo a run puts them back). A note you edited is left alone and listed as stale, in the dry run's plan and in the summary, and so is an edited family note that still holds the table of a type the tree has moved elsewhere. Move what you want to keep, then delete the old note or table. While anything is stale, the run exits with code 2.

Only this domain's catalog notes are ever removed or listed: notes whose frontmatter names the domain and that hold a catalog table. Another domain's catalog sharing the folder, a topics note or a note of your own is left alone, and a family note is never named over one of them. If a domain's Unplaced note would land on another domain's, it isn't written, and the run lists it as a conflict: give the two domains folders of their own.

If the tree adds a type to a family whose note you've edited, the note has no table for it yet, so the run lists it as a conflict and doesn't write the note. Add the markers by hand under the heading you want, with the type's id from the tree, then run again:

<!-- catalog:begin mg.t3.2.4 -->
<!-- catalog:end mg.t3.2.4 -->

Once the tree file is there, the plain command writes the tree layout too. Neither it nor the tree subcommand calls a model: the tree names everything, so the clusters are gathered without naming. If the tree no longer fits that run's clusters (a re-index added or removed a card, or the clustering changed), the run prints the problems, writes nothing for that domain and exits with code 2. It never falls back to the flat layout, which would rewrite your tree's notes. Redraft the tree from itself (see Running it again), or move the tree file aside if you want the flat layout back.

Variants#

A type's versions are one row per source, named the way each source names it, or not at all. A long table of those names says little. Variants group them by how they work: under the type, each variant is a named mechanism, and its rows are the sources' versions of it, its instances. Collective casting, say, might hold a variant for a kin group casting as one body and another for minds merged into one will, whatever each book calls them.

The packet asks the model for variants too. They go under a type, beside its take:

            - id: mg.t1.1.1
              name: Collective casting
              about: Many casters work one spell.
              take:
                - "f012"
              variants:
                - id: mg.v1.1.1.1
                  name: Tribal
                  about: A kin group casts as one body.
                  take:
                    - "f012/lantern-road/circle song"
                    - "f012/salt-isles/clan chant"
                - id: mg.v1.1.1.2
                  name: Shared mind
                  about: Minds merge into one will.
                  take:
                    - "f012/gray-fen/hive thought"

A variant's id extends its type's numbers: mg.v1.1.1.2 is the second variant of mg.t1.1.1. Its take lists version keys only, each one of the versions its type takes, and a version goes in one variant at most. The check names a key from another type, a type id in a variant's take, a version in two variants, and a variant id of the wrong form or under the wrong type.

A version no variant takes isn't a problem. The check lists it, and the type's section shows it in an Other versions table after its variants. In the type's section, each variant gets a bold line with its name, id and about, its own rarity, and a table of Source, As named (the source's own name for it), What this version does, and Where. A type's card has a button for its variants, which shows them as cards. A type without variants is written as before.

Cards#

The catalog note's families part is a topic-map-catalog code block, which Topic Map draws as cards. The cards show one level at a time. At the top are the families, each with its color along the top, its icon, its name, its about, and its cards, sources and rarity. Select a family to see its sub-families, a sub-family to see its types, and a type's variants button to see its variants. Back, or a step in the trail above the cards, goes up again, and Open note opens the family note at the section you're in. Selecting a type opens its section in the family note. The Unplaced button under the families opens the Unplaced note.

Names, abouts, icons, colors and which node sits where are read from the tree file each time the cards are drawn, so an edit to the tree file shows at once. The counts come from the last run: a node the tree gained since then says it isn't counted yet. Without Topic Map, the block reads as an indented list of the same nodes, one a line, with their ids and counts.

Give a family an icon and a color in the tree file, beside its name:

  - id: mg.f1
    name: Shape-changing
    about: Magic that changes a body's form.
    icon: "🦢"
    color: "#3d85c6"

Both are optional. An icon is one emoji or up to four letters; a color is a #rrggbb hex color, in quotes, since YAML reads an unquoted # as the start of a comment. A family without a color takes one of the theme's colors. The check names the line of an icon or color it can't use.

Renaming#

The tree file's name is the one name each node has. Rename on a card (the pencil, when you point at it) asks for the new name, shows every file it would change or rename, and on Rename writes it everywhere at once:

the tree file, changing only that name, so your comments and layout stay;
the catalog note's block;
the node's heading in its family note, between its markers (a variant's bold line in its type's section);
for a family, the family note's family: frontmatter and its file name. The note is renamed through Obsidian, so links to it follow when Settings → Files and links → Automatically update internal links is on.

Nothing else in a note changes. A heading that would match one you wrote in the same note gets a number after it, so both still link. A rename is logged like a run: the notice after it has Undo, and Undo a run takes it back later. Editing the tree file's name by hand changes the cards at once, and the notes at the next run.

Moving#

Move on a sub-family or type card (the arrows, beside Rename) puts it under another parent: a type under another sub-family, in any family, or a sub-family under another family. Find the new parent in the list, which is grouped by family; Topic Map shows every file it would change, and writes them on Move:

the tree file, where the node's lines go to the end of the new parent's list, re-indented, with your comments and layout elsewhere kept;
the family notes, where the node's sections (its heading, about and Versions table, or a sub-family with all its types) move from the old note to the new one, inside their markers;
the catalog note's cards, with fresh counts for the old and new parents.

A move pins the node's place: it gets pinned: [parent] in the tree file (a rename adds pinned: [name]), so a redraft keeps it where you put it (see Pins). The node keeps its id, its versions and its variants. A sub-family or family left with nothing in it stays, and the plan says so. A heading that would match another in the new note gets a number after it. If a section's markers are gone, nothing is moved: put them back, or run the catalog again, then move. A move is logged like a run, so the notice after it has Undo, and Undo a run takes it back later.

Pins#

A pin records a choice you made by hand, so a redraft keeps it. Rename adds name to the node's pinned list and Move adds parent; you can write them yourself too:

          - id: mg.t1.1.2
            pinned: [name, parent]
            name: Ice lance

Any node can carry pinned: a family, a sub-family, a type or a variant. The only values are name and parent, and the check names the line of any other. Undo on a rename or move takes away only a pin that run added, so a pin that was there before stays.

The first rename, move or redraft on a tree written before pins existed also pins what the run log says you renamed or moved earlier (runs not undone, on nodes still in the tree), once. The tree file then carries pins_from_log: <date> under packet:, and the log is never read for pins again, so a pin you take out by hand stays out. A redraft does this before it matches anything, so a name you gave a type before this release is kept and pinned; its dry run lists the pins it would add.

Running it again#

Without a tree, a re-run doesn't map onto the notes from the one before.

If a family gets a new name, its old note stays behind, and a new one is made beside it.
If you edited a family note and a type's id matches by chance, another type's table can land under your edited heading.

Undo a run reverses either. A tree fixes both: its ids are stable, and it's kept beside the catalog note as the record.

With a tree, a re-index that adds or removes cards, or a change to the clustering, makes new clusters the tree doesn't fit. Redraft it from itself, a dry run first:

npm run catalog -- tree --vault <vault folder> --domain magic --from-tree --dry-run

The redraft matches the new clusters to your types by their cards, not by their names or numbers:

Kept. A version key you put in a take by hand follows its version first, as it wins over a cluster in the tree file. Then a new cluster belongs to a type when at least half of the rest of it is that type's cards. A type keeps its id, its name, its about, its place, its pins and its variants, and takes the clusters that are mostly its own: one that split keeps both halves, and one that grew takes the new cards. When two types could claim a cluster, the one with more of its cards wins, then the one earlier in the file. Half is the line because a cluster can then be mostly one type's at most: below it, a cluster that is a third each of three types would go to whichever was listed first. A tree that still fits its clusters comes out the same, byte for byte.
Placed. A cluster no type claims stays unplaced if most of it is what the tree left unplaced before. Otherwise it becomes a new type, with a fresh id no node has used, in the sub-family that holds most of its cards, else the one where the rest of its packet family went. It takes its name from the packet when that was made from these cards with --name (the packet subcommand), else a placeholder such as Type 4 that the dry run marks as needing a name. A cluster with nowhere to go is left for the Unplaced note.
Couldn't keep. A type none of whose cards are mostly in one cluster any more, a variant none of whose versions is left in its type, and a pinned sub-family or family whose cards all left. Each is listed with why, and stays in the tree as it was, with take: [] (a take written as an empty list isn't a problem), so you decide whether to delete it, merge it or give it a take by hand. A type that is kept but holds less than half the cards it had is listed too, so you can look at where they went.

The redraft never renames or moves a node it keeps: what you chose stays, pinned or not.

To know a type's old cards, the redraft reads the packet the tree was drafted from (<Domain>.packet.json, while its packet: matches), else the Versions tables in the family notes, else the version keys in the tree's take lists. A type it can't read in any of them is listed as couldn't keep.

The dry run prints the kept, placed and couldn't-keep lists, then the plan, with the tree file as one of the writes. Without --dry-run, the tree file and the notes are written in one run, so one Undo takes back both. The couldn't-keep list goes on the run's record and beside its log as <run>.redraft.md, and the packet for the new clusters is written beside the catalog note in the same run (Undo puts the old one back), so the next redraft reads this one's cards exactly. Another plugin can read the list through the API (catalogRedraftReport).

If naming fails for some clusters, the run prints each failure, calls those clusters "unnamed cluster" and exits with 3. Run it again once the cause is fixed.

The exit codes, for scripts: 0 when everything was written; 1 on an error; 2 when a note was in conflict, a note or table is stale, or the tree had problems; 3 when a naming call failed (3 wins over 2).

Names are also saved in the collection's topic names file, which Find topics shares. That file isn't part of the run log, so Undo leaves it.