crabidy/docs/src/store.md

The crabidy store

/crabidy is one filesystem provider — the only writable one — where the server keeps everything you save. Its listings look like any other library subtree (see The library model), but every track toml under it either links back to a source provider or links into a shared, content-addressed store of audio files. The two halves live in separate directories, split by XDG kind:

• the toml tree at ~/.local/state/crabidy/ — the folders and *.cbd-track.toml files the /crabidy provider lists (this is state);
• the content store at ~/.local/share/crabidy/ — a flat directory of audio files, each paired with a sidecar, that captured tomls point at (this is data).

One writer, CrabidyStore, owns both roots and serializes every mutation. The provider only reads: it lists the tomls and resolves store links when a captured track plays.

There is one `/crabidy` provider, not the older `/queues`, `/bookmarks`, and
`/captures` split. Bookmarks (`w`) and captures (`W`) are now two kinds of
save that coexist in the same tree, told apart by the captured marker.

The two roots

direction: right

orchestrator: Library (path-routed) {
  tidal
  youtube
  fs
  crabidy: "/crabidy (reads tomls,\nresolves store links)"
}

state: "~/.local/state/crabidy/\ntoml tree" { shape: cylinder }
share: {
  label: "~/.local/share/crabidy/\ncontent store:\naudio + sidecars"
  shape: cylinder
}

writer: "CrabidyStore (single writer)" {
  index: "StoreIndex\n(provider-id -> entry,\nhash -> entry)"
}

orchestrator.crabidy -> state: lists folders + tomls
orchestrator.crabidy -> share: resolves store playables
writer -> state: writes save folders + track tomls
writer -> share: writes audio + sidecars
writer.index -> share: built by scanning sidecars at open

Top-level folders under /crabidy are user saves, each created by w or W. A saved queue is flat; a saved library subtree keeps its structure (it falls out of walking the source). The reserved current folder is the live-queue mirror — see Queue and playback.

The store and its sidecars

~/.local/share/crabidy/ is a flat directory. Each unique playable is a pair of files:

• <name> — the audio file, named after the source's natural name: a local file's basename, or a sanitized <title>.<ext> for a download (the extension comes from the download's content type or URL). On a name collision with different content, a numeral is appended before the extension: song.flac, song (2).flac, … Identical content never reaches naming — it de-duplicates first.
• <name>.cbd-store.toml — the sidecar: the entry's content hash and every provider identity that maps to it.

# song.flac.cbd-store.toml
hash = "blake3:1f0c…"          # content hash of the audio file

[[provider]]                    # one entry per provider identity
provider = "tidal"              # provider name (source path root)
id = "125169484"                # provider-internal id
title = "Bohemian Rhapsody"
artist = "Queen"
duration = 355
aliases = ["Bohemian Rhapsody (Remastered)"]   # other titles for this id

[[provider]]                    # same audio reached via a second identity
provider = "youtube"
id = "fJ9rUzIMcZQ"
title = "Queen – Bohemian Rhapsody (Official Video)"

The sidecars are the single source of truth — there is no separate persisted index. At open, CrabidyStore scans every *.cbd-store.toml and builds an in-memory StoreIndex:

• by_provider_id: (provider, id) -> store name
• by_hash: content hash -> store name

Both lookups are O(1), and the index is updated on every write. A malformed sidecar is skipped with a warning, so one bad file never poisons the index.

The store name is opaque — only the toml's title shows in the UI. Two saves
that reference the same audio point their tomls at the same `<name>`, so
capturing a track twice from different places costs one copy on disk.

Provider identity

The store keys on a provider-internal id, not on a path (the same item is reachable through a playlist, a search, an album…). Each provider sets it on the tracks it produces:

• tidal → the numeric track id
• youtube → the video id
• fs → the source file's absolute path (two /fs tomls pointing at one file share an id, so they de-duplicate)

An empty id means the provider exposes none; identity then falls back to the content hash alone.

Saving: w and W

A save takes a source — a live-queue snapshot or a library-node path — and writes a new top-level folder /crabidy/<name>:

• w (bookmark). Writes a folder of link tomls, one per track, each carrying the entry's metadata and a link playable back to the source. No audio, no store. Reloading rewrites each link back to its target.
• W (capture). Writes the same folder, but each track's audio is fetched into the content store and its toml carries a store playable. Works on both a library node and the queue.

A download capture (`W`) can take a long time — it fetches every track's
audio. De-duplication (below) makes a re-capture cheap, but the first one is
bounded only by the source size and the byte budget.

Atomic saves, no overwrite

Every save is built in a hidden .tmp-<name> sibling folder and renamed into place on success; a failed run removes the temp folder and leaves nothing behind. If /crabidy/<name> already exists, the save is refused with name "<name>" already exists — nothing is clobbered. To replace a save, you delete the old folder first.

Resumability comes from the store, not the folder: a failed capture leaves the name free to retry, while any audio already committed to the store persists and makes the retry fast (the de-dup flow reuses it). The folder itself is all-or-nothing. current is exempt — the playback loop overwrites it on every queue change, and you cannot save over that reserved name.

De-duplication on capture

For each source track, a capture walks these steps in order and writes exactly one toml at the end:

direction: down

start: "resolve source track\n(provider, id, natural name)"
already: "already store-backed?\n(local file under the store root)"
byid: "index.by_provider_id[(provider, id)] ?"
getbytes: "obtain bytes\n(download to temp, or local file)"
byhash: "index.by_hash[hash(bytes)] ?"
newentry: "NEW: copy into store (+numeral)\nand write sidecar"
addid: "add identity to sidecar,\ndiscard the temp copy"
writetoml: "write track toml with a store playable"

start -> already
already -> writetoml: "yes: reuse the target name (no copy)"
already -> byid: "no"
byid -> writetoml: "HIT: reuse; record a differing title as an alias"
byid -> getbytes: "MISS"
getbytes -> byhash
byhash -> addid: "HIT (same content, new identity)"
byhash -> newentry: "MISS"
addid -> writetoml
newentry -> writetoml

1. Already store-backed? If the source resolves to a local file already inside the store root, there is nothing to fetch — the toml links to that same <name>. (Capturing an already-captured item does nothing.)
2. Provider-id lookup. If (provider, id) is already in the index, reuse that entry and skip the download. If the current title differs from the stored one, it is appended to that identity's aliases. This is the common re-capture path.
3. Miss → obtain bytes. A streamed source is downloaded to a temp file; a local source is the bytes. The bytes are hashed with blake3.
4. Hash lookup. If the hash is already in the index, identical content is already stored under some other identity: a new [[provider]] entry is added to that sidecar, the temp download is discarded, and the toml points at the existing entry. No duplicate.
5. Miss → new store entry. A store name is chosen from the natural name (with a numeral on collision), the temp file is moved (or the local file copied) into the store, and the sidecar is written with the hash and the first identity.

The byte budget counts only bytes fetched this run, so de-dup makes a big save cheaper and never starves it. Local /fs files are always copied — the original in your music folder stays put — and de-duplicated by hash like anything else.

Skipped tracks

A source that genuinely cannot be captured — a track already marked skipped, a stream that fails to resolve, a missing or unreadable local file — is recorded as a skipped toml ([playable] skipped = true) rather than silently omitted, so the save's track list matches the source. Skipped tracks render red in the TUI (see The terminal UI) and playback skips over them (see Queue and playback).

Deletion

Deletion on /crabidy goes through directly, with no confirmation, and never touches the store:

• Delete a track → remove its .cbd-track.toml only.
• Delete a folder → remove that toml folder only.

A store playable resolves under ~/.local/share/…, which is outside the /crabidy toml root at ~/.local/state/…, so the shared audio is never deleted — another save may still reference it. Because nothing expensive is ever destroyed, there is no delete-confirmation step.

Nothing reclaims store entries automatically when their last referencing toml
is deleted — the store never shrinks on its own. Those unreferenced entries are
surfaced for manual reclamation by the `/orphans` provider (below).

Reclaiming orphans — /orphans

Because deleting a save only removes tomls (never store audio), the store accumulates entries that nothing references any more. The /orphans provider is the reclamation view: it lists every store entry, walks the mounted local file providers (the /crabidy tree and /fs) to cross off the ones a Playable::Store toml still references, and presents the rest.

Each orphan is an ordinary editable/deletable/queueable node, so the usual keys apply:

• e renames the store entry — both the audio file and its *.cbd-store.toml sidecar — refusing a name already taken by another entry.
• d deletes the audio file and its sidecar from disk.
• a/Enter queue it, so you can listen before deciding.

The set is recomputed on every visit (no cached list), so a fresh capture that re-references an entry makes it drop off the list. "Referenced" means reachable through a mounted file provider: a scan --capture toml under a folder that is not mounted under /fs is not seen, so its target shows here as an orphan. See architecture/orphans.md for the boundary and the (benign) capture-then-delete race.

The captured marker

The library marks what you already hold. A captured row ends with a ↓ (down-arrow) — a trailing status marker after any action-key brackets ([e]/[d]), e.g. Bohemian… ↓:

• A track is captured when its playable is store-backed, or when its (provider, id) is in the store index. Because the check is one index lookup, it works while browsing any provider — you can see, in /tidal, which tracks you have already captured.
• A node is captured when all of its tracks are captured and it has no child nodes.

See also

• The library model — nodes, tracks, paths, and links.
• Providers — how each media source sets its identity.
• Queue and playback — the current mirror and skipped tracks.
• Configuration — where the two roots come from.