crabidy/architecture/crabidy-store.md

The crabidy provider and the content-addressed store

Context and problem statement

Today the server exposes three hand-managed subtrees — /queues, /bookmarks, /captures — each a separate fsdy::Client instance rooted under ~/.config/crabidy/. Bookmarks (w) write link tomls; captures (W) download audio next to each toml (capture-deletion.md, incremental-captures.md). This has three problems the user wants fixed green-field (no data migration):

1. No de-duplication. Capturing the same track from two places (a playlist and a search, or two saved queues) downloads and stores the audio twice. Re-capturing after a reorder re-downloads everything.
2. Audio lives next to metadata. Deleting a capture folder must carefully remove downloaded audio from disk behind a confirmation (capture-deletion.md), because the audio is only referenced from that one folder. This couples deletion to expensive-payload bookkeeping.
3. Three UI concepts (queues, bookmarks, captures) for what the user thinks of as "my crabidy stuff." They want one provider.

This design replaces all three with one filesystem provider, /crabidy, whose track tomls link into a single content-addressed store of playable files. Captures de-duplicate by provider identity and by content hash; deletion becomes a plain toml removal that never touches the store.

This supersedes bookmarks.md, captures.md, capture-deletion.md, and the on-disk/resumption parts of incremental-captures.md (the skipped-track and progress-stream parts of that doc survive; see D9).

Assumptions (confirmed by the request)

• Green-field. No migration of existing ~/.config/crabidy/{queues, bookmarks,captures} data. On first run the new locations are simply empty.
• Two roots, split by XDG kind. The store (playable audio + its sidecars) is data → ~/.local/share/crabidy/ (dirs::data_dir()). The tomls (the /crabidy tree) are state → ~/.local/state/crabidy/ (dirs::state_dir()).
• Single server, single writer. One process owns both roots; capture mutations are serialized. No cross-host concurrent writers.
• The store never shrinks automatically. Deleting a toml never deletes store audio (D7). Orphaned store entries are accepted; a garbage collector is future work (D10).
• current stays special. The live queue is still mirrored to a reserved, user-untouchable folder — now /crabidy/current, flat as before.

D1 — One provider: /crabidy

ProviderOrchestrator drops the queues/bookmarks/captures fields and gains one crabidy field: an fsdy::Client rooted at ~/.local/state/crabidy/, mounted at /crabidy, built with_editable_top_level(&["current"]) (top-level saves are renamable/deletable; current is reserved) .with_downloadable_nodes() .with_deletable_tree() (every node deletes directly — see D7). A companion writer, CrabidyStore, owns both roots and all mutation.

• Top-level folders under /crabidy are user saves, each created by w/W.
• Saved queues are flat; saved library subtrees preserve their structure (falls out of walking the source — a flat queue yields a flat save).
• /crabidy/current is the live-queue mirror the playback loop keeps in sync (replaces /queues/current). It is flat and reserved.

Non-toml files in a save folder (e.g. client log files that also land in ~/.local/state/crabidy/) are ignored by listing, as today — the provider only surfaces subdirectories and *.cbd-track.toml files. (Optional tidy-up, not required: move client logs to ~/.local/state/crabidy/logs/.)

direction: right
orchestrator: ProviderOrchestrator (root, path-routed) {
  tidal
  youtube
  fs
  crabidy: /crabidy (fsdy::Client, read + delete)
}
state: "~/.local/state/crabidy/\n(toml tree)" { shape: cylinder }
share: {
  label: "~/.local/share/crabidy/\ncontent store: audio + sidecars"
  shape: cylinder
}
store_writer: CrabidyStore (single writer, owns both roots) {
  index: "StoreIndex\n(provider-id → entry,\nhash → entry)"
}
orchestrator.crabidy -> state: lists tomls
orchestrator.crabidy -> share: resolves Store playables
store_writer -> state: writes save folders / track tomls
store_writer -> share: writes audio + sidecars
store_writer.index -> share: derived by scanning sidecars at open

D2 — The store and its sidecars

~/.local/share/crabidy/ is a flat directory. Each unique playable is a pair:

• <name> — the audio file, named after the source's natural name (a local file's basename, else a sanitized <title>.<ext> with the extension from the download's Content-Type/URL). On name collision with different content, append a numeral: song.flac, song (2).flac, … (identical content never reaches naming — it de-dupes first, D4).
• <name>.cbd-store.toml — the sidecar, the store entry's metadata:

# song.flac.cbd-store.toml
hash = "blake3:1f0c…"           # content hash of the audio file

[[provider]]                     # one entry per provider-id that maps here
provider = "tidal"               # provider name (path root of the source)
id = "125169484"                 # provider-internal id (see D3)
title = "Bohemian Rhapsody"
artist = "Queen"
duration = 355
aliases = ["Bohemian Rhapsody (Remastered)"]       # other titles for this id
[provider.album]
title = "A Night at the Opera"
release_date = "1975-11-21"

[[provider]]                     # same content reached via a second identity
provider = "youtube"
id = "fJ9rUzIMcZQ"
title = "Queen – Bohemian Rhapsody (Official Video)"

The sidecars are the single source of truth — there is no separate persisted index file. CrabidyStore builds an in-memory StoreIndex by scanning *.cbd-store.toml at open and updates it on every write:

• by_provider_id: HashMap<(provider, id), StoreRef>
• by_hash: HashMap<Hash, StoreRef>

A StoreRef is the store <name> (which is the toml link target and the sidecar key). Lookups are O(1).

"grep without shelling out." The request describes searching sidecars for a provider id / hash like ripgrep, in-process. The StoreIndex is it, memoized: it is derived by reading the sidecars in-process at open (no shell, no external index), so the store stays self-describing. A live content-scan-then-parse (via the grep-searcher crate) was considered as the literal realization; rejected because an in-memory map built once is simpler, strictly faster for repeated captures in a session, and needs no new dependency. If the store ever grows beyond memory, a lazy content-scan is the fallback (D10).

D3 — Provider identity on the wire

The store keys on a provider-internal id — "the id that clearly identifies the item inside the provider," which the path cannot be (the same item is reachable via a playlist, a search, an album…). Today no such id exists; identity is the path string. We add it to the wire model:

• Track.provider_item_id (proto field 7, string) — set by the owning provider when it produces a Track. Empty when unknown.
  • tidal → the numeric track id.
  • youtube → the video id.
  • fs → the source file's canonical absolute path (two /fs tomls pointing at the same file share an id → they de-dupe).
  • crabidy (already store-backed) → the store <name>; used only to detect "already captured" (D5).
• The provider name is the first path segment of the (resolved) source track path — to_track already rewrites a link track's path to its target, so a queued link routes to the real provider and carries that provider's id.

For the capture indicator (D8) we also add:

• Track.is_captured (field 8, bool) — the server sets it at listing time when the store index has an entry for this track's (provider, id) or its playable is already a store link. Cheap (one hashmap lookup) and works while browsing any provider, so you can see what you have captured.
• LibraryNode.is_captured (field 10) and LibraryNodeChild.is_captured (field 8) — a node is captured iff all its tracks and child nodes are captured (D8 covers how the /crabidy provider computes this cheaply).

D4 — Capturing one track: the de-dup flow

store = true means the track links into the store; link means a bookmark link to the source provider. Capture (W) produces store links; bookmark (w) produces plain links and never touches the store.

For each source track a capture walk visits, in order:

direction: down
start: "resolve source track\n(provider, id, natural name)"
already: "already store-backed?\n(Store playable, or fs file under the store root)"
byid: "index.by_provider_id[(provider,id)] ?"
getbytes: "obtain bytes\n(download → temp, or local file)"
byhash: "index.by_hash[hash(bytes)] ?"
newentry: "NEW: copy into store\n(+numeral) + write sidecar"
addid: "add provider to sidecar,\ndiscard the temp copy"
writetoml: "write track toml with playable.store = <name>"

start -> already
already -> writetoml: "yes → reuse target store name (no-op copy)"
already -> byid: "no"
byid -> writetoml: "HIT → reuse; record alias if title differs"
byid -> getbytes: "MISS"
getbytes -> byhash
byhash -> addid: "HIT (same content, new identity)"
byhash -> newentry: "MISS"
addid -> writetoml
newentry -> writetoml

1. Already store-backed? If the source track's playable is a Store link, or it is an /fs file whose path is already under the store root, there is nothing to fetch — the save's toml links to the same store <name>. (This is the "capture on an already-captured fs item → do nothing" case.)
2. Provider-id lookup. index.by_provider_id[(provider, id)] — a hit means we already have this exact provider item. Point the save's toml at that store entry. If the current title differs from the stored one, append it to that entry's aliases. No download. (Common re-capture path — makes redoing a save cheap.)
3. Miss → obtain bytes. Streamed source: windowed HTTP download to a temp file (unchanged mechanics). Local source (/fs pointing at a normal file, not under the store): the file is the bytes. Hash the bytes (blake3).
4. Hash lookup. index.by_hash[hash] — a hit means identical content is already stored under some other identity. Add a [[provider]] entry to that sidecar, discard the temp download (or do not copy the fs file), point the toml at the existing store name. No duplicate.
5. Miss → new store entry. Choose a store name from the natural name (+numeral on collision), move the temp file (or copy the fs file) into the store, write the sidecar with hash and the first [[provider]] entry, update the index. The /fs case copies — the original music-folder file stays put.

The byte budget (DOWNLOAD_CAPS.max_bytes) still counts bytes fetched this run (hits cost nothing), so dedup makes big saves cheaper, never starves them.

D5 — Save (w/W) and conflict handling

A save takes a source (a live-queue snapshot or a library-node path) and a mode (Link for w, Capture for W) and writes a new top-level folder /crabidy/<name>:

• w on a library node or the queue → folder of link tomls (fsdy::TrackFile::from_track), the bookmark semantics, no store, no audio.
• W on a library node or the queue → w plus run the D4 capture per track; tomls carry playable.store. Works on both library nodes and the queue.

Conflict = refuse. If /crabidy/<name> already exists, do nothing and warn name "<name>" already exists; the user deletes the old folder and saves again. This replaces the old overwrite (bookmark tmp-swap) and merge-by-name (capture resume). The save is still built in a hidden .tmp-<name> sibling and swapped in atomically, so a crashed/failed run leaves no blocking partial folder and the name stays free to retry — while any audio already committed to the store persists and makes the retry fast via D4. Resumability thus moves from the folder to the store; the folder is all-or-nothing.

current is exempt: the playback loop overwrites it on every queue change; the user cannot save over the reserved name current.

D6 — RPC surface

• CaptureLibraryNode(path, name, download) stays and now covers all four gestures: w/W on a library node (path = /tidal/...), and w/W on the queue (path = /crabidy/current). download=false → Link, true → Capture. Validation errors (bad name, conflict, source not downloadable) return synchronously; progress streams via the existing CaptureProgress update (unchanged, D9).
• SaveQueue(name) is retained (this deviates from the original plan to remove it — see plan/summary.md). It is the queue w gesture and is reimplemented server-side as a link save of the live queue into /crabidy/<name> (via CrabidyStore::save_snapshot). Queue W (QueueDownloadCapture) goes through CaptureLibraryNode on /crabidy/current with download = true.
• DeleteLibraryNode(path) unchanged in shape; behavior simplified (D7).

D7 — Deletion

Deletion on /crabidy (the only writable fs provider) goes through directly, no confirmation, and never touches the store:

• Delete a track → remove its .cbd-track.toml only.
• Delete a folder → remove_dir_all of the toml folder only.

The existing fsdy::delete_track_file guard — "only delete the referenced audio if it is contained under the instance root" — already makes this safe: a Store playable resolves under ~/.local/share/…, which is outside the /crabidy toml root at ~/.local/state/…, so the audio is never deleted. And w-saves are links with no audio at all.

Consequences: drop the TUI's delete_needs_confirmation / ConfirmDelete path (the whole capture-deletion.md confirmation feature is gone — nothing expensive is destroyed anymore) and drop with_deletable_tree's audio-removal branch usage for this provider (folder/toml removal remains).

D8 — UI: collapse to one provider, mark captured nodes

• /queues, /bookmarks, /captures disappear from the root; one /crabidy child appears (title crabidy). Inside it, w-saves (links) and W-saves (store-backed) coexist, distinguished by the captured marker.
• Captured marker: a captured row is prefixed with | as the first character of the row (before the selection padding), e.g. |Bohemian….
  • A track row is captured per Track.is_captured (D3): store-backed, or its (provider, id) is in the store index — visible even while browsing tidal.
  • A node/child row is captured iff all its tracks and child nodes are captured. The /crabidy provider computes a node's own is_captured when it lists it (it reads every track toml anyway → all Store?). To mark child folders in a parent listing without deep recursion, a save records its mode in a one-line marker at the save root written by w/W; nested folders under a W-save are captured by construction. Exact recursion depth is an implementation detail (see plan) — the invariant is "captured = fully local."
• The existing capture progress lines (capturing <name> 12/34 …) stay as-is.

D9 — What carries over from incremental-captures.md

Kept: the Skipped playable (D1 there), skipped tracks in the queue and playback skipping them (D3), the CaptureProgress stream and accept-then-stream RPC (D4), and the focused-selection contrast fix (D7). A source that genuinely cannot be fetched still writes a skipped toml. What changes: the store write mode (audio now goes to the shared store, tomls carry store, dedup per D4) and resume semantics (folder is atomic; store provides the savings, per D5).

D10 — Out of scope / future

• Store garbage collection. Nothing reclaims store entries whose last referencing toml was deleted. A future GC would scan all /crabidy tomls for live store names and remove unreferenced pairs; needs its own design.
• Lazy content-scan instead of the in-memory index, for stores too large to index in memory (D2).
• Cross-provider captured marking of folders (e.g. a tidal album shown captured) beyond the cheap track-level lookup.
• Cancelling a running capture (unchanged from prior scope).

Risks

• Wasted download on a hash-only match (provider-id missed but content is identical): we download, then discard. Unavoidable for content dedup; the provider-id path avoids it in the common case.
• Natural-name collisions across unrelated tracks are handled by the numeral suffix; the store name is opaque to users (only the toml title shows in UI).
• Index/disk skew if something outside the server edits the store: the index is rebuilt at every start, and the server is the sole writer, so skew is bounded to a single run — acceptable.
• blake3 dependency added (fast, maintained, no C toolchain). Alternative sha2 rejected for speed; hashing whole tracks is on the capture hot path.