crabidy/architecture/crabidy-store.md

# The `crabidy` provider and the content-addressed store

## Context and problem statement

Today the server exposes three hand-managed subtrees — `/queues`, `/bookmarks`,
`/captures` — each a separate `fsdy::Client` instance rooted under
`~/.config/crabidy/`. Bookmarks (`w`) write *link* tomls; captures (`W`)
download audio next to each toml (`capture-deletion.md`,
`incremental-captures.md`). This has three problems the user wants fixed
green-field (no data migration):

1. **No de-duplication.** Capturing the same track from two places (a playlist
   and a search, or two saved queues) downloads and stores the audio twice.
   Re-capturing after a reorder re-downloads everything.
2. **Audio lives next to metadata.** Deleting a capture folder must carefully
   remove downloaded audio from disk behind a confirmation
   (`capture-deletion.md`), because the audio is only referenced from that one
   folder. This couples deletion to expensive-payload bookkeeping.
3. **Three UI concepts** (queues, bookmarks, captures) for what the user thinks
   of as "my crabidy stuff." They want one provider.

This design replaces all three with **one filesystem provider, `/crabidy`**,
whose track tomls *link into a single content-addressed store* of playable
files. Captures de-duplicate by provider identity and by content hash; deletion
becomes a plain toml removal that never touches the store.

This **supersedes** `bookmarks.md`, `captures.md`, `capture-deletion.md`, and
the on-disk/resumption parts of `incremental-captures.md` (the skipped-track and
progress-stream parts of that doc survive; see D9).

## Assumptions (confirmed by the request)

- **Green-field.** No migration of existing `~/.config/crabidy/{queues,
  bookmarks,captures}` data. On first run the new locations are simply empty.
- **Two roots, split by XDG kind.** The store (playable audio + its sidecars)
  is *data* → `~/.local/share/crabidy/` (`dirs::data_dir()`). The tomls (the
  `/crabidy` tree) are *state* → `~/.local/state/crabidy/` (`dirs::state_dir()`).
- **Single server, single writer.** One process owns both roots; capture
  mutations are serialized. No cross-host concurrent writers.
- **The store never shrinks automatically.** Deleting a toml never deletes store
  audio (D7). Orphaned store entries are accepted; a garbage collector is future
  work (D10).
- **`current` stays special.** The live queue is still mirrored to a reserved,
  user-untouchable folder — now `/crabidy/current`, flat as before.

## D1 — One provider: `/crabidy`

`ProviderOrchestrator` drops the `queues`/`bookmarks`/`captures` fields and gains
one `crabidy` field: an `fsdy::Client` rooted at `~/.local/state/crabidy/`,
mounted at `/crabidy`, built `with_editable_top_level(&["current"])` (top-level
saves are renamable/deletable; `current` is reserved) `.with_downloadable_nodes()`
`.with_deletable_tree()` (every node deletes directly — see D7). A companion
writer, `CrabidyStore`, owns *both* roots and all mutation.

- Top-level folders under `/crabidy` are **user saves**, each created by `w`/`W`.
- **Saved queues are flat**; **saved library subtrees preserve their structure**
  (falls out of walking the source — a flat queue yields a flat save).
- `/crabidy/current` is the live-queue mirror the playback loop keeps in sync
  (replaces `/queues/current`). It is flat and reserved.

Non-toml files in a save folder (e.g. client log files that also land in
`~/.local/state/crabidy/`) are ignored by listing, as today — the provider only
surfaces subdirectories and `*.cbd-track.toml` files. (Optional tidy-up, not
required: move client logs to `~/.local/state/crabidy/logs/`.)

```d2
direction: right
orchestrator: ProviderOrchestrator (root, path-routed) {
  tidal
  youtube
  fs
  crabidy: /crabidy (fsdy::Client, read + delete)
}
state: "~/.local/state/crabidy/\n(toml tree)" { shape: cylinder }
share: {
  label: "~/.local/share/crabidy/\ncontent store: audio + sidecars"
  shape: cylinder
}
store_writer: CrabidyStore (single writer, owns both roots) {
  index: "StoreIndex\n(provider-id → entry,\nhash → entry)"
}
orchestrator.crabidy -> state: lists tomls
orchestrator.crabidy -> share: resolves Store playables
store_writer -> state: writes save folders / track tomls
store_writer -> share: writes audio + sidecars
store_writer.index -> share: derived by scanning sidecars at open
```

## D2 — The store and its sidecars

`~/.local/share/crabidy/` is a **flat** directory. Each unique playable is a
pair:

- `<name>` — the audio file, named after the source's natural name (a local
  file's basename, else a sanitized `<title>.<ext>` with the extension from the
  download's `Content-Type`/URL). On name collision with *different* content,
  append a numeral: `song.flac`, `song (2).flac`, … (identical content never
  reaches naming — it de-dupes first, D4).
- `<name>.cbd-store.toml` — the sidecar, the store entry's metadata:

```toml
# song.flac.cbd-store.toml
hash = "blake3:1f0c…"           # content hash of the audio file

[[provider]]                     # one entry per provider-id that maps here
provider = "tidal"               # provider name (path root of the source)
id = "125169484"                 # provider-internal id (see D3)
title = "Bohemian Rhapsody"
artist = "Queen"
duration = 355
aliases = ["Bohemian Rhapsody (Remastered)"]       # other titles for this id
[provider.album]
title = "A Night at the Opera"
release_date = "1975-11-21"

[[provider]]                     # same content reached via a second identity
provider = "youtube"
id = "fJ9rUzIMcZQ"
title = "Queen – Bohemian Rhapsody (Official Video)"
```

The **sidecars are the single source of truth** — there is no separate persisted
index file. `CrabidyStore` builds an in-memory `StoreIndex` by scanning
`*.cbd-store.toml` at open and updates it on every write:

- `by_provider_id: HashMap<(provider, id), StoreRef>`
- `by_hash: HashMap<Hash, StoreRef>`

A `StoreRef` is the store `<name>` (which is the toml link target and the sidecar
key). Lookups are O(1).

> **"grep without shelling out."** The request describes searching sidecars
> for a provider id / hash like ripgrep, in-process. The `StoreIndex` *is* it,
> memoized: it is derived by reading the sidecars in-process at open (no shell,
> no external index), so the store stays self-describing. A live
> content-scan-then-parse (via the `grep-searcher` crate) was considered as the
> literal realization; rejected because an in-memory map built once is simpler,
> strictly faster for repeated captures in a session, and needs no new
> dependency. If the store ever grows beyond memory, a lazy content-scan is the
> fallback (D10).

## D3 — Provider identity on the wire

The store keys on a **provider-internal id** — "the id that clearly identifies
the item inside the provider," which the path cannot be (the same item is
reachable via a playlist, a search, an album…). Today no such id exists; identity
is the path string. We add it to the wire model:

- `Track.provider_item_id` (proto field 7, `string`) — set by the owning
  provider when it produces a `Track`. Empty when unknown.
  - **tidal** → the numeric track id.
  - **youtube** → the video id.
  - **fs** → the source file's canonical absolute path (two `/fs` tomls pointing
    at the same file share an id → they de-dupe).
  - **crabidy** (already store-backed) → the store `<name>`; used only to detect
    "already captured" (D5).
- The **provider name** is the first path segment of the (resolved) source track
  path — `to_track` already rewrites a link track's `path` to its target, so a
  queued link routes to the real provider and carries that provider's id.

For the capture indicator (D8) we also add:

- `Track.is_captured` (field 8, `bool`) — the server sets it at listing time when
  the store index has an entry for this track's `(provider, id)` **or** its
  playable is already a store link. Cheap (one hashmap lookup) and works while
  browsing *any* provider, so you can see what you have captured.
- `LibraryNode.is_captured` (field 10) and `LibraryNodeChild.is_captured`
  (field 8) — a node is captured iff all its tracks and child nodes are captured
  (D8 covers how the `/crabidy` provider computes this cheaply).

## D4 — Capturing one track: the de-dup flow

`store = true` means the track links into the store; `link` means a bookmark link
to the source provider. Capture (`W`) produces store links; bookmark (`w`)
produces plain links and never touches the store.

For each source track a capture walk visits, in order:

```d2
direction: down
start: "resolve source track\n(provider, id, natural name)"
already: "already store-backed?\n(Store playable, or fs file under the store root)"
byid: "index.by_provider_id[(provider,id)] ?"
getbytes: "obtain bytes\n(download → temp, or local file)"
byhash: "index.by_hash[hash(bytes)] ?"
newentry: "NEW: copy into store\n(+numeral) + write sidecar"
addid: "add provider to sidecar,\ndiscard the temp copy"
writetoml: "write track toml with playable.store = <name>"

start -> already
already -> writetoml: "yes → reuse target store name (no-op copy)"
already -> byid: "no"
byid -> writetoml: "HIT → reuse; record alias if title differs"
byid -> getbytes: "MISS"
getbytes -> byhash
byhash -> addid: "HIT (same content, new identity)"
byhash -> newentry: "MISS"
addid -> writetoml
newentry -> writetoml
```

1. **Already store-backed?** If the source track's playable is a `Store` link, or
   it is an `/fs` file whose path is already under the store root, there is
   nothing to fetch — the save's toml links to the same store `<name>`. (This is
   the "capture on an already-captured fs item → do nothing" case.)
2. **Provider-id lookup.** `index.by_provider_id[(provider, id)]` — a hit means
   we already have this exact provider item. Point the save's toml at that store
   entry. If the current title differs from the stored one, append it to that
   entry's `aliases`. **No download.** (Common re-capture path — makes redoing a
   save cheap.)
3. **Miss → obtain bytes.** Streamed source: windowed HTTP download to a temp
   file (unchanged mechanics). Local source (`/fs` pointing at a normal file, not
   under the store): the file is the bytes. Hash the bytes (blake3).
4. **Hash lookup.** `index.by_hash[hash]` — a hit means identical content is
   already stored under some other identity. **Add** a `[[provider]]` entry to
   that sidecar, **discard** the temp download (or do not copy the fs file),
   point the toml at the existing store name. No duplicate.
5. **Miss → new store entry.** Choose a store name from the natural name
   (+numeral on collision), move the temp file (or copy the fs file) into the
   store, write the sidecar with `hash` and the first `[[provider]]` entry, update
   the index. The `/fs` case *copies* — the original music-folder file stays put.

The byte budget (`DOWNLOAD_CAPS.max_bytes`) still counts bytes fetched *this run*
(hits cost nothing), so dedup makes big saves cheaper, never starves them.

## D5 — Save (`w`/`W`) and conflict handling

A **save** takes a *source* (a live-queue snapshot or a library-node path) and a
*mode* (`Link` for `w`, `Capture` for `W`) and writes a new top-level folder
`/crabidy/<name>`:

- **`w`** on a library node or the queue → folder of **link** tomls
  (`fsdy::TrackFile::from_track`), the bookmark semantics, no store, no audio.
- **`W`** on a library node or the queue → `w` **plus** run the D4 capture per
  track; tomls carry `playable.store`. Works on both library nodes and the queue.

**Conflict handling.** Each save records its origin in a hidden
`.cbd-save.toml` marker at the save root (`source = "<captured node path>"`,
`capture = true|false`). When `/crabidy/<name>` already exists:

- **`w` (link save)** always refuses — do nothing and warn `name "<name>"
  already exists`; the user deletes the old folder and saves again.
- **`W` (capture)** refuses **unless it is a re-capture of the same source**:
  if the existing save's marker `source` equals the node being captured, the
  save **replaces** it; a different source still refuses. So pressing `W`
  again on the same item (or `W` on the queue again) refreshes the capture in
  place, while `W` naming a *different* item after an existing save is still
  protected.

The save is built in a hidden `.tmp-<name>` sibling and swapped in atomically;
a permitted replace removes the old folder and renames the temp over it. A
crashed/failed run leaves **no** blocking partial folder, and any audio already
committed to the store persists and makes a retry fast via D4 (resumability
lives in the store, not the folder). Replacing is cheap and safe because a
save folder holds only tomls — the shared store audio is never rewritten or
deleted (D7). A save that predates the marker (no `.cbd-save.toml`) is treated
as a different source and refuses; delete it once to re-establish it.

`current` is exempt: the playback loop overwrites it on every queue change; the
user cannot save over the reserved name `current`.

## D6 — RPC surface

- **`CaptureLibraryNode(path, name, download)`** stays and now covers *all four*
  gestures: `w`/`W` on a library node (`path = /tidal/...`), and `w`/`W` on the
  queue (`path = /crabidy/current`). `download=false` → `Link`, `true` →
  `Capture`. Validation errors (bad name, conflict, source not downloadable)
  return synchronously; progress streams via the existing `CaptureProgress`
  update (unchanged, D9).
- **`SaveQueue(name)` is retained** (this deviates from the original plan to
  remove it — see `plan/summary.md`). It is the queue `w` gesture and is
  reimplemented server-side as a *link* save of the live queue into
  `/crabidy/<name>` (via `CrabidyStore::save_snapshot`). Queue `W`
  (`QueueDownloadCapture`) goes through `CaptureLibraryNode` on
  `/crabidy/current` with `download = true`.
- `DeleteLibraryNode(path)` unchanged in shape; behavior simplified (D7).

## D7 — Deletion

Deletion on `/crabidy` (the only writable fs provider) **goes through directly,
no confirmation, and never touches the store**:

- Delete a track → remove its `.cbd-track.toml` only.
- Delete a folder → `remove_dir_all` of the toml folder only.

The existing `fsdy::delete_track_file` guard — "only delete the referenced audio
if it is contained under the instance root" — already makes this safe: a
`Store` playable resolves under `~/.local/share/…`, which is *outside* the
`/crabidy` toml root at `~/.local/state/…`, so the audio is never deleted. And
`w`-saves are links with no audio at all.

Consequences: drop the TUI's `delete_needs_confirmation` / `ConfirmDelete` path
(the whole `capture-deletion.md` confirmation feature is gone — nothing expensive
is destroyed anymore) and drop `with_deletable_tree`'s audio-removal branch usage
for this provider (folder/toml removal remains).

## D8 — UI: collapse to one provider, mark captured nodes

- `/queues`, `/bookmarks`, `/captures` disappear from the root; one `/crabidy`
  child appears (title `crabidy`). Inside it, `w`-saves (links) and `W`-saves
  (store-backed) coexist, distinguished by the captured marker.
- **Captured marker:** a captured row carries a trailing `↓` at the *end of
  the row* — a status marker after the action-key brackets (outside them),
  e.g. `Bohemian…  ↓`.
  - A **track** row is captured per `Track.is_captured` (D3): store-backed, or its
    `(provider, id)` is in the store index — visible even while browsing tidal.
  - A **node/child** row is captured iff all its tracks and child nodes are
    captured. The `/crabidy` provider computes a node's own `is_captured` when it
    lists it (it reads every track toml anyway → all `Store`?). To mark child
    *folders* in a parent listing without deep recursion, a save records its mode
    in a one-line marker at the save root written by `w`/`W`; nested folders under
    a `W`-save are captured by construction. Exact recursion depth is an
    implementation detail (see plan) — the invariant is "captured = fully local."
- The existing capture **progress** lines (`capturing <name> 12/34 …`) stay as-is.

## D9 — What carries over from `incremental-captures.md`

Kept: the `Skipped` playable (D1 there), skipped tracks in the queue and playback
skipping them (D3), the `CaptureProgress` stream and accept-then-stream RPC (D4),
and the focused-selection contrast fix (D7). A source that genuinely cannot be
fetched still writes a skipped toml. What changes: the *store write mode* (audio
now goes to the shared store, tomls carry `store`, dedup per D4) and *resume
semantics* (folder is atomic; store provides the savings, per D5).

## D10 — Out of scope / future

- **Store garbage collection.** Nothing reclaims store entries whose last
  referencing toml was deleted. A future GC would scan all `/crabidy` tomls for
  live `store` names and remove unreferenced pairs; needs its own design.
- **Lazy content-scan** instead of the in-memory index, for stores too large to
  index in memory (D2).
- **Cross-provider captured marking of folders** (e.g. a tidal album shown
  captured) beyond the cheap track-level lookup.
- **Cancelling a running capture** (unchanged from prior scope).

## Risks

- **Wasted download on a hash-only match** (provider-id missed but content is
  identical): we download, then discard. Unavoidable for content dedup; the
  provider-id path avoids it in the common case.
- **Natural-name collisions** across unrelated tracks are handled by the numeral
  suffix; the store name is opaque to users (only the toml title shows in UI).
- **Index/disk skew** if something outside the server edits the store: the index
  is rebuilt at every start, and the server is the sole writer, so skew is
  bounded to a single run — acceptable.
- **blake3 dependency** added (fast, maintained, no C toolchain). Alternative
  `sha2` rejected for speed; hashing whole tracks is on the capture hot path.