crabidy/architecture/captures.md

Captures (downloaded subtrees)

Partially superseded by incremental-captures.md: download captures are now incremental (no tmp-and-swap, re-capturing a name resumes it), uncapturable tracks are recorded as skipped tomls instead of being omitted, and the capture RPC streams progress. Bookmarks keep the tmp-and-swap described here.

Context and problem statement

Bookmarks (w) mirror a library subtree as link files — replaying a bookmark still needs the original provider. The user wants W on a library node to do the same capture into a separate local provider called captures, except each track's audio is downloaded next to its .cbd-track.toml, and the toml points at that file. Playback of a capture then needs no provider round trip at all — it is a fully local copy.

A library node decides whether it allows W; Tidal implements it.

Assumptions (confirmed against the code)

• PlayableSpec.file already supports relative paths, resolved against the track file's directory at get_urls_for_track time (fs-provider D3). A toml next to its audio file can say file = "0001 Song.flac" and the whole capture folder stays relocatable (tmp-and-swap, rename, backup).
• fsdy::list_dir only surfaces directories and *.cbd-track.toml files; downloaded audio siblings are invisible to the library listing.
• The bookmark walk (bookmark_store::write_capture) is an iterative pre-order worklist whose only per-track action is "serialize and write one file" — exactly the seam where a download variant plugs in.
• reqwest (rustls, stream) is already a workspace dependency; tidal stream URLs come from get_urls_for_track on the orchestrator, so the download needs no new provider methods.
• Uppercase bindings (K, J) already exist in the TUI bindings table; W in Scope::Library is free.
• LibraryNode/LibraryNodeChild already model per-node capabilities (is_queable, is_creatable, …) — the "does this node allow W" decision extends that pattern.

Decisions

D1 — Fourth fsdy instance at /captures

<config>/crabidy/captures/ is mounted read-only as /captures with an editable top level (no reserved names), exactly like /bookmarks. Init is non-fatal: an unopenable store disables W and the mount, never the server. Loading a capture is browsing /captures and queueing a folder — zero new replay mechanisms.

D2 — One shared walk, two track sinks

Options considered:

• (a) Copy bookmark_store.rs and swap the per-track write.
• (b) Extract the walk into a shared capture module parameterized by a track sink; bookmarks and captures become thin stores over it.

Decision: (b) — the walk (worklist, caps, temp-and-swap, all-or-nothing cleanup, BadSource mapping) is behavior we already tested once and must not fork. crabidy-server/src/capture.rs owns CaptureError, the caps, and write_tree(client, source, tmp, caps, sink); the sink is an enum (no async-trait indirection):

• Sink::Link — today's bookmark behavior, byte-identical (TrackFile::from_track, link playable).
• Sink::Download(Downloader) — captures (D3).

BookmarkStore keeps its API; CaptureStore (in capture_store.rs) is its sibling over the captures directory.

D3 — Download sink: audio next to the toml, toml points at it

Per track, in listing order:

1. get_urls_for_track through the orchestrator (any provider that yields URLs works; Tidal is the target). First URL wins.
2. HTTP GET via one shared reqwest client — connect timeout, one total per-track deadline covering the whole body, no retries (a capture is re-runnable and overwrite = refresh; a retry policy can come later). The body is streamed to NNNN <title>.<ext> (shared ordered_name sanitizer, same 4-digit prefix as the toml so the pair sorts together).
3. The extension comes from the response Content-Type (audio/flac → flac, audio/mp4/audio/m4a → m4a, audio/mpeg → mp3, audio/ogg → ogg, audio/wav → wav), falling back to the URL path's extension, then bin (the player probes by content; the extension is a hint).
4. The toml is written after the download succeeds, with playable.file = "<audio file name>" (relative, new TrackFile::from_track_with_file), keeping metadata identical to a bookmark entry.

Caps: MAX_CAPTURE_DIRS stays 1 000; downloads get their own MAX_DOWNLOAD_TRACKS = 500 and a total byte budget MAX_DOWNLOAD_BYTES = 4 GiB counted while streaming — a runaway artist capture must not fill the disk. Downloads run sequentially (gentle on the provider, trivially bounded memory); the whole capture already runs on a spawned task, so the orchestrator keeps serving.

All-or-nothing is kept for real failures: any failed download (bad status, transport error, timeout) aborts the capture and removes the temp folder. The one softening: a track whose source cannot be captured at all — its stream fails to resolve, or resolves to a non-http(s) target (a local file playable) — is skipped with a warning instead of aborting. Queue and bookmark captures mix providers (D4), and one local /fs entry must not kill the downloadable rest; the skipped track simply has no pair in the capture.

D4 — Nodes opt in via is_downloadable

New proto fields LibraryNode.is_downloadable = 8 and LibraryNodeChild.is_downloadable = 7 (additive). Tidal sets the flag centrally at the end of get_lib_node: a node is downloadable when it is queueable or lists tracks (the "or lists tracks" covers search-term result nodes, which are not queueable as a whole but whose track results are downloadable); children mirror is_queable. The /queues and /bookmarks instances opt in wholesale (fsdy::Client::with_downloadable_nodes): their entries are links into downloadable providers, so W on a saved queue or bookmark downloads its resolvable tracks and skips the rest (D3). /fs and /captures stay false — capturing a capture is pointless, and local trees have nothing to download.

Tracks carry no flag: a listed track inherits its containing node's is_downloadable (TUI) — a Tidal album's tracks are downloadable because the album is. The server enforces at the capture root: a directory source must report is_downloadable, a track source's parent node must (CaptureError::Unsupported otherwise). Nested nodes inside the walk are not re-checked — the pressed node's decision governs its subtree.

D5 — Wire: the existing rpc gains a download flag

CaptureLibraryNodeRequest gets bool download = 3 (additive; old clients keep bookmarking). ProviderCommand::CaptureLibraryNode carries it and the handler picks the store. Error mapping extends the bookmark contract: InvalidName/BadSource → invalid_argument, TooLarge/Disabled/Unsupported → failed_precondition, download and disk failures → internal.

D6 — TUI: W on the library pane

Action::LibraryDownloadNode bound to W in Scope::Library ("Download selection as capture"). Gate: the bare selection must be queueable and downloadable (selected_downloadable(); child flag for nodes, the current node's flag for tracks; marks ignored like w). The existing input overlay opens with InputPurpose::Capture { path, download: true } (label capture), prefilled with the selection title. Submit sends MessageFromUi::CaptureNode { path, name, download } → the rpc. Failures are logged, never fatal to the poll loop.

D7 — Out of scope (explicitly)

• Retry/resume of failed or partial downloads (re-run the capture).
• Quality/codec selection, transcoding, tagging the audio files.
• Progress display in the TUI while a capture downloads.
• Deduplicating audio across captures, or refreshing links in existing bookmarks into downloads.
• DRM circumvention: the download uses exactly the stream URLs the provider already serves for playback.

Structure

direction: right

server: crabidy-server {
  orch: ProviderOrchestrator
  cap: "capture.rs\nshared walk + caps + swap" {
    link: "Sink::Link"
    dl: "Sink::Download\n(reqwest, timeouts, byte budget)"
  }
  bs: BookmarkStore
  cs: CaptureStore
}

tidal: "tidaldy\n(is_downloadable = is_queable)"

disk: "config/crabidy" {
  shape: cylinder
  b: "bookmarks/<name>/  (link tomls)"
  c: "captures/<name>/  (audio + file tomls)"
}

cfs: "fsdy /captures\n(editable top level)"

server.orch -> server.bs: "CaptureLibraryNode\ndownload=false"
server.orch -> server.cs: "CaptureLibraryNode\ndownload=true"
server.bs -> server.cap.link
server.cs -> server.cap.dl
server.cap.dl -> tidal: "get_urls_for_track\n+ HTTP GET stream"
server.bs -> disk.b
server.cs -> disk.c
cfs -> disk.c: "list + parse (read only)"
server.orch -> cfs: "/captures/..."

Key flow: W on a Tidal album

shape: sequence_diagram
tui: TUI
rpc: gRPC
orch: Orchestrator
cs: CaptureStore
tidal: Tidal

tui -> rpc: "CaptureLibraryNode(path, name, download=true)"
rpc -> orch: "ProviderCommand (spawned)"
orch -> cs: "capture(name)"
cs -> orch: "get_lib_node: root allows download?"
cs -> tidal: "per track: get_urls_for_track"
cs -> tidal: "HTTP GET (deadline, byte budget)"
cs -> cs: "audio + toml pair\n(toml after audio, file = relative)"
cs -> rpc: "tmp-and-swap captures/<name>/"
rpc -> tui: OK

Risks and open questions

• Disk usage: 500 tracks of FLAC can be tens of GiB; the byte budget caps one capture, not the folder's total. Accepted — the user manages captures/ like any local music folder (and can delete via d).
• Stream URL churn: Tidal URLs are short-lived; the download happens immediately after fetching each URL, so expiry only matters for very slow transfers, which the per-track deadline already bounds.
• Licensing: captures are personal-use copies of streams the account can already play; nothing here bypasses provider protection.
• Open (future): a progress event stream for long captures; retry with classification + jitter; per-provider download quality knobs.