crabidy/architecture/incremental-captures.md

Incremental captures, skipped tracks, and capture progress

Context and problem statement

Download captures (W, architecture/captures.md) are all-or-nothing: the whole subtree is built in a hidden temp folder and swapped into place; any failure destroys everything downloaded so far. For a large node that means hours of downloading can evaporate on one bad track, and re-running restarts from zero. Mixed-provider sources (queues, bookmarks) silently omit uncapturable tracks, so the capture's track list quietly diverges from the source. And while a capture runs, the user sees nothing — worse, the TUI's poll loop awaits the capture RPC, so the client is effectively frozen until the capture finishes.

This design makes download captures incremental and resumable, records uncapturable tracks as a first-class skipped playable, streams capture progress to clients, and warns about long captures up front. A small, unrelated fix rides along: colored library items (editable/creatable/marked) are unreadable under the focused selection bar (D7).

Assumptions

• "Capture" here means the download capture (W). Bookmark captures (w) stay atomic tmp-and-swap: they are cheap, and "overwrite = refresh" is the right semantic for links. They do adopt the skipped playable for skipped source tracks (D1) and report progress (D5).
• Resuming keys on the name: capturing into an existing capture name merges into that folder. Entry identity is the deterministic toml file name (NNNN <title>.cbd-track.toml), so resuming assumes the source keeps its order — appending to a queue is fine, reordering it re-captures under new names and leaves stale files behind (the user can delete the capture and start over). Accepted.
• One capture per name at a time is the user's responsibility (same as the old racing-tmp behavior); concurrent same-name captures interleave per file, last writer wins. Accepted.

D1 — A skipped playable

Track files get a fourth playable: [playable] skipped = true, validated with the same exactly-one cardinality as file/url/link (skipped = false counts as unset and is rejected). Semantics: this position in the tree is a real track whose audio could not be captured.

• fsdy::Playable::Skipped; TrackFile::from_track_skipped(track) builds one from a wire track.
• Wire: Track.is_skipped (proto field 6). TrackFile::to_track sets it; the track's path stays the lib path (like file), there is nothing to route to.
• TrackFile::from_track (queue persistence, bookmarks) writes a skipped playable when the source track is_skipped — skipped-ness survives queue persistence and bookmark round trips instead of degrading into a dead link.
• get_urls_for_track on a skipped file returns ProviderError::FetchError (playback never asks, see D4; direct callers get a normal typed error).

Alternative considered: model skipped-ness as absence (keep omitting the track) plus a client-side diff against the source. Rejected — the source may be gone tomorrow; the capture itself must record the gap.

D2 — Incremental download captures

capture_into splits into two phases:

1. Enumerate: the existing iterative pre-order walk collects every directory and track (with its listing index) first, enforcing max_dirs/max_tracks. This makes the total known before the first download — progress can be a real ratio — and costs only metadata calls.
2. Fetch: process the collected tracks in order, feeding progress after each one.

The sink decides the write mode:

• Sink::Link (bookmarks): unchanged tmp-and-swap into .tmp-<name>, all-or-nothing.
• Sink::Download (captures): writes directly into dir/<name>/, creating directories as needed, never deleting existing content. Per track, in order:
  • The target toml exists, parses, and its playable is not skipped, and (for a file playable) the referenced audio exists → reuse (counts as done, no download). A url/link playable also counts as satisfied — only this store writes here, but hand-edited files should not be clobbered.
  • Otherwise the track is (re)captured. A source that resolves to a local file path rather than an http(s) URL — an fs playable, or a track from an existing capture — is copied into the capture next to its toml (counting against the same byte budget as a download), so a queue mixing streamed and local tracks captures fully. A source that genuinely cannot be captured — the track is itself skipped, its stream fails to resolve, resolves to nothing, or the local file is missing/unreadable — writes a skipped toml (from_track_skipped) and counts as skipped. This replaces the old silent omission (and the older behavior of skipping local files outright).
  • A real download failure (HTTP status, transport, timeout, byte budget) aborts the run but keeps everything written so far — re-running the same name resumes exactly where it stopped, re-attempting skipped and missing entries only.

Audio is still written before its toml, so a crash mid-download leaves a toml-less audio file that the resume simply re-downloads (truncating on create). The byte budget counts only bytes downloaded this run, so resuming a large capture is never starved by what is already on disk.

direction: right
walk: capture_into {
  enumerate: "phase 1: enumerate\n(dirs + tracks, caps)"
  fetch: "phase 2: fetch\n(per track, in order)"
  enumerate -> fetch: "total known"
}
walk.fetch -> reuse: "toml ok + audio present"
walk.fetch -> skipped: "source uncapturable\n→ skipped = true toml"
walk.fetch -> download: "download + toml"
walk.fetch -> abort: "download failure\n(keeps progress)"

D3 — Skipped tracks in the queue

Skipped tracks queue like any other (the user sees the gap instead of a silently shorter queue). The TUI renders them red; playback skips them.

Playback::play already loops past tracks whose URLs fail to resolve. It now additionally:

• skips is_skipped tracks without a provider round trip, and
• bounds the whole skip loop by the queue length at entry — an all-skipped queue with repeat on used to be an infinite provider-hammering spin; now it stops the player with a warning after one full pass.

D4 — Capture progress on the update stream

New stream update (proto):

message CaptureProgress {
  string name = 1;           // capture / bookmark name
  bool download = 2;         // W capture vs w bookmark
  uint32 tracks_done = 3;    // settled: reused + downloaded + linked +
                             // skipped — reaches tracks_total on success
  uint32 tracks_total = 4;   // known after enumeration (0 until then)
  uint32 tracks_skipped = 5; // of those, skipped tomls written this run
  bool finished = 6;
  string error = 7;          // set iff finished with a failure
}

CaptureLibraryNode now returns once the capture is accepted: the provider validates the name, the store, and the source's download blessing, replies, and runs the walk on its spawned task, streaming CaptureProgress through a bounded channel that the RPC layer forwards into the existing update broadcast. Completion and failure arrive as the final progress event (finished, error), not as the RPC result.

Rationale: the TUI's orchestration loop select!s over one RPC at a time — a capture RPC that lasts an hour freezes every other interaction. Validation errors still come back synchronously with the old status mapping; walk errors move to the stream (and the server log, as before).

D5 — TUI: progress, red skipped tracks, warnings

• Skipped tracks are red (and not bold) in both the queue and library listings, driven by Track.is_skipped. The playing-track red keeps precedence in the queue.
• Progress lines render at the bottom of the library pane, one per active capture: capturing <name> 12/34 (2 skipped) (bookmarks: bookmarking). A finished capture lingers ~5 s as captured <name>: 34 tracks (2 skipped); a failed one shows the error in red for ~10 s. State lives in the App, fed by the update stream; the 100 ms render tick handles expiry.
• Warnings: the W help-table description and the capture input overlay's label both say a download capture can take a long time (and that re-capturing the same name resumes it).

D6 — Out of scope

• Cancelling a running capture from the TUI.
• Retrying real download failures within a run (rerun-to-resume covers it).
• Garbage-collecting stale entries when the source shrank or reordered.
• Multi-hop link resolution for skipped detection (a link whose target is a skipped file plays as a normal link failure).

D7 — Focused-selection contrast fix

Library items styled with a foreground color (creatable/editable/deletable → secondary, marked → green; queue: skipped/current → red) are hard to read when the focused selection bar (bg = COLOR_PRIMARY, a light blue) sits on them. Fix: when an item is the selected row of a focused pane, its foreground switches to the dark COLOR_PRIMARY_DARK so it reads against the light bar. The unfocused bar is dark and keeps the colored foregrounds.

Risks

• Enumerate-then-fetch holds the full entry list in memory: bounded by max_tracks (500 download / 20 000 bookmark) — fine.
• A source whose listing order changes between runs duplicates content under new prefixes (assumption above). Accepted; documented in the help text via the "resumes by name" phrasing.
• The progress channel is bounded (64); a slow broadcast consumer only slows the walk, never blocks it permanently (the forwarder drains continuously).