crabidy/architecture/incremental-captures.md

# Incremental captures, skipped tracks, and capture progress

## Context and problem statement

Download captures (`W`, architecture/captures.md) are all-or-nothing: the
whole subtree is built in a hidden temp folder and swapped into place; any
failure destroys everything downloaded so far. For a large node that means
hours of downloading can evaporate on one bad track, and re-running restarts
from zero. Mixed-provider sources (queues, bookmarks) silently *omit*
uncapturable tracks, so the capture's track list quietly diverges from the
source. And while a capture runs, the user sees nothing — worse, the TUI's
poll loop awaits the capture RPC, so the client is effectively frozen until
the capture finishes.

This design makes download captures **incremental and resumable**, records
uncapturable tracks as a first-class **skipped** playable, streams **capture
progress** to clients, and warns about long captures up front. A small,
unrelated fix rides along: colored library items (editable/creatable/marked)
are unreadable under the focused selection bar (D7).

## Assumptions

- "Capture" here means the *download* capture (`W`). Bookmark captures (`w`)
  stay atomic tmp-and-swap: they are cheap, and "overwrite = refresh" is the
  right semantic for links. They do adopt the skipped playable for skipped
  source tracks (D1) and report progress (D5).
- Resuming keys on the **name**: capturing into an existing capture name
  merges into that folder. Entry identity is the deterministic toml file name
  (`NNNN <title>.cbd-track.toml`), so resuming assumes the source keeps its
  order — appending to a queue is fine, reordering it re-captures under new
  names and leaves stale files behind (the user can delete the capture and
  start over). Accepted.
- One capture per name at a time is the user's responsibility (same as the
  old racing-tmp behavior); concurrent same-name captures interleave per
  file, last writer wins. Accepted.

## D1 — A `skipped` playable

Track files get a fourth playable: `[playable] skipped = true`, validated
with the same exactly-one cardinality as `file`/`url`/`link`
(`skipped = false` counts as unset and is rejected). Semantics: *this
position in the tree is a real track whose audio could not be captured*.

- `fsdy::Playable::Skipped`; `TrackFile::from_track_skipped(track)` builds
  one from a wire track.
- Wire: `Track.is_skipped` (proto field 6). `TrackFile::to_track` sets it;
  the track's `path` stays the lib path (like `file`), there is nothing to
  route to.
- `TrackFile::from_track` (queue persistence, bookmarks) writes a skipped
  playable when the source track `is_skipped` — skipped-ness survives queue
  persistence and bookmark round trips instead of degrading into a dead
  link.
- `get_urls_for_track` on a skipped file returns `ProviderError::FetchError`
  (playback never asks, see D4; direct callers get a normal typed error).

Alternative considered: model skipped-ness as *absence* (keep omitting the
track) plus a client-side diff against the source. Rejected — the source may
be gone tomorrow; the capture itself must record the gap.

## D2 — Incremental download captures

`capture_into` splits into two phases:

1. **Enumerate**: the existing iterative pre-order walk collects every
   directory and track (with its listing index) first, enforcing
   `max_dirs`/`max_tracks`. This makes the total known before the first
   download — progress can be a real ratio — and costs only metadata calls.
2. **Fetch**: process the collected tracks in order, feeding progress after
   each one.

The sink decides the write mode:

- `Sink::Link` (bookmarks): unchanged tmp-and-swap into `.tmp-<name>`,
  all-or-nothing.
- `Sink::Download` (captures): writes **directly** into `dir/<name>/`,
  creating directories as needed, never deleting existing content. Per
  track, in order:
  - The target toml exists, parses, and its playable is *not* skipped, and
    (for a `file` playable) the referenced audio exists → **reuse** (counts
    as done, no download). A `url`/`link` playable also counts as satisfied
    — only this store writes here, but hand-edited files should not be
    clobbered.
  - Otherwise the track is (re)captured. A source that resolves to a
    local **file** path rather than an http(s) URL — an fs playable, or a
    track from an existing capture — is **copied** into the capture next
    to its toml (counting against the same byte budget as a download), so
    a queue mixing streamed and local tracks captures fully. A source
    that genuinely cannot be captured — the track is itself skipped, its
    stream fails to resolve, resolves to nothing, or the local file is
    missing/unreadable — writes a **skipped toml** (`from_track_skipped`)
    and counts as skipped. This replaces the old silent omission (and the
    older behavior of skipping local files outright).
  - A real download failure (HTTP status, transport, timeout, byte budget)
    **aborts the run but keeps everything written so far** — re-running the
    same name resumes exactly where it stopped, re-attempting skipped and
    missing entries only.

Audio is still written before its toml, so a crash mid-download leaves a
toml-less audio file that the resume simply re-downloads (truncating on
create). The byte budget counts only bytes downloaded *this run*, so resuming
a large capture is never starved by what is already on disk.

```d2
direction: right
walk: capture_into {
  enumerate: "phase 1: enumerate\n(dirs + tracks, caps)"
  fetch: "phase 2: fetch\n(per track, in order)"
  enumerate -> fetch: "total known"
}
walk.fetch -> reuse: "toml ok + audio present"
walk.fetch -> skipped: "source uncapturable\n→ skipped = true toml"
walk.fetch -> download: "download + toml"
walk.fetch -> abort: "download failure\n(keeps progress)"
```

## D3 — Skipped tracks in the queue

Skipped tracks queue like any other (the user sees the gap instead of a
silently shorter queue). The TUI renders them red; playback skips them.

`Playback::play` already loops past tracks whose URLs fail to resolve. It now
additionally:

- skips `is_skipped` tracks **without a provider round trip**, and
- bounds the whole skip loop by the queue length at entry — an all-skipped
  queue with repeat on used to be an infinite provider-hammering spin; now
  it stops the player with a warning after one full pass.

## D4 — Capture progress on the update stream

New stream update (proto):

```proto
message CaptureProgress {
  string name = 1;           // capture / bookmark name
  bool download = 2;         // W capture vs w bookmark
  uint32 tracks_done = 3;    // settled: reused + downloaded + linked +
                             // skipped — reaches tracks_total on success
  uint32 tracks_total = 4;   // known after enumeration (0 until then)
  uint32 tracks_skipped = 5; // of those, skipped tomls written this run
  bool finished = 6;
  string error = 7;          // set iff finished with a failure
}
```

`CaptureLibraryNode` now returns once the capture is **accepted**: the
provider validates the name, the store, and the source's download blessing,
replies, and runs the walk on its spawned task, streaming `CaptureProgress`
through a bounded channel that the RPC layer forwards into the existing
update broadcast. Completion and failure arrive as the final progress event
(`finished`, `error`), not as the RPC result.

Rationale: the TUI's orchestration loop `select!`s over one RPC at a time —
a capture RPC that lasts an hour freezes every other interaction. Validation
errors still come back synchronously with the old status mapping; walk
errors move to the stream (and the server log, as before).

## D5 — TUI: progress, red skipped tracks, warnings

- **Skipped tracks are red** (and not bold) in both the queue and library
  listings, driven by `Track.is_skipped`. The playing-track red keeps
  precedence in the queue.
- **Progress lines** render at the bottom of the library pane, one per
  active capture: `capturing <name> 12/34 (2 skipped)` (bookmarks:
  `bookmarking`). A finished capture lingers ~5 s as
  `captured <name>: 34 tracks (2 skipped)`; a failed one shows the error in
  red for ~10 s. State lives in the `App`, fed by the update stream; the
  100 ms render tick handles expiry.
- **Warnings**: the `W` help-table description and the capture input
  overlay's label both say a download capture can take a long time (and that
  re-capturing the same name resumes it).

## D6 — Out of scope

- Cancelling a running capture from the TUI.
- Retrying real download failures within a run (rerun-to-resume covers it).
- Garbage-collecting stale entries when the source shrank or reordered.
- Multi-hop link resolution for skipped detection (a link whose target is a
  skipped file plays as a normal link failure).

## D7 — Focused-selection contrast fix

Library items styled with a foreground color (creatable/editable/deletable →
secondary, marked → green; queue: skipped/current → red) are hard to read
when the focused selection bar (`bg = COLOR_PRIMARY`, a light blue) sits on
them. Fix: when an item is the selected row of a *focused* pane, its
foreground switches to the dark `COLOR_PRIMARY_DARK` so it reads against the
light bar. The unfocused bar is dark and keeps the colored foregrounds.

## Risks

- Enumerate-then-fetch holds the full entry list in memory: bounded by
  `max_tracks` (500 download / 20 000 bookmark) — fine.
- A source whose listing order changes between runs duplicates content under
  new prefixes (assumption above). Accepted; documented in the help text via
  the "resumes by name" phrasing.
- The progress channel is bounded (64); a slow broadcast consumer only slows
  the walk, never blocks it permanently (the forwarder drains continuously).