crabidy/architecture/progressive-queueing.md

# Progressive queueing of large collections

## Context and problem statement

Queueing a nested node (an artist with many albums, a large playlist) today
freezes the UI's mental model: nothing changes for many seconds, then the full
queue appears at once. Three compounding causes, all in the resolve path:

1. `Playback::resolve_tracks` collects **every** track before the queue is
   touched, so the single `Queue` broadcast happens only at the very end and
   playback cannot start earlier.
2. The resolve runs **inline in the playback loop**, so every other playback
   command — pause, next, volume — is blocked for the duration.
3. `tidaldy` paginates collections to exhaustion (50 tracks per sequential
   request) inside one `get_lib_node` call, so even a single large playlist
   produces no intermediate result.

The feature: resolve progressively. Tracks are applied to the queue in chunks
as the provider produces them, each chunk is broadcast, playback starts with
the first chunk, and clients see a loading indicator (animated dots as a
pseudo last queue item) while resolution is still running.

Run autonomously per standing user instruction; every decision below records
the options considered and the rationale.

## Assumptions

- Chunks must arrive **in playback order** — the user explicitly wants "the
  first elements first, and later chunks follow". Order is known up front
  (collection order), so no reordering step is needed.
- The four resolve-based queue operations (`Replace`, `Queue`, `Append`,
  `Insert`) all benefit equally and should share one mechanism.
- Wire changes must stay additive (old clients keep working; they simply see
  the queue fill progressively without an indicator).
- Multi-client remains supported: the indicator must be server-derived state,
  not client-local guessing.

## Decisions

### D1 — End-to-end progressive resolution, not a client-only spinner

Options considered:

1. **Client-only indicator**: the TUI shows dots between sending a queue op
   and receiving the next `Queue` update. No wire or server change.
2. **Server-side chunked resolution** with a wire-visible "still resolving"
   flag; the indicator falls out of the flag.

**Decision: (2).** Option 1 papers over the latency without fixing it —
playback would still start only after the full resolve, other clients would
see nothing, and the "did it even register?" dead time remains. Option 2
fixes the actual complaint (start playing early, fill visibly) and gives
every client the indicator for free. The server broadcasts an immediate
`Queue` update (unchanged tracks, `resolving = true`) when the op is
accepted, so feedback appears within one round trip.

### D2 — `Queue.resolving` field, not a new stream-update variant

Options considered:

1. New `GetUpdateStream` oneof variant `resolving(bool)`.
2. New field `bool resolving = 4` on the `Queue` message itself.

**Decision: (2).** The flag is queue state and must be atomic with the track
snapshot it describes; a separate variant can arrive out of order relative to
`Queue` updates (the broadcast channel is lossy for slow clients). Additive
field, wire-compatible both ways: old clients ignore it, old servers never
set it.

### D3 — Chunked resolution lives in `ProviderClient`, with a default

Options considered:

1. Keep the BFS in `ProviderOrchestrator` (chunk = one node's tracks); no
   trait change. A 1000-track playlist is still one 20-request blob.
2. Add a chunked resolve method to the `ProviderClient` trait with a default
   implementation (the generic walk, one chunk per node); `tidaldy`
   overrides it to stream **page-sized chunks (50)** for the paginated
   collections (playlists, albums).

**Decision: (2).**

```rust
/// Streams the playable tracks under `path` into `chunk_tx` in playback
/// order. Zero or more chunks, then the sender is dropped: a dropped
/// SENDER means resolution finished. A dropped RECEIVER cancels
/// resolution (the provider stops fetching and returns Ok).
async fn resolve_tracks_into(
    &self,
    path: &str,
    chunk_tx: flume::Sender<Vec<Track>>,
) -> Result<(), ProviderError>;
```

The channel semantics are the contract and are documented on the trait —
this is a local bounded channel used as a stream, not a queue pretending to
be durable. Unreadable nodes are skipped with a warning (today's behavior);
only a completely unresolvable root returns an error. The default
implementation walks the tree **depth-first pre-order** over queueable
descendants, emitting one chunk per node. This replaces the old orchestrator
BFS, and fixes a latent ordering bug while doing so: the old worklist popped
LIFO, so an artist's albums were flattened in *reverse* order.

`tidaldy` overrides the method: playlist and album paths stream one chunk
per fetched page instead of paginating to exhaustion first (a new
`make_paginated_request` variant hands each page to a sink); everything else
follows the generic walk. Search-term track lists are a single page already.
The playlist arm also stops fetching the playlist metadata (title) — the
resolve needs only tracks.

`flume` is already a workspace dependency; `crabidy-core` adopts it for the
trait signature.

### D4 — Neither event loop blocks: spawned resolves, single-writer queue

Options considered:

1. Consume chunks inline in the playback loop's `handle_command` (loop still
   blocked for the whole resolve; pause/next dead — today's hidden defect).
2. Spawn the resolve; queue mutations travel back to the playback loop as
   internal commands, so the loop stays the **single writer** of queue state.

**Decision: (2), on both loops.**

- **Provider side**: `ProviderOrchestrator::run` wraps the orchestrator in an
  `Arc`; the `ResolveTracks` arm spawns the resolve onto its own task instead
  of awaiting it inline. Without this, the first chunk would deadlock the
  system: playback applies chunk 1 → `play()` → sends `GetTrackUrls` and
  awaits the reply — but the provider loop would still be busy resolving.
  Other provider commands keep flowing while (possibly several) resolves run.
- **Playback side**: each queue op registers a *pending op* (id from an
  `AtomicU64`, kind, insertion cursor) and spawns a forwarder task that
  drives `ProviderCommand::ResolveTracks` per path (concurrently, under the
  exponential read-ahead window of D8, but forwarding chunks in strict
  multi-path order) and forwards each chunk to the playback channel as
  `PlaybackCommand::ApplyResolvedChunk { op_id, tracks }`, followed by
  `ResolveFinished { op_id }`. Queue state is only ever mutated inside the
  loop, exactly as before; user commands interleave between chunks.

Backpressure is real at every hop: provider → forwarder over a small bounded
chunk channel, forwarder → playback over the existing `bounded(64)` command
channel. A slow consumer slows the HTTP fetching down instead of buffering
unboundedly.

### D5 — Chunk application semantics per op kind

Each pending op keeps an insertion cursor:

- **Replace**: first chunk `replace_with_tracks` (broadcast resets the
  queue, current position 0, playback starts); later chunks append.
- **Append**: every chunk `append_tracks`.
- **Queue** (after current): cursor starts at the current position; each
  chunk `insert_tracks(cursor)`, then `cursor += chunk.len()`.
- **Insert**: same, starting at the requested position.

Playback start reuses the existing `Option<Track>` returns from the
`QueueManager` mutations — only a chunk that makes a track current (replace,
or any insert into an empty queue) yields one, so exactly the first relevant
chunk starts the player and later chunks never restart it.

Interleaved edits from other clients during a resolve can shift the cursor's
target (e.g. removing tracks before it). This is accepted as benign:
`insert_tracks` already clamps, the queue self-heals on the next broadcast,
and simultaneous multi-client edits during a resolve are rare. Under
shuffle, arriving chunks are shuffled behind the current track like any
other insert — chunk order is irrelevant when shuffle is on. Each op counts
its applied tracks; an op that finishes with zero keeps today's
"resolved to no playable tracks" warning.

### D6 — `Replace` and `Clear` cancel in-flight resolves

Without cancellation, "replace the queue" or "clear the queue" during a
large resolve would be followed by the old op's remaining chunks trickling
back in — a corrupted queue, and the exact ghost behavior this feature is
meant to kill. Options: let stale chunks land (wrong), or cancel.

**Decision: cancel.** Each pending op carries an `Arc<AtomicBool>` shared
with its forwarder. `Replace` and `Clear` mark every pending op cancelled
and drop it from the map. The forwarder checks the flag per chunk and, when
set, drops the chunk receiver — the provider's next `send` fails and the
resolve task stops fetching (the documented receiver-drop semantics from
D3). Chunks already in flight for an unknown op id are ignored by the loop.
`Queue`/`Append`/`Insert` do **not** cancel: concurrent additive ops are
legal; their chunks interleave between ops while each op's internal order is
preserved.

### D7 — TUI indicator: an animated pseudo-item, outside the list model

While the latest `Queue` update carries `resolving = true`, the queue pane
renders one extra line after the last track: one to three dots cycling
(~400 ms per step, derived from elapsed time — the render loop already
redraws at least every 100 ms), in `COLOR_SECONDARY`. The pseudo-item is
appended at render time only and never enters `self.list`, so selection,
removal, and `get_size` cannot reach it — no new input states, nothing to
misclick.

### D8 — Exponential read-ahead across paths

The forwarder of D4 first resolved paths one at a time. That starts the first
track quickly (good) but fills the rest only as fast as one provider resolve
at a time. When enumeration is slow (per-path Tidal/YouTube/fyyd round trips)
and the leading tracks are very short or `is_skipped`, playback drains the
resolved queue faster than a sequential resolver refills it — and stalls into
silence, the exact thing progressive queueing exists to avoid.

Options considered:

1. **Sequential per path** (original): simplest, but a short/skipped head can
   outrun a slow resolver.
2. **Resolve every path at once**: fills fastest, but a large marked
   selection fires an unbounded burst of concurrent provider calls (rate
   limits, memory) the instant playback starts — wasteful when the user skips
   away after two tracks.
3. **Exponential read-ahead window**: a concurrency window that starts at 1
   and doubles (1, 2, 4, 8, 16, then steady 16) after each path completes.

**Decision: (3).** The first path resolves alone, so time-to-first-track is
unchanged from (1); the window then grows geometrically, so the resolved
queue runs exponentially ahead of linear playback and a short/skipped head
cannot catch it after the first couple of tracks. The cap (16) bounds
concurrent provider load. Chunks are still forwarded in strict path order —
the forwarder fully drains the oldest in-flight resolve before the next, so
concurrency never reorders the queue (D5's per-op cursor and "first chunk
starts the player" are untouched). Cancellation (D6) drops the in-flight
receivers, stopping every concurrent resolve at once.

This is a read-ahead over *paths*. A single collection path (one album or
playlist) is still enumerated by its provider's `resolve_tracks_into` — the
per-page streaming of D3 is that path's read-ahead — so the window is the win
for multi-item selections; single-collection latency stays a provider concern.

## Flows

```d2
shape: sequence_diagram
user: { shape: person }
tui: cbd-tui
rpc: gRPC handler
playback: playback loop
fwd: forwarder task
provider: provider loop
resolve: resolve task
tidal: Tidal API

user -> tui: queue large artist
tui -> rpc: Append(paths)
rpc -> playback: "PlaybackCommand::Append (fire-and-forget)"
playback -> playback: register pending op
playback -> tui: "Queue update (resolving=true)"
playback -> fwd: spawn
fwd -> provider: "ResolveTracks(path, chunk_tx)"
provider -> resolve: spawn
resolve -> tidal: fetch page 1
resolve -> fwd: chunk 1
fwd -> playback: "ApplyResolvedChunk(op, chunk 1)"
playback -> tui: "Queue update (resolving=true)"
playback -> playback: "play() first track"
resolve -> tidal: fetch page 2
resolve -> fwd: chunk 2
fwd -> playback: "ApplyResolvedChunk(op, chunk 2)"
playback -> tui: "Queue update (resolving=true)"
fwd -> playback: "ResolveFinished(op)"
playback -> tui: "Queue update (resolving=false)"
```

The dots pseudo-item is visible in the TUI exactly while updates carry
`resolving = true`; user commands (pause, next, remove) flow through the
playback loop between chunk applications instead of waiting for the end.

```d2
direction: right
core: "ProviderClient::resolve_tracks_into" {
  default: "default: pre-order walk,\none chunk per node"
}
tidaldy: "tidaldy override" {
  pages: "playlist/album:\none chunk per 50-track page"
}
playback: "playback loop" {
  ops: "pending ops:\ncursor + cancel flag"
}
core -> tidaldy: overridden by
tidaldy.pages -> playback.ops: "bounded chunks, in order"
playback.ops -> playback.ops: "apply + broadcast per chunk"
```

## Boundaries and risks

- **Proto**: one additive field (`Queue.resolving = 4`). No RPC shape
  changes; the queue ops stay fire-and-forget.
- **Trait**: one new `ProviderClient` method with a default implementation —
  existing providers (there is one) compile unchanged if they skip the
  override; the override is where the provider-level win lives.
- **Ordering fix is a behavior change**: multi-album artists now queue in
  listing order instead of reversed. Strictly a fix, noted here because
  someone may have gotten used to the bug.
- **Concurrent additive ops interleave between ops.** Each op's internal
  order is kept; the interleaving matches command arrival order at the loop.
  Accepted — same semantics a human doing two appends "at once" expects.
- **Old TUI + new server**: queue fills progressively, no indicator — pure
  improvement, no breakage. New TUI + old server: `resolving` is always
  false, indicator never shows, behavior as today.
- **Not in scope**: pagination of `get_lib_node` for *browsing* (the library
  pane still fetches collections to exhaustion before rendering), queue
  persistence, a progress percentage (total counts are known per collection
  but not aggregated across a nested walk).